FAQ
Hi Marek,

Moving this to cdh-user@cloudera.org as its CDH specific. I've bcc'd
mapreduce-user@ and cc'd you in case you aren't a subscriber (@
https://groups.google.com/a/cloudera.org/group/cdh-user)

A few questions though:
- What OS are you running and what arch? Check via "lsb_release -a"
and "uname -a"
-- Is your JVM also same as your arch? Check via
"/usr/java/default/bin/java -version"
- Why do you provide
LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64 as
mapred.child.env? It isn't usually required if you have
hadoop-0.20-native installed and the TaskTrackers were restarted since
then.
- Do other codecs (such as GZip and Deflate (Default)) work fine?
On Mon, Jun 11, 2012 at 7:24 PM, Marek Miglinski wrote:
Hi,

I have a Clouderas CDH3u3 installed on my cluster and mapred.child.env set to "LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64" (with libsnappy.so in the folder) in mapred-site.xml. Cloudera says that Snappy is included in their hadoop-0.20-native package and it is also installed on each of the nodes.

But when I run a mapreduce task with "mapred.map.output.compression.codec" set to "org.apache.hadoop.io.compress.SnappyCodec" I get an exception:

java.lang.RuntimeException: native snappy library not available


Any idea why?



Thanks,
Marek M.


--
Harsh J

Search Discussions

  • Migmarek at Jun 11, 2012 at 2:27 pm
    Issue fixed by removing LD_LIBRARY_PATH from mapred.child.env in
    mapred-site.xml
    I had libsnappy.so in native directory and it seems that mapred was trying
    to use this library, but not the one from hadoop-0.20.native package. The
    reason I was using LD_LIBRARY_PATH is Lzo codec that required native
    libraries.

    Thank you for your help.
    On Monday, June 11, 2012 5:06:36 PM UTC+3, Harsh J wrote:

    Hi Marek,

    Moving this to cdh-user@cloudera.org as its CDH specific. I've bcc'd
    mapreduce-user@ and cc'd you in case you aren't a subscriber (@
    https://groups.google.com/a/cloudera.org/group/cdh-user)

    A few questions though:
    - What OS are you running and what arch? Check via "lsb_release -a"
    and "uname -a"
    -- Is your JVM also same as your arch? Check via
    "/usr/java/default/bin/java -version"
    - Why do you provide
    LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64 as
    mapred.child.env? It isn't usually required if you have
    hadoop-0.20-native installed and the TaskTrackers were restarted since
    then.
    - Do other codecs (such as GZip and Deflate (Default)) work fine?
    On Mon, Jun 11, 2012 at 7:24 PM, Marek Miglinski wrote:
    Hi,

    I have a Clouderas CDH3u3 installed on my cluster and mapred.child.env
    set to "LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64"
    (with libsnappy.so in the folder) in mapred-site.xml. Cloudera says that
    Snappy is included in their hadoop-0.20-native package and it is also
    installed on each of the nodes.
    But when I run a mapreduce task with
    "mapred.map.output.compression.codec" set to
    "org.apache.hadoop.io.compress.SnappyCodec" I get an exception:
    java.lang.RuntimeException: native snappy library not available


    Any idea why?



    Thanks,
    Marek M.


    --
    Harsh J
    On Monday, June 11, 2012 5:06:36 PM UTC+3, Harsh J wrote:

    Hi Marek,

    Moving this to cdh-user@cloudera.org as its CDH specific. I've bcc'd
    mapreduce-user@ and cc'd you in case you aren't a subscriber (@
    https://groups.google.com/a/cloudera.org/group/cdh-user)

    A few questions though:
    - What OS are you running and what arch? Check via "lsb_release -a"
    and "uname -a"
    -- Is your JVM also same as your arch? Check via
    "/usr/java/default/bin/java -version"
    - Why do you provide
    LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64 as
    mapred.child.env? It isn't usually required if you have
    hadoop-0.20-native installed and the TaskTrackers were restarted since
    then.
    - Do other codecs (such as GZip and Deflate (Default)) work fine?
    On Mon, Jun 11, 2012 at 7:24 PM, Marek Miglinski wrote:
    Hi,

    I have a Clouderas CDH3u3 installed on my cluster and mapred.child.env
    set to "LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64"
    (with libsnappy.so in the folder) in mapred-site.xml. Cloudera says that
    Snappy is included in their hadoop-0.20-native package and it is also
    installed on each of the nodes.
    But when I run a mapreduce task with
    "mapred.map.output.compression.codec" set to
    "org.apache.hadoop.io.compress.SnappyCodec" I get an exception:
    java.lang.RuntimeException: native snappy library not available


    Any idea why?



    Thanks,
    Marek M.


    --
    Harsh J
    On Monday, June 11, 2012 5:06:36 PM UTC+3, Harsh J wrote:

    Hi Marek,

    Moving this to cdh-user@cloudera.org as its CDH specific. I've bcc'd
    mapreduce-user@ and cc'd you in case you aren't a subscriber (@
    https://groups.google.com/a/cloudera.org/group/cdh-user)

    A few questions though:
    - What OS are you running and what arch? Check via "lsb_release -a"
    and "uname -a"
    -- Is your JVM also same as your arch? Check via
    "/usr/java/default/bin/java -version"
    - Why do you provide
    LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64 as
    mapred.child.env? It isn't usually required if you have
    hadoop-0.20-native installed and the TaskTrackers were restarted since
    then.
    - Do other codecs (such as GZip and Deflate (Default)) work fine?
    On Mon, Jun 11, 2012 at 7:24 PM, Marek Miglinski wrote:
    Hi,

    I have a Clouderas CDH3u3 installed on my cluster and mapred.child.env
    set to "LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64"
    (with libsnappy.so in the folder) in mapred-site.xml. Cloudera says that
    Snappy is included in their hadoop-0.20-native package and it is also
    installed on each of the nodes.
    But when I run a mapreduce task with
    "mapred.map.output.compression.codec" set to
    "org.apache.hadoop.io.compress.SnappyCodec" I get an exception:
    java.lang.RuntimeException: native snappy library not available


    Any idea why?



    Thanks,
    Marek M.


    --
    Harsh J
  • Harsh J at Jun 11, 2012 at 2:34 pm
    Good to know! For LZO, you may also use
    https://github.com/toddlipcon/hadoop-lzo-packager to create RPMs/DEBs
    that fit in snugly with CDH.
    On Mon, Jun 11, 2012 at 7:56 PM, wrote:
    Issue fixed by removing LD_LIBRARY_PATH from mapred.child.env in
    mapred-site.xml
    I had libsnappy.so in native directory and it seems that mapred was trying
    to use this library, but not the one from hadoop-0.20.native package. The
    reason I was using LD_LIBRARY_PATH is Lzo codec that required native
    libraries.

    Thank you for your help.

    On Monday, June 11, 2012 5:06:36 PM UTC+3, Harsh J wrote:

    Hi Marek,

    Moving this to cdh-user@cloudera.org as its CDH specific. I've bcc'd
    mapreduce-user@ and cc'd you in case you aren't a subscriber (@
    https://groups.google.com/a/cloudera.org/group/cdh-user)

    A few questions though:
    - What OS are you running and what arch? Check via "lsb_release -a"
    and "uname -a"
    -- Is your JVM also same as your arch? Check via
    "/usr/java/default/bin/java -version"
    - Why do you provide
    LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64 as
    mapred.child.env? It isn't usually required if you have
    hadoop-0.20-native installed and the TaskTrackers were restarted since
    then.
    - Do other codecs (such as GZip and Deflate (Default)) work fine?

    On Mon, Jun 11, 2012 at 7:24 PM, Marek Miglinski <mmiglinski@seven.com>
    wrote:
    Hi,

    I have a Clouderas CDH3u3 installed on my cluster and mapred.child.env
    set to "LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64"
    (with libsnappy.so in the folder) in mapred-site.xml. Cloudera says that
    Snappy is included in their hadoop-0.20-native package and it is also
    installed on each of the nodes.

    But when I run a mapreduce task with
    "mapred.map.output.compression.codec" set to
    "org.apache.hadoop.io.compress.SnappyCodec" I get an exception:

    java.lang.RuntimeException: native snappy library not available


    Any idea why?



    Thanks,
    Marek M.


    --
    Harsh J
    On Monday, June 11, 2012 5:06:36 PM UTC+3, Harsh J wrote:

    Hi Marek,

    Moving this to cdh-user@cloudera.org as its CDH specific. I've bcc'd
    mapreduce-user@ and cc'd you in case you aren't a subscriber (@
    https://groups.google.com/a/cloudera.org/group/cdh-user)

    A few questions though:
    - What OS are you running and what arch? Check via "lsb_release -a"
    and "uname -a"
    -- Is your JVM also same as your arch? Check via
    "/usr/java/default/bin/java -version"
    - Why do you provide
    LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64 as
    mapred.child.env? It isn't usually required if you have
    hadoop-0.20-native installed and the TaskTrackers were restarted since
    then.
    - Do other codecs (such as GZip and Deflate (Default)) work fine?

    On Mon, Jun 11, 2012 at 7:24 PM, Marek Miglinski <mmiglinski@seven.com>
    wrote:
    Hi,

    I have a Clouderas CDH3u3 installed on my cluster and mapred.child.env
    set to "LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64"
    (with libsnappy.so in the folder) in mapred-site.xml. Cloudera says that
    Snappy is included in their hadoop-0.20-native package and it is also
    installed on each of the nodes.

    But when I run a mapreduce task with
    "mapred.map.output.compression.codec" set to
    "org.apache.hadoop.io.compress.SnappyCodec" I get an exception:

    java.lang.RuntimeException: native snappy library not available


    Any idea why?



    Thanks,
    Marek M.


    --
    Harsh J
    On Monday, June 11, 2012 5:06:36 PM UTC+3, Harsh J wrote:

    Hi Marek,

    Moving this to cdh-user@cloudera.org as its CDH specific. I've bcc'd
    mapreduce-user@ and cc'd you in case you aren't a subscriber (@
    https://groups.google.com/a/cloudera.org/group/cdh-user)

    A few questions though:
    - What OS are you running and what arch? Check via "lsb_release -a"
    and "uname -a"
    -- Is your JVM also same as your arch? Check via
    "/usr/java/default/bin/java -version"
    - Why do you provide
    LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64 as
    mapred.child.env? It isn't usually required if you have
    hadoop-0.20-native installed and the TaskTrackers were restarted since
    then.
    - Do other codecs (such as GZip and Deflate (Default)) work fine?

    On Mon, Jun 11, 2012 at 7:24 PM, Marek Miglinski <mmiglinski@seven.com>
    wrote:
    Hi,

    I have a Clouderas CDH3u3 installed on my cluster and mapred.child.env
    set to "LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64"
    (with libsnappy.so in the folder) in mapred-site.xml. Cloudera says that
    Snappy is included in their hadoop-0.20-native package and it is also
    installed on each of the nodes.

    But when I run a mapreduce task with
    "mapred.map.output.compression.codec" set to
    "org.apache.hadoop.io.compress.SnappyCodec" I get an exception:

    java.lang.RuntimeException: native snappy library not available


    Any idea why?



    Thanks,
    Marek M.


    --
    Harsh J


    --
    Harsh J

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcdh-user @
categorieshadoop
postedJun 11, '12 at 2:07p
activeJun 11, '12 at 2:34p
posts3
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Harsh J: 2 posts Migmarek: 1 post

People

Translate

site design / logo © 2022 Grokbase