FAQ
Hello All,

We use the tarball hadoop and hbase distributions and are having some trouble upgrading to CDH4 from CDH3, specifically around the native snappy codec.

It was not included with the tarball (gz was included and compiled, when I moved them to $HADOOP_HOME/lib/native/Linux-amd64-64 it works when I used the hbase compression test). I built the snappy libraries it in the mrv1 hadoop release tarball using the same method I used for CDH3:

In $HADOOP_HOME:

sh cloudera/do-release-build

That built the libraries, I then dropped the newly made native snappy libraries in the same place as the normal native hadoop libraries and ran the hbase test:

hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://namenode:8020/titan/tmp snappy

and it seems to find the native library and load them, but then fails with this:

12/06/27 00:01:21 INFO hfile.CacheConfig: Allocating LruBlockCache with maximum size 246.8m
12/06/27 00:01:21 WARN snappy.LoadSnappy: Snappy native library is available
12/06/27 00:01:21 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/06/27 00:01:21 INFO snappy.LoadSnappy: Snappy native library loaded
12/06/27 00:01:21 WARN snappy.SnappyCompressor: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs()V
12/06/27 00:01:21 INFO compress.CodecPool: Got brand-new compressor [.snappy]
12/06/27 00:01:21 DEBUG hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native Method)
at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:146)
at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:140)
at org.apache.hadoop.hbase.io.hfile.Compression$FinishOnFlushCompressionStream.flush(Compression.java:67)
....


Before I built the native libraries it just errored out saying the native libraries for snappy were not available. I saw the notes in snappy-java about having the snappy-java.jar only get loaded in once, so I cleaned it out from all the places it was being loaded in, checking via running:

hbase classpath

but to no luck after removing all of the extra occurrences. Any ideas? I'll be poking around this more, hopefully I missed something obvious.

thanks,
-chris

--

Search Discussions

  • Alejandro Abdelnur at Jun 27, 2012 at 3:27 pm
    Hi Chris,

    The tarball are meant for use without native libraries. If you want to use
    native libraries you should install from the corresponding RPM/DEB packages
    for the linux distribution your are using.

    Thanks and best regards.
    On Wed, Jun 27, 2012 at 12:16 AM, Chris Tarnas wrote:

    Hello All,

    We use the tarball hadoop and hbase distributions and are having some
    trouble upgrading to CDH4 from CDH3, specifically around the native snappy
    codec.

    It was not included with the tarball (gz was included and compiled, when I
    moved them to $HADOOP_HOME/lib/native/Linux-amd64-64 it works when I used
    the hbase compression test). I built the snappy libraries it in the mrv1
    hadoop release tarball using the same method I used for CDH3:

    In $HADOOP_HOME:

    sh cloudera/do-release-build

    That built the libraries, I then dropped the newly made native snappy
    libraries in the same place as the normal native hadoop libraries and ran
    the hbase test:

    hbase org.apache.hadoop.hbase.util.CompressionTest
    hdfs://namenode:8020/titan/tmp snappy

    and it seems to find the native library and load them, but then fails with
    this:

    12/06/27 00:01:21 INFO hfile.CacheConfig: Allocating LruBlockCache with
    maximum size 246.8m
    12/06/27 00:01:21 WARN snappy.LoadSnappy: Snappy native library is
    available
    12/06/27 00:01:21 INFO util.NativeCodeLoader: Loaded the native-hadoop
    library
    12/06/27 00:01:21 INFO snappy.LoadSnappy: Snappy native library loaded
    12/06/27 00:01:21 WARN snappy.SnappyCompressor:
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs()V
    12/06/27 00:01:21 INFO compress.CodecPool: Got brand-new compressor
    [.snappy]
    12/06/27 00:01:21 DEBUG hfile.HFileWriterV2: Initialized with
    CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false]
    [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false]
    [cacheEvictOnClose=false] [cacheCompressed=false]
    Exception in thread "main" java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native
    Method)
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:146)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:140)
    at
    org.apache.hadoop.hbase.io.hfile.Compression$FinishOnFlushCompressionStream.flush(Compression.java:67)
    ....


    Before I built the native libraries it just errored out saying the native
    libraries for snappy were not available. I saw the notes in snappy-java
    about having the snappy-java.jar only get loaded in once, so I cleaned it
    out from all the places it was being loaded in, checking via running:

    hbase classpath

    but to no luck after removing all of the extra occurrences. Any ideas?
    I'll be poking around this more, hopefully I missed something obvious.

    thanks,
    -chris

    --



    --
    Alejandro

    --
  • Chris Tarnas at Jun 27, 2012 at 7:15 pm
    Hi Alejandro

    In the past this worked fine (and the tarballs do include the all native libraries other than snappy). Is this a recent change?

    thanks,
    -chris
    On Jun 27, 2012, at 8:27 AM, Alejandro Abdelnur wrote:

    Hi Chris,

    The tarball are meant for use without native libraries. If you want to use native libraries you should install from the corresponding RPM/DEB packages for the linux distribution your are using.

    Thanks and best regards.

    On Wed, Jun 27, 2012 at 12:16 AM, Chris Tarnas wrote:
    Hello All,

    We use the tarball hadoop and hbase distributions and are having some trouble upgrading to CDH4 from CDH3, specifically around the native snappy codec.

    It was not included with the tarball (gz was included and compiled, when I moved them to $HADOOP_HOME/lib/native/Linux-amd64-64 it works when I used the hbase compression test). I built the snappy libraries it in the mrv1 hadoop release tarball using the same method I used for CDH3:

    In $HADOOP_HOME:

    sh cloudera/do-release-build

    That built the libraries, I then dropped the newly made native snappy libraries in the same place as the normal native hadoop libraries and ran the hbase test:

    hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://namenode:8020/titan/tmp snappy

    and it seems to find the native library and load them, but then fails with this:

    12/06/27 00:01:21 INFO hfile.CacheConfig: Allocating LruBlockCache with maximum size 246.8m
    12/06/27 00:01:21 WARN snappy.LoadSnappy: Snappy native library is available
    12/06/27 00:01:21 INFO util.NativeCodeLoader: Loaded the native-hadoop library
    12/06/27 00:01:21 INFO snappy.LoadSnappy: Snappy native library loaded
    12/06/27 00:01:21 WARN snappy.SnappyCompressor: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs()V
    12/06/27 00:01:21 INFO compress.CodecPool: Got brand-new compressor [.snappy]
    12/06/27 00:01:21 DEBUG hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
    Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
    at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native Method)
    at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
    at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:146)
    at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:140)
    at org.apache.hadoop.hbase.io.hfile.Compression$FinishOnFlushCompressionStream.flush(Compression.java:67)
    ....


    Before I built the native libraries it just errored out saying the native libraries for snappy were not available. I saw the notes in snappy-java about having the snappy-java.jar only get loaded in once, so I cleaned it out from all the places it was being loaded in, checking via running:

    hbase classpath

    but to no luck after removing all of the extra occurrences. Any ideas? I'll be poking around this more, hopefully I missed something obvious.

    thanks,
    -chris

    --






    --
    Alejandro

    --

  • Alejandro Abdelnur at Jun 27, 2012 at 7:45 pm
    Chris,

    No it has been always like that, the nativelibs are compiled using versions
    of native dependencies (ie libc) available in the platform where they are
    compiled. The current CDH tarballs are generic, not per platform,
    the existence of native libraries in them seems an oversight, let me check
    this and I'll get back to you with a definitive answer.

    Thanks.
    On Wed, Jun 27, 2012 at 10:10 AM, Chris Tarnas wrote:

    Hi Alejandro

    In the past this worked fine (and the tarballs do include the all native
    libraries other than snappy). Is this a recent change?

    thanks,
    -chris

    On Jun 27, 2012, at 8:27 AM, Alejandro Abdelnur wrote:

    Hi Chris,

    The tarball are meant for use without native libraries. If you want to use
    native libraries you should install from the corresponding RPM/DEB packages
    for the linux distribution your are using.

    Thanks and best regards.
    On Wed, Jun 27, 2012 at 12:16 AM, Chris Tarnas wrote:

    Hello All,

    We use the tarball hadoop and hbase distributions and are having some
    trouble upgrading to CDH4 from CDH3, specifically around the native snappy
    codec.

    It was not included with the tarball (gz was included and compiled, when
    I moved them to $HADOOP_HOME/lib/native/Linux-amd64-64 it works when I used
    the hbase compression test). I built the snappy libraries it in the mrv1
    hadoop release tarball using the same method I used for CDH3:

    In $HADOOP_HOME:

    sh cloudera/do-release-build

    That built the libraries, I then dropped the newly made native snappy
    libraries in the same place as the normal native hadoop libraries and ran
    the hbase test:

    hbase org.apache.hadoop.hbase.util.CompressionTest
    hdfs://namenode:8020/titan/tmp snappy

    and it seems to find the native library and load them, but then fails
    with this:

    12/06/27 00:01:21 INFO hfile.CacheConfig: Allocating LruBlockCache with
    maximum size 246.8m
    12/06/27 00:01:21 WARN snappy.LoadSnappy: Snappy native library is
    available
    12/06/27 00:01:21 INFO util.NativeCodeLoader: Loaded the native-hadoop
    library
    12/06/27 00:01:21 INFO snappy.LoadSnappy: Snappy native library loaded
    12/06/27 00:01:21 WARN snappy.SnappyCompressor:
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs()V
    12/06/27 00:01:21 INFO compress.CodecPool: Got brand-new compressor
    [.snappy]
    12/06/27 00:01:21 DEBUG hfile.HFileWriterV2: Initialized with
    CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false]
    [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false]
    [cacheEvictOnClose=false] [cacheCompressed=false]
    Exception in thread "main" java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native
    Method)
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:146)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:140)
    at
    org.apache.hadoop.hbase.io.hfile.Compression$FinishOnFlushCompressionStream.flush(Compression.java:67)
    ....


    Before I built the native libraries it just errored out saying the native
    libraries for snappy were not available. I saw the notes in snappy-java
    about having the snappy-java.jar only get loaded in once, so I cleaned it
    out from all the places it was being loaded in, checking via running:

    hbase classpath

    but to no luck after removing all of the extra occurrences. Any ideas?
    I'll be poking around this more, hopefully I missed something obvious.

    thanks,
    -chris

    --



    --
    Alejandro

    --





    --
    Alejandro
  • Todd Lipcon at Jun 27, 2012 at 8:00 pm
    Hi Chris,

    You're in "advanced" territory here -- but I think you might have luck if
    you copy the actual libsnappy.so into the native lib directory as well. You
    need to make sure that the native compilation step picked up the snappy
    includes/lib when you built it, as well -- otherwise you end up with a
    hadoop native lib without snappy support built in.

    -Todd
    On Wed, Jun 27, 2012 at 12:44 PM, Alejandro Abdelnur wrote:

    Chris,

    No it has been always like that, the nativelibs are compiled using
    versions of native dependencies (ie libc) available in the platform where
    they are compiled. The current CDH tarballs are generic, not per platform,
    the existence of native libraries in them seems an oversight, let me check
    this and I'll get back to you with a definitive answer.

    Thanks.

    On Wed, Jun 27, 2012 at 10:10 AM, Chris Tarnas wrote:

    Hi Alejandro

    In the past this worked fine (and the tarballs do include the all native
    libraries other than snappy). Is this a recent change?

    thanks,
    -chris

    On Jun 27, 2012, at 8:27 AM, Alejandro Abdelnur wrote:

    Hi Chris,

    The tarball are meant for use without native libraries. If you want to
    use native libraries you should install from the corresponding RPM/DEB
    packages for the linux distribution your are using.

    Thanks and best regards.
    On Wed, Jun 27, 2012 at 12:16 AM, Chris Tarnas wrote:

    Hello All,

    We use the tarball hadoop and hbase distributions and are having some
    trouble upgrading to CDH4 from CDH3, specifically around the native snappy
    codec.

    It was not included with the tarball (gz was included and compiled, when
    I moved them to $HADOOP_HOME/lib/native/Linux-amd64-64 it works when I used
    the hbase compression test). I built the snappy libraries it in the mrv1
    hadoop release tarball using the same method I used for CDH3:

    In $HADOOP_HOME:

    sh cloudera/do-release-build

    That built the libraries, I then dropped the newly made native snappy
    libraries in the same place as the normal native hadoop libraries and ran
    the hbase test:

    hbase org.apache.hadoop.hbase.util.CompressionTest
    hdfs://namenode:8020/titan/tmp snappy

    and it seems to find the native library and load them, but then fails
    with this:

    12/06/27 00:01:21 INFO hfile.CacheConfig: Allocating LruBlockCache with
    maximum size 246.8m
    12/06/27 00:01:21 WARN snappy.LoadSnappy: Snappy native library is
    available
    12/06/27 00:01:21 INFO util.NativeCodeLoader: Loaded the native-hadoop
    library
    12/06/27 00:01:21 INFO snappy.LoadSnappy: Snappy native library loaded
    12/06/27 00:01:21 WARN snappy.SnappyCompressor:
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs()V
    12/06/27 00:01:21 INFO compress.CodecPool: Got brand-new compressor
    [.snappy]
    12/06/27 00:01:21 DEBUG hfile.HFileWriterV2: Initialized with
    CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false]
    [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false]
    [cacheEvictOnClose=false] [cacheCompressed=false]
    Exception in thread "main" java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native
    Method)
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:146)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:140)
    at
    org.apache.hadoop.hbase.io.hfile.Compression$FinishOnFlushCompressionStream.flush(Compression.java:67)
    ....


    Before I built the native libraries it just errored out saying the
    native libraries for snappy were not available. I saw the notes in
    snappy-java about having the snappy-java.jar only get loaded in once, so I
    cleaned it out from all the places it was being loaded in, checking via
    running:

    hbase classpath

    but to no luck after removing all of the extra occurrences. Any ideas?
    I'll be poking around this more, hopefully I missed something obvious.

    thanks,
    -chris

    --



    --
    Alejandro

    --





    --
    Alejandro


    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Chris Tarnas at Jun 27, 2012 at 9:47 pm
    Hi Todd,

    I've tried various permutations of dropping in libsnappy.so into the native lib directory (from the build, built separately, system, etc), all with no luck - it loads the library but then gets the UnsatisfiedLinkError when it tries to use it. I then tried to build all of hadoop in the src directory (using mvn compile -Pnative ) , and it complies hadoop-common just fine, but when I run the test suite it fails, one of the areas is snappy. It looks like the underlying problem is not in the libsnappy.so (which I built) but in org.apache.hadoop.io.compress.snappy.SnappyCompressor - that is where the linking error is coming from, in libhadoop.so. I suppose it could be not finding libsnappy, but we have the system one installed here as well as the one in the hadoop native libs dir.

    Are there issues compiling with some versions of Java or GCC? I'm using a fairly generic CentOS 6.0 box with the included GCC 4.4.4-13 and Oracle's Java 1.6.0_22? Or possibly some maigc needed with the environment with the integrated snappy? This worked fine in CDH3...


    thanks again,
    -chris

    On Jun 27, 2012, at 1:00 PM, Todd Lipcon wrote:

    Hi Chris,

    You're in "advanced" territory here -- but I think you might have luck if you copy the actual libsnappy.so into the native lib directory as well. You need to make sure that the native compilation step picked up the snappy includes/lib when you built it, as well -- otherwise you end up with a hadoop native lib without snappy support built in.

    -Todd

    On Wed, Jun 27, 2012 at 12:44 PM, Alejandro Abdelnur wrote:
    Chris,

    No it has been always like that, the nativelibs are compiled using versions of native dependencies (ie libc) available in the platform where they are compiled. The current CDH tarballs are generic, not per platform, the existence of native libraries in them seems an oversight, let me check this and I'll get back to you with a definitive answer.

    Thanks.


    On Wed, Jun 27, 2012 at 10:10 AM, Chris Tarnas wrote:
    Hi Alejandro

    In the past this worked fine (and the tarballs do include the all native libraries other than snappy). Is this a recent change?

    thanks,
    -chris
    On Jun 27, 2012, at 8:27 AM, Alejandro Abdelnur wrote:

    Hi Chris,

    The tarball are meant for use without native libraries. If you want to use native libraries you should install from the corresponding RPM/DEB packages for the linux distribution your are using.

    Thanks and best regards.

    On Wed, Jun 27, 2012 at 12:16 AM, Chris Tarnas wrote:
    Hello All,

    We use the tarball hadoop and hbase distributions and are having some trouble upgrading to CDH4 from CDH3, specifically around the native snappy codec.

    It was not included with the tarball (gz was included and compiled, when I moved them to $HADOOP_HOME/lib/native/Linux-amd64-64 it works when I used the hbase compression test). I built the snappy libraries it in the mrv1 hadoop release tarball using the same method I used for CDH3:

    In $HADOOP_HOME:

    sh cloudera/do-release-build

    That built the libraries, I then dropped the newly made native snappy libraries in the same place as the normal native hadoop libraries and ran the hbase test:

    hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://namenode:8020/titan/tmp snappy

    and it seems to find the native library and load them, but then fails with this:

    12/06/27 00:01:21 INFO hfile.CacheConfig: Allocating LruBlockCache with maximum size 246.8m
    12/06/27 00:01:21 WARN snappy.LoadSnappy: Snappy native library is available
    12/06/27 00:01:21 INFO util.NativeCodeLoader: Loaded the native-hadoop library
    12/06/27 00:01:21 INFO snappy.LoadSnappy: Snappy native library loaded
    12/06/27 00:01:21 WARN snappy.SnappyCompressor: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs()V
    12/06/27 00:01:21 INFO compress.CodecPool: Got brand-new compressor [.snappy]
    12/06/27 00:01:21 DEBUG hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
    Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
    at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native Method)
    at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
    at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:146)
    at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:140)
    at org.apache.hadoop.hbase.io.hfile.Compression$FinishOnFlushCompressionStream.flush(Compression.java:67)
    ....


    Before I built the native libraries it just errored out saying the native libraries for snappy were not available. I saw the notes in snappy-java about having the snappy-java.jar only get loaded in once, so I cleaned it out from all the places it was being loaded in, checking via running:

    hbase classpath

    but to no luck after removing all of the extra occurrences. Any ideas? I'll be poking around this more, hopefully I missed something obvious.

    thanks,
    -chris

    --






    --
    Alejandro

    --




    --
    Alejandro



    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Chris Tarnas at Jun 27, 2012 at 9:47 pm
    I re-built for the n-th time, this time unsetting HADOOP_HOME. Not sure if that was all, but it now works. Very strange.


    thank you,
    -chris


    On Jun 27, 2012, at 1:46 PM, Chris Tarnas wrote:

    Hi Todd,

    I've tried various permutations of dropping in libsnappy.so into the native lib directory (from the build, built separately, system, etc), all with no luck - it loads the library but then gets the UnsatisfiedLinkError when it tries to use it. I then tried to build all of hadoop in the src directory (using mvn compile -Pnative ) , and it complies hadoop-common just fine, but when I run the test suite it fails, one of the areas is snappy. It looks like the underlying problem is not in the libsnappy.so (which I built) but in org.apache.hadoop.io.compress.snappy.SnappyCompressor - that is where the linking error is coming from, in libhadoop.so. I suppose it could be not finding libsnappy, but we have the system one installed here as well as the one in the hadoop native libs dir.

    Are there issues compiling with some versions of Java or GCC? I'm using a fairly generic CentOS 6.0 box with the included GCC 4.4.4-13 and Oracle's Java 1.6.0_22? Or possibly some maigc needed with the environment with the integrated snappy? This worked fine in CDH3...


    thanks again,
    -chris

    On Jun 27, 2012, at 1:00 PM, Todd Lipcon wrote:

    Hi Chris,

    You're in "advanced" territory here -- but I think you might have luck if you copy the actual libsnappy.so into the native lib directory as well. You need to make sure that the native compilation step picked up the snappy includes/lib when you built it, as well -- otherwise you end up with a hadoop native lib without snappy support built in.

    -Todd

    On Wed, Jun 27, 2012 at 12:44 PM, Alejandro Abdelnur wrote:
    Chris,

    No it has been always like that, the nativelibs are compiled using versions of native dependencies (ie libc) available in the platform where they are compiled. The current CDH tarballs are generic, not per platform, the existence of native libraries in them seems an oversight, let me check this and I'll get back to you with a definitive answer.

    Thanks.


    On Wed, Jun 27, 2012 at 10:10 AM, Chris Tarnas wrote:
    Hi Alejandro

    In the past this worked fine (and the tarballs do include the all native libraries other than snappy). Is this a recent change?

    thanks,
    -chris
    On Jun 27, 2012, at 8:27 AM, Alejandro Abdelnur wrote:

    Hi Chris,

    The tarball are meant for use without native libraries. If you want to use native libraries you should install from the corresponding RPM/DEB packages for the linux distribution your are using.

    Thanks and best regards.

    On Wed, Jun 27, 2012 at 12:16 AM, Chris Tarnas wrote:
    Hello All,

    We use the tarball hadoop and hbase distributions and are having some trouble upgrading to CDH4 from CDH3, specifically around the native snappy codec.

    It was not included with the tarball (gz was included and compiled, when I moved them to $HADOOP_HOME/lib/native/Linux-amd64-64 it works when I used the hbase compression test). I built the snappy libraries it in the mrv1 hadoop release tarball using the same method I used for CDH3:

    In $HADOOP_HOME:

    sh cloudera/do-release-build

    That built the libraries, I then dropped the newly made native snappy libraries in the same place as the normal native hadoop libraries and ran the hbase test:

    hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://namenode:8020/titan/tmp snappy

    and it seems to find the native library and load them, but then fails with this:

    12/06/27 00:01:21 INFO hfile.CacheConfig: Allocating LruBlockCache with maximum size 246.8m
    12/06/27 00:01:21 WARN snappy.LoadSnappy: Snappy native library is available
    12/06/27 00:01:21 INFO util.NativeCodeLoader: Loaded the native-hadoop library
    12/06/27 00:01:21 INFO snappy.LoadSnappy: Snappy native library loaded
    12/06/27 00:01:21 WARN snappy.SnappyCompressor: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs()V
    12/06/27 00:01:21 INFO compress.CodecPool: Got brand-new compressor [.snappy]
    12/06/27 00:01:21 DEBUG hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
    Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
    at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native Method)
    at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
    at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:146)
    at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:140)
    at org.apache.hadoop.hbase.io.hfile.Compression$FinishOnFlushCompressionStream.flush(Compression.java:67)
    ....


    Before I built the native libraries it just errored out saying the native libraries for snappy were not available. I saw the notes in snappy-java about having the snappy-java.jar only get loaded in once, so I cleaned it out from all the places it was being loaded in, checking via running:

    hbase classpath

    but to no luck after removing all of the extra occurrences. Any ideas? I'll be poking around this more, hopefully I missed something obvious.

    thanks,
    -chris

    --






    --
    Alejandro

    --




    --
    Alejandro



    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Chris Tarnas at Jul 20, 2012 at 2:51 pm
    Hi Howard,

    Further investigation led me to find that the -Dsnappy.prefix= build option doesn't work quite right on its own. You need to either have snappy installed in the standard system location or have all of the environment variables set to point to your snappy install's libraries and headers.

    -chris
    On Jul 20, 2012, at 5:12 AM, howard wrote:

    Hi,chris
    how do you resolve the problem?
    export HADOOP_HOME=

    I find the same problem in CDH3U4,but in CDH3U1,there is no problem.

    在 2012年6月28日星期四UTC+8上午5时00分32秒,Chris Tarnas写道:
    I re-built for the n-th time, this time unsetting HADOOP_HOME. Not sure if that was all, but it now works. Very strange.


    thank you,
    -chris


    On Jun 27, 2012, at 1:46 PM, Chris Tarnas wrote:

    Hi Todd,

    I've tried various permutations of dropping in libsnappy.so into the native lib directory (from the build, built separately, system, etc), all with no luck - it loads the library but then gets the UnsatisfiedLinkError when it tries to use it. I then tried to build all of hadoop in the src directory (using mvn compile -Pnative ) , and it complies hadoop-common just fine, but when I run the test suite it fails, one of the areas is snappy. It looks like the underlying problem is not in the libsnappy.so (which I built) but in org.apache.hadoop.io.compress.snappy.SnappyCompressor - that is where the linking error is coming from, in libhadoop.so. I suppose it could be not finding libsnappy, but we have the system one installed here as well as the one in the hadoop native libs dir.

    Are there issues compiling with some versions of Java or GCC? I'm using a fairly generic CentOS 6.0 box with the included GCC 4.4.4-13 and Oracle's Java 1.6.0_22? Or possibly some maigc needed with the environment with the integrated snappy? This worked fine in CDH3...


    thanks again,
    -chris

    On Jun 27, 2012, at 1:00 PM, Todd Lipcon wrote:

    Hi Chris,

    You're in "advanced" territory here -- but I think you might have luck if you copy the actual libsnappy.so into the native lib directory as well. You need to make sure that the native compilation step picked up the snappy includes/lib when you built it, as well -- otherwise you end up with a hadoop native lib without snappy support built in.

    -Todd

    On Wed, Jun 27, 2012 at 12:44 PM, Alejandro Abdelnur wrote:
    Chris,

    No it has been always like that, the nativelibs are compiled using versions of native dependencies (ie libc) available in the platform where they are compiled. The current CDH tarballs are generic, not per platform, the existence of native libraries in them seems an oversight, let me check this and I'll get back to you with a definitive answer.

    Thanks.


    On Wed, Jun 27, 2012 at 10:10 AM, Chris Tarnas wrote:
    Hi Alejandro

    In the past this worked fine (and the tarballs do include the all native libraries other than snappy). Is this a recent change?

    thanks,
    -chris
    On Jun 27, 2012, at 8:27 AM, Alejandro Abdelnur wrote:

    Hi Chris,

    The tarball are meant for use without native libraries. If you want to use native libraries you should install from the corresponding RPM/DEB packages for the linux distribution your are using.

    Thanks and best regards.

    On Wed, Jun 27, 2012 at 12:16 AM, Chris Tarnas wrote:
    Hello All,

    We use the tarball hadoop and hbase distributions and are having some trouble upgrading to CDH4 from CDH3, specifically around the native snappy codec.

    It was not included with the tarball (gz was included and compiled, when I moved them to $HADOOP_HOME/lib/native/Linux-amd64-64 it works when I used the hbase compression test). I built the snappy libraries it in the mrv1 hadoop release tarball using the same method I used for CDH3:

    In $HADOOP_HOME:

    sh cloudera/do-release-build

    That built the libraries, I then dropped the newly made native snappy libraries in the same place as the normal native hadoop libraries and ran the hbase test:

    hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://namenode:8020/titan/tmp snappy

    and it seems to find the native library and load them, but then fails with this:

    12/06/27 00:01:21 INFO hfile.CacheConfig: Allocating LruBlockCache with maximum size 246.8m
    12/06/27 00:01:21 WARN snappy.LoadSnappy: Snappy native library is available
    12/06/27 00:01:21 INFO util.NativeCodeLoader: Loaded the native-hadoop library
    12/06/27 00:01:21 INFO snappy.LoadSnappy: Snappy native library loaded
    12/06/27 00:01:21 WARN snappy.SnappyCompressor: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs()V
    12/06/27 00:01:21 INFO compress.CodecPool: Got brand-new compressor [.snappy]
    12/06/27 00:01:21 DEBUG hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
    Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
    at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native Method)
    at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
    at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:146)
    at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:140)
    at org.apache.hadoop.hbase.io.hfile.Compression$FinishOnFlushCompressionStream.flush(Compression.java:67)
    ....


    Before I built the native libraries it just errored out saying the native libraries for snappy were not available. I saw the notes in snappy-java about having the snappy-java.jar only get loaded in once, so I cleaned it out from all the places it was being loaded in, checking via running:

    hbase classpath

    but to no luck after removing all of the extra occurrences. Any ideas? I'll be poking around this more, hopefully I missed something obvious.

    thanks,
    -chris

    --






    --
    Alejandro

    --




    --
    Alejandro



    --
    Todd Lipcon
    Software Engineer, Cloudera
    --
  • Jian Fang at Jul 21, 2012 at 2:09 am
    I have a similar problem. I try to install CDH4.0.1 tar balls, but the
    native snappy library is not available. I used CDH3 and it comes with the
    native snappy library and it works fine.

    So, do you mean I have to download snappy source code and compile it on
    every node? That does not really sound a good solution to me and I may not
    have the root privilege either.

    Thanks,

    John
    On Wednesday, June 27, 2012 11:27:23 AM UTC-4, Alejandro Abdelnur wrote:

    Hi Chris,

    The tarball are meant for use without native libraries. If you want to use
    native libraries you should install from the corresponding RPM/DEB packages
    for the linux distribution your are using.

    Thanks and best regards.
    On Wed, Jun 27, 2012 at 12:16 AM, Chris Tarnas wrote:

    Hello All,

    We use the tarball hadoop and hbase distributions and are having some
    trouble upgrading to CDH4 from CDH3, specifically around the native snappy
    codec.

    It was not included with the tarball (gz was included and compiled, when
    I moved them to $HADOOP_HOME/lib/native/Linux-amd64-64 it works when I used
    the hbase compression test). I built the snappy libraries it in the mrv1
    hadoop release tarball using the same method I used for CDH3:

    In $HADOOP_HOME:

    sh cloudera/do-release-build

    That built the libraries, I then dropped the newly made native snappy
    libraries in the same place as the normal native hadoop libraries and ran
    the hbase test:

    hbase org.apache.hadoop.hbase.util.CompressionTest
    hdfs://namenode:8020/titan/tmp snappy

    and it seems to find the native library and load them, but then fails
    with this:

    12/06/27 00:01:21 INFO hfile.CacheConfig: Allocating LruBlockCache with
    maximum size 246.8m
    12/06/27 00:01:21 WARN snappy.LoadSnappy: Snappy native library is
    available
    12/06/27 00:01:21 INFO util.NativeCodeLoader: Loaded the native-hadoop
    library
    12/06/27 00:01:21 INFO snappy.LoadSnappy: Snappy native library loaded
    12/06/27 00:01:21 WARN snappy.SnappyCompressor:
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs()V
    12/06/27 00:01:21 INFO compress.CodecPool: Got brand-new compressor
    [.snappy]
    12/06/27 00:01:21 DEBUG hfile.HFileWriterV2: Initialized with
    CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false]
    [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false]
    [cacheEvictOnClose=false] [cacheCompressed=false]
    Exception in thread "main" java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native
    Method)
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:146)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:140)
    at
    org.apache.hadoop.hbase.io.hfile.Compression$FinishOnFlushCompressionStream.flush(Compression.java:67)
    ....


    Before I built the native libraries it just errored out saying the native
    libraries for snappy were not available. I saw the notes in snappy-java
    about having the snappy-java.jar only get loaded in once, so I cleaned it
    out from all the places it was being loaded in, checking via running:

    hbase classpath

    but to no luck after removing all of the extra occurrences. Any ideas?
    I'll be poking around this more, hopefully I missed something obvious.

    thanks,
    -chris

    --



    --
    Alejandro
    --
  • Joey Echeverria at Jul 21, 2012 at 2:13 am
    You can compile it on one node and transfer the compiled artifact to
    all nodes. If you want to avoid that, you can use one of the OS
    packages available.

    -Joey
    On Fri, Jul 20, 2012 at 10:09 PM, Jian Fang wrote:
    I have a similar problem. I try to install CDH4.0.1 tar balls, but the
    native snappy library is not available. I used CDH3 and it comes with the
    native snappy library and it works fine.

    So, do you mean I have to download snappy source code and compile it on
    every node? That does not really sound a good solution to me and I may not
    have the root privilege either.

    Thanks,

    John

    On Wednesday, June 27, 2012 11:27:23 AM UTC-4, Alejandro Abdelnur wrote:

    Hi Chris,

    The tarball are meant for use without native libraries. If you want to use
    native libraries you should install from the corresponding RPM/DEB packages
    for the linux distribution your are using.

    Thanks and best regards.
    On Wed, Jun 27, 2012 at 12:16 AM, Chris Tarnas wrote:

    Hello All,

    We use the tarball hadoop and hbase distributions and are having some
    trouble upgrading to CDH4 from CDH3, specifically around the native snappy
    codec.

    It was not included with the tarball (gz was included and compiled, when
    I moved them to $HADOOP_HOME/lib/native/Linux-amd64-64 it works when I used
    the hbase compression test). I built the snappy libraries it in the mrv1
    hadoop release tarball using the same method I used for CDH3:

    In $HADOOP_HOME:

    sh cloudera/do-release-build

    That built the libraries, I then dropped the newly made native snappy
    libraries in the same place as the normal native hadoop libraries and ran
    the hbase test:

    hbase org.apache.hadoop.hbase.util.CompressionTest
    hdfs://namenode:8020/titan/tmp snappy

    and it seems to find the native library and load them, but then fails
    with this:

    12/06/27 00:01:21 INFO hfile.CacheConfig: Allocating LruBlockCache with
    maximum size 246.8m
    12/06/27 00:01:21 WARN snappy.LoadSnappy: Snappy native library is
    available
    12/06/27 00:01:21 INFO util.NativeCodeLoader: Loaded the native-hadoop
    library
    12/06/27 00:01:21 INFO snappy.LoadSnappy: Snappy native library loaded
    12/06/27 00:01:21 WARN snappy.SnappyCompressor:
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs()V
    12/06/27 00:01:21 INFO compress.CodecPool: Got brand-new compressor
    [.snappy]
    12/06/27 00:01:21 DEBUG hfile.HFileWriterV2: Initialized with
    CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false]
    [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false]
    [cacheEvictOnClose=false] [cacheCompressed=false]
    Exception in thread "main" java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native
    Method)
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:146)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:140)
    at
    org.apache.hadoop.hbase.io.hfile.Compression$FinishOnFlushCompressionStream.flush(Compression.java:67)
    ....


    Before I built the native libraries it just errored out saying the native
    libraries for snappy were not available. I saw the notes in snappy-java
    about having the snappy-java.jar only get loaded in once, so I cleaned it
    out from all the places it was being loaded in, checking via running:

    hbase classpath

    but to no luck after removing all of the extra occurrences. Any ideas?
    I'll be poking around this more, hopefully I missed something obvious.

    thanks,
    -chris

    --



    --
    Alejandro
    --



    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.

    --
  • Jian Fang at Jul 21, 2012 at 2:18 am
    The problem is I don't have the right to install file on system directory
    owned by root. Would it work if I copy the snappy files from CDH3
    Linux-amd-64 directory to CDH4 hadoop/lib/native directory?

    Thanks,

    John
    On Fri, Jul 20, 2012 at 10:13 PM, Joey Echeverria wrote:

    You can compile it on one node and transfer the compiled artifact to
    all nodes. If you want to avoid that, you can use one of the OS
    packages available.

    -Joey
    On Fri, Jul 20, 2012 at 10:09 PM, Jian Fang wrote:
    I have a similar problem. I try to install CDH4.0.1 tar balls, but the
    native snappy library is not available. I used CDH3 and it comes with the
    native snappy library and it works fine.

    So, do you mean I have to download snappy source code and compile it on
    every node? That does not really sound a good solution to me and I may not
    have the root privilege either.

    Thanks,

    John

    On Wednesday, June 27, 2012 11:27:23 AM UTC-4, Alejandro Abdelnur wrote:

    Hi Chris,

    The tarball are meant for use without native libraries. If you want to
    use
    native libraries you should install from the corresponding RPM/DEB
    packages
    for the linux distribution your are using.

    Thanks and best regards.
    On Wed, Jun 27, 2012 at 12:16 AM, Chris Tarnas wrote:

    Hello All,

    We use the tarball hadoop and hbase distributions and are having some
    trouble upgrading to CDH4 from CDH3, specifically around the native
    snappy
    codec.

    It was not included with the tarball (gz was included and compiled,
    when
    I moved them to $HADOOP_HOME/lib/native/Linux-amd64-64 it works when I
    used
    the hbase compression test). I built the snappy libraries it in the
    mrv1
    hadoop release tarball using the same method I used for CDH3:

    In $HADOOP_HOME:

    sh cloudera/do-release-build

    That built the libraries, I then dropped the newly made native snappy
    libraries in the same place as the normal native hadoop libraries and
    ran
    the hbase test:

    hbase org.apache.hadoop.hbase.util.CompressionTest
    hdfs://namenode:8020/titan/tmp snappy

    and it seems to find the native library and load them, but then fails
    with this:

    12/06/27 00:01:21 INFO hfile.CacheConfig: Allocating LruBlockCache with
    maximum size 246.8m
    12/06/27 00:01:21 WARN snappy.LoadSnappy: Snappy native library is
    available
    12/06/27 00:01:21 INFO util.NativeCodeLoader: Loaded the native-hadoop
    library
    12/06/27 00:01:21 INFO snappy.LoadSnappy: Snappy native library loaded
    12/06/27 00:01:21 WARN snappy.SnappyCompressor:
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs()V
    12/06/27 00:01:21 INFO compress.CodecPool: Got brand-new compressor
    [.snappy]
    12/06/27 00:01:21 DEBUG hfile.HFileWriterV2: Initialized with
    CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false]
    [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false]
    [cacheEvictOnClose=false] [cacheCompressed=false]
    Exception in thread "main" java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native
    Method)
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:146)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:140)
    at
    org.apache.hadoop.hbase.io.hfile.Compression$FinishOnFlushCompressionStream.flush(Compression.java:67)
    ....


    Before I built the native libraries it just errored out saying the
    native
    libraries for snappy were not available. I saw the notes in
    snappy-java
    about having the snappy-java.jar only get loaded in once, so I cleaned
    it
    out from all the places it was being loaded in, checking via running:

    hbase classpath

    but to no luck after removing all of the extra occurrences. Any ideas?
    I'll be poking around this more, hopefully I missed something obvious.

    thanks,
    -chris

    --



    --
    Alejandro
    --



    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.

    --


    --
  • Joey Echeverria at Jul 21, 2012 at 2:19 am
    Where is Hadoop installed? How was it installed?

    -Joey
    On Fri, Jul 20, 2012 at 10:18 PM, Jian Fang wrote:
    The problem is I don't have the right to install file on system directory
    owned by root. Would it work if I copy the snappy files from CDH3
    Linux-amd-64 directory to CDH4 hadoop/lib/native directory?

    Thanks,

    John
    On Fri, Jul 20, 2012 at 10:13 PM, Joey Echeverria wrote:

    You can compile it on one node and transfer the compiled artifact to
    all nodes. If you want to avoid that, you can use one of the OS
    packages available.

    -Joey

    On Fri, Jul 20, 2012 at 10:09 PM, Jian Fang <john.jian.fang@gmail.com>
    wrote:
    I have a similar problem. I try to install CDH4.0.1 tar balls, but the
    native snappy library is not available. I used CDH3 and it comes with
    the
    native snappy library and it works fine.

    So, do you mean I have to download snappy source code and compile it on
    every node? That does not really sound a good solution to me and I may
    not
    have the root privilege either.

    Thanks,

    John

    On Wednesday, June 27, 2012 11:27:23 AM UTC-4, Alejandro Abdelnur wrote:

    Hi Chris,

    The tarball are meant for use without native libraries. If you want to
    use
    native libraries you should install from the corresponding RPM/DEB
    packages
    for the linux distribution your are using.

    Thanks and best regards.
    On Wed, Jun 27, 2012 at 12:16 AM, Chris Tarnas wrote:

    Hello All,

    We use the tarball hadoop and hbase distributions and are having some
    trouble upgrading to CDH4 from CDH3, specifically around the native
    snappy
    codec.

    It was not included with the tarball (gz was included and compiled,
    when
    I moved them to $HADOOP_HOME/lib/native/Linux-amd64-64 it works when I
    used
    the hbase compression test). I built the snappy libraries it in the
    mrv1
    hadoop release tarball using the same method I used for CDH3:

    In $HADOOP_HOME:

    sh cloudera/do-release-build

    That built the libraries, I then dropped the newly made native snappy
    libraries in the same place as the normal native hadoop libraries and
    ran
    the hbase test:

    hbase org.apache.hadoop.hbase.util.CompressionTest
    hdfs://namenode:8020/titan/tmp snappy

    and it seems to find the native library and load them, but then fails
    with this:

    12/06/27 00:01:21 INFO hfile.CacheConfig: Allocating LruBlockCache
    with
    maximum size 246.8m
    12/06/27 00:01:21 WARN snappy.LoadSnappy: Snappy native library is
    available
    12/06/27 00:01:21 INFO util.NativeCodeLoader: Loaded the native-hadoop
    library
    12/06/27 00:01:21 INFO snappy.LoadSnappy: Snappy native library loaded
    12/06/27 00:01:21 WARN snappy.SnappyCompressor:
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs()V
    12/06/27 00:01:21 INFO compress.CodecPool: Got brand-new compressor
    [.snappy]
    12/06/27 00:01:21 DEBUG hfile.HFileWriterV2: Initialized with
    CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false]
    [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false]
    [cacheEvictOnClose=false] [cacheCompressed=false]
    Exception in thread "main" java.lang.UnsatisfiedLinkError:

    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
    at

    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native
    Method)
    at

    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
    at

    org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:146)
    at

    org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:140)
    at

    org.apache.hadoop.hbase.io.hfile.Compression$FinishOnFlushCompressionStream.flush(Compression.java:67)
    ....


    Before I built the native libraries it just errored out saying the
    native
    libraries for snappy were not available. I saw the notes in
    snappy-java
    about having the snappy-java.jar only get loaded in once, so I cleaned
    it
    out from all the places it was being loaded in, checking via running:

    hbase classpath

    but to no luck after removing all of the extra occurrences. Any ideas?
    I'll be poking around this more, hopefully I missed something obvious.

    thanks,
    -chris

    --



    --
    Alejandro
    --



    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.

    --

    --



    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.

    --
  • Jian Fang at Jul 21, 2012 at 2:21 am
    I installed it using CDH4.0.1 tarballs to a directory owned by our team.
    On Fri, Jul 20, 2012 at 10:19 PM, Joey Echeverria wrote:

    Where is Hadoop installed? How was it installed?

    -Joey
    On Fri, Jul 20, 2012 at 10:18 PM, Jian Fang wrote:
    The problem is I don't have the right to install file on system directory
    owned by root. Would it work if I copy the snappy files from CDH3
    Linux-amd-64 directory to CDH4 hadoop/lib/native directory?

    Thanks,

    John
    On Fri, Jul 20, 2012 at 10:13 PM, Joey Echeverria wrote:

    You can compile it on one node and transfer the compiled artifact to
    all nodes. If you want to avoid that, you can use one of the OS
    packages available.

    -Joey

    On Fri, Jul 20, 2012 at 10:09 PM, Jian Fang <john.jian.fang@gmail.com>
    wrote:
    I have a similar problem. I try to install CDH4.0.1 tar balls, but the
    native snappy library is not available. I used CDH3 and it comes with
    the
    native snappy library and it works fine.

    So, do you mean I have to download snappy source code and compile it
    on
    every node? That does not really sound a good solution to me and I may
    not
    have the root privilege either.

    Thanks,

    John


    On Wednesday, June 27, 2012 11:27:23 AM UTC-4, Alejandro Abdelnur
    wrote:
    Hi Chris,

    The tarball are meant for use without native libraries. If you want
    to
    use
    native libraries you should install from the corresponding RPM/DEB
    packages
    for the linux distribution your are using.

    Thanks and best regards.
    On Wed, Jun 27, 2012 at 12:16 AM, Chris Tarnas wrote:

    Hello All,

    We use the tarball hadoop and hbase distributions and are having
    some
    trouble upgrading to CDH4 from CDH3, specifically around the native
    snappy
    codec.

    It was not included with the tarball (gz was included and compiled,
    when
    I moved them to $HADOOP_HOME/lib/native/Linux-amd64-64 it works
    when I
    used
    the hbase compression test). I built the snappy libraries it in the
    mrv1
    hadoop release tarball using the same method I used for CDH3:

    In $HADOOP_HOME:

    sh cloudera/do-release-build

    That built the libraries, I then dropped the newly made native
    snappy
    libraries in the same place as the normal native hadoop libraries
    and
    ran
    the hbase test:

    hbase org.apache.hadoop.hbase.util.CompressionTest
    hdfs://namenode:8020/titan/tmp snappy

    and it seems to find the native library and load them, but then
    fails
    with this:

    12/06/27 00:01:21 INFO hfile.CacheConfig: Allocating LruBlockCache
    with
    maximum size 246.8m
    12/06/27 00:01:21 WARN snappy.LoadSnappy: Snappy native library is
    available
    12/06/27 00:01:21 INFO util.NativeCodeLoader: Loaded the
    native-hadoop
    library
    12/06/27 00:01:21 INFO snappy.LoadSnappy: Snappy native library
    loaded
    12/06/27 00:01:21 WARN snappy.SnappyCompressor:
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs()V
    12/06/27 00:01:21 INFO compress.CodecPool: Got brand-new compressor
    [.snappy]
    12/06/27 00:01:21 DEBUG hfile.HFileWriterV2: Initialized with
    CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false]
    [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false]
    [cacheEvictOnClose=false] [cacheCompressed=false]
    Exception in thread "main" java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native
    Method)
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:146)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:140)
    at
    org.apache.hadoop.hbase.io.hfile.Compression$FinishOnFlushCompressionStream.flush(Compression.java:67)
    ....


    Before I built the native libraries it just errored out saying the
    native
    libraries for snappy were not available. I saw the notes in
    snappy-java
    about having the snappy-java.jar only get loaded in once, so I
    cleaned
    it
    out from all the places it was being loaded in, checking via
    running:
    hbase classpath

    but to no luck after removing all of the extra occurrences. Any
    ideas?
    I'll be poking around this more, hopefully I missed something
    obvious.
    thanks,
    -chris

    --



    --
    Alejandro
    --



    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.

    --

    --



    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.

    --


    --
  • Joey Echeverria at Jul 21, 2012 at 11:44 am
    Then you shouldn't need to modify a directory owned by root. Just build the native libs once, and copy into the directory where you put Hadoop.

    -Joey

    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
    On Jul 20, 2012, at 10:21 PM, Jian Fang wrote:

    I installed it using CDH4.0.1 tarballs to a directory owned by our team.

    On Fri, Jul 20, 2012 at 10:19 PM, Joey Echeverria wrote:
    Where is Hadoop installed? How was it installed?

    -Joey
    On Fri, Jul 20, 2012 at 10:18 PM, Jian Fang wrote:
    The problem is I don't have the right to install file on system directory
    owned by root. Would it work if I copy the snappy files from CDH3
    Linux-amd-64 directory to CDH4 hadoop/lib/native directory?

    Thanks,

    John
    On Fri, Jul 20, 2012 at 10:13 PM, Joey Echeverria wrote:

    You can compile it on one node and transfer the compiled artifact to
    all nodes. If you want to avoid that, you can use one of the OS
    packages available.

    -Joey

    On Fri, Jul 20, 2012 at 10:09 PM, Jian Fang <john.jian.fang@gmail.com>
    wrote:
    I have a similar problem. I try to install CDH4.0.1 tar balls, but the
    native snappy library is not available. I used CDH3 and it comes with
    the
    native snappy library and it works fine.

    So, do you mean I have to download snappy source code and compile it on
    every node? That does not really sound a good solution to me and I may
    not
    have the root privilege either.

    Thanks,

    John

    On Wednesday, June 27, 2012 11:27:23 AM UTC-4, Alejandro Abdelnur wrote:

    Hi Chris,

    The tarball are meant for use without native libraries. If you want to
    use
    native libraries you should install from the corresponding RPM/DEB
    packages
    for the linux distribution your are using.

    Thanks and best regards.
    On Wed, Jun 27, 2012 at 12:16 AM, Chris Tarnas wrote:

    Hello All,

    We use the tarball hadoop and hbase distributions and are having some
    trouble upgrading to CDH4 from CDH3, specifically around the native
    snappy
    codec.

    It was not included with the tarball (gz was included and compiled,
    when
    I moved them to $HADOOP_HOME/lib/native/Linux-amd64-64 it works when I
    used
    the hbase compression test). I built the snappy libraries it in the
    mrv1
    hadoop release tarball using the same method I used for CDH3:

    In $HADOOP_HOME:

    sh cloudera/do-release-build

    That built the libraries, I then dropped the newly made native snappy
    libraries in the same place as the normal native hadoop libraries and
    ran
    the hbase test:

    hbase org.apache.hadoop.hbase.util.CompressionTest
    hdfs://namenode:8020/titan/tmp snappy

    and it seems to find the native library and load them, but then fails
    with this:

    12/06/27 00:01:21 INFO hfile.CacheConfig: Allocating LruBlockCache
    with
    maximum size 246.8m
    12/06/27 00:01:21 WARN snappy.LoadSnappy: Snappy native library is
    available
    12/06/27 00:01:21 INFO util.NativeCodeLoader: Loaded the native-hadoop
    library
    12/06/27 00:01:21 INFO snappy.LoadSnappy: Snappy native library loaded
    12/06/27 00:01:21 WARN snappy.SnappyCompressor:
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs()V
    12/06/27 00:01:21 INFO compress.CodecPool: Got brand-new compressor
    [.snappy]
    12/06/27 00:01:21 DEBUG hfile.HFileWriterV2: Initialized with
    CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false]
    [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false]
    [cacheEvictOnClose=false] [cacheCompressed=false]
    Exception in thread "main" java.lang.UnsatisfiedLinkError:

    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
    at

    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native
    Method)
    at

    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
    at

    org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:146)
    at

    org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:140)
    at

    org.apache.hadoop.hbase.io.hfile.Compression$FinishOnFlushCompressionStream.flush(Compression.java:67)
    ....


    Before I built the native libraries it just errored out saying the
    native
    libraries for snappy were not available. I saw the notes in
    snappy-java
    about having the snappy-java.jar only get loaded in once, so I cleaned
    it
    out from all the places it was being loaded in, checking via running:

    hbase classpath

    but to no luck after removing all of the extra occurrences. Any ideas?
    I'll be poking around this more, hopefully I missed something obvious.

    thanks,
    -chris

    --



    --
    Alejandro
    --



    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.

    --

    --



    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.

    --




    --

    --
  • Jian Fang at Jul 21, 2012 at 4:07 pm
    Already did that. I compiled snappy 1.0.5 and copy the library files to
    $HADOOP_HOME/lib/native and $HBASE_HOME/lib/native/Linux_amd64-64/. I set
    my HBase region to use snappy compression, but the hbase region server
    cannot start because of "no snappy library available".

    Any suggestions?

    Thanks,

    John
    On Sat, Jul 21, 2012 at 7:44 AM, Joey Echeverria wrote:

    Then you shouldn't need to modify a directory owned by root. Just build
    the native libs once, and copy into the directory where you put Hadoop.

    -Joey

    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.

    On Jul 20, 2012, at 10:21 PM, Jian Fang wrote:

    I installed it using CDH4.0.1 tarballs to a directory owned by our team.
    On Fri, Jul 20, 2012 at 10:19 PM, Joey Echeverria wrote:

    Where is Hadoop installed? How was it installed?

    -Joey

    On Fri, Jul 20, 2012 at 10:18 PM, Jian Fang <john.jian.fang@gmail.com>
    wrote:
    The problem is I don't have the right to install file on system directory
    owned by root. Would it work if I copy the snappy files from CDH3
    Linux-amd-64 directory to CDH4 hadoop/lib/native directory?

    Thanks,

    John

    On Fri, Jul 20, 2012 at 10:13 PM, Joey Echeverria <joey@cloudera.com>
    wrote:
    You can compile it on one node and transfer the compiled artifact to
    all nodes. If you want to avoid that, you can use one of the OS
    packages available.

    -Joey

    On Fri, Jul 20, 2012 at 10:09 PM, Jian Fang <john.jian.fang@gmail.com>
    wrote:
    I have a similar problem. I try to install CDH4.0.1 tar balls, but
    the
    native snappy library is not available. I used CDH3 and it comes with
    the
    native snappy library and it works fine.

    So, do you mean I have to download snappy source code and compile it
    on
    every node? That does not really sound a good solution to me and I
    may
    not
    have the root privilege either.

    Thanks,

    John


    On Wednesday, June 27, 2012 11:27:23 AM UTC-4, Alejandro Abdelnur
    wrote:
    Hi Chris,

    The tarball are meant for use without native libraries. If you want
    to
    use
    native libraries you should install from the corresponding RPM/DEB
    packages
    for the linux distribution your are using.

    Thanks and best regards.

    On Wed, Jun 27, 2012 at 12:16 AM, Chris Tarnas <cft@email.com>
    wrote:
    Hello All,

    We use the tarball hadoop and hbase distributions and are having
    some
    trouble upgrading to CDH4 from CDH3, specifically around the native
    snappy
    codec.

    It was not included with the tarball (gz was included and compiled,
    when
    I moved them to $HADOOP_HOME/lib/native/Linux-amd64-64 it works
    when I
    used
    the hbase compression test). I built the snappy libraries it in the
    mrv1
    hadoop release tarball using the same method I used for CDH3:

    In $HADOOP_HOME:

    sh cloudera/do-release-build

    That built the libraries, I then dropped the newly made native
    snappy
    libraries in the same place as the normal native hadoop libraries
    and
    ran
    the hbase test:

    hbase org.apache.hadoop.hbase.util.CompressionTest
    hdfs://namenode:8020/titan/tmp snappy

    and it seems to find the native library and load them, but then
    fails
    with this:

    12/06/27 00:01:21 INFO hfile.CacheConfig: Allocating LruBlockCache
    with
    maximum size 246.8m
    12/06/27 00:01:21 WARN snappy.LoadSnappy: Snappy native library is
    available
    12/06/27 00:01:21 INFO util.NativeCodeLoader: Loaded the
    native-hadoop
    library
    12/06/27 00:01:21 INFO snappy.LoadSnappy: Snappy native library
    loaded
    12/06/27 00:01:21 WARN snappy.SnappyCompressor:
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs()V
    12/06/27 00:01:21 INFO compress.CodecPool: Got brand-new compressor
    [.snappy]
    12/06/27 00:01:21 DEBUG hfile.HFileWriterV2: Initialized with
    CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false]
    [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false]
    [cacheEvictOnClose=false] [cacheCompressed=false]
    Exception in thread "main" java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native
    Method)
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:146)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:140)
    at
    org.apache.hadoop.hbase.io.hfile.Compression$FinishOnFlushCompressionStream.flush(Compression.java:67)
    ....


    Before I built the native libraries it just errored out saying the
    native
    libraries for snappy were not available. I saw the notes in
    snappy-java
    about having the snappy-java.jar only get loaded in once, so I
    cleaned
    it
    out from all the places it was being loaded in, checking via
    running:
    hbase classpath

    but to no luck after removing all of the extra occurrences. Any
    ideas?
    I'll be poking around this more, hopefully I missed something
    obvious.
    thanks,
    -chris

    --



    --
    Alejandro
    --



    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.

    --

    --



    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.

    --


    --




    --


    --
  • John Fang at Jul 22, 2012 at 12:34 am
    Could anyone help? This is a road blocker now.

    Thanks in advance,

    John
    On Saturday, July 21, 2012 12:07:52 PM UTC-4, John Fang wrote:

    Already did that. I compiled snappy 1.0.5 and copy the library files to
    $HADOOP_HOME/lib/native and $HBASE_HOME/lib/native/Linux_amd64-64/. I set
    my HBase region to use snappy compression, but the hbase region server
    cannot start because of "no snappy library available".

    Any suggestions?

    Thanks,

    John
    On Sat, Jul 21, 2012 at 7:44 AM, Joey Echeverria wrote:

    Then you shouldn't need to modify a directory owned by root. Just build
    the native libs once, and copy into the directory where you put Hadoop.

    -Joey

    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.

    On Jul 20, 2012, at 10:21 PM, Jian Fang wrote:

    I installed it using CDH4.0.1 tarballs to a directory owned by our team.
    On Fri, Jul 20, 2012 at 10:19 PM, Joey Echeverria wrote:

    Where is Hadoop installed? How was it installed?

    -Joey

    On Fri, Jul 20, 2012 at 10:18 PM, Jian Fang <john.jian.fang@gmail.com>
    wrote:
    The problem is I don't have the right to install file on system directory
    owned by root. Would it work if I copy the snappy files from CDH3
    Linux-amd-64 directory to CDH4 hadoop/lib/native directory?

    Thanks,

    John

    On Fri, Jul 20, 2012 at 10:13 PM, Joey Echeverria <joey@cloudera.com>
    wrote:
    You can compile it on one node and transfer the compiled artifact to
    all nodes. If you want to avoid that, you can use one of the OS
    packages available.

    -Joey

    On Fri, Jul 20, 2012 at 10:09 PM, Jian Fang <john.jian.fang@gmail.com
    wrote:
    I have a similar problem. I try to install CDH4.0.1 tar balls, but
    the
    native snappy library is not available. I used CDH3 and it comes
    with
    the
    native snappy library and it works fine.

    So, do you mean I have to download snappy source code and compile
    it on
    every node? That does not really sound a good solution to me and I
    may
    not
    have the root privilege either.

    Thanks,

    John


    On Wednesday, June 27, 2012 11:27:23 AM UTC-4, Alejandro Abdelnur
    wrote:
    Hi Chris,

    The tarball are meant for use without native libraries. If you
    want to
    use
    native libraries you should install from the corresponding RPM/DEB
    packages
    for the linux distribution your are using.

    Thanks and best regards.

    On Wed, Jun 27, 2012 at 12:16 AM, Chris Tarnas <cft@email.com>
    wrote:
    Hello All,

    We use the tarball hadoop and hbase distributions and are having
    some
    trouble upgrading to CDH4 from CDH3, specifically around the
    native
    snappy
    codec.

    It was not included with the tarball (gz was included and
    compiled,
    when
    I moved them to $HADOOP_HOME/lib/native/Linux-amd64-64 it works
    when I
    used
    the hbase compression test). I built the snappy libraries it in
    the
    mrv1
    hadoop release tarball using the same method I used for CDH3:

    In $HADOOP_HOME:

    sh cloudera/do-release-build

    That built the libraries, I then dropped the newly made native
    snappy
    libraries in the same place as the normal native hadoop libraries
    and
    ran
    the hbase test:

    hbase org.apache.hadoop.hbase.util.CompressionTest
    hdfs://namenode:8020/titan/tmp snappy

    and it seems to find the native library and load them, but then
    fails
    with this:

    12/06/27 00:01:21 INFO hfile.CacheConfig: Allocating LruBlockCache
    with
    maximum size 246.8m
    12/06/27 00:01:21 WARN snappy.LoadSnappy: Snappy native library is
    available
    12/06/27 00:01:21 INFO util.NativeCodeLoader: Loaded the
    native-hadoop
    library
    12/06/27 00:01:21 INFO snappy.LoadSnappy: Snappy native library
    loaded
    12/06/27 00:01:21 WARN snappy.SnappyCompressor:
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs()V
    12/06/27 00:01:21 INFO compress.CodecPool: Got brand-new
    compressor
    [.snappy]
    12/06/27 00:01:21 DEBUG hfile.HFileWriterV2: Initialized with
    CacheConfig:enabled [cacheDataOnRead=true]
    [cacheDataOnWrite=false]
    [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false]
    [cacheEvictOnClose=false] [cacheCompressed=false]
    Exception in thread "main" java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native
    Method)
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:146)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:140)
    at
    org.apache.hadoop.hbase.io.hfile.Compression$FinishOnFlushCompressionStream.flush(Compression.java:67)
    ....


    Before I built the native libraries it just errored out saying the
    native
    libraries for snappy were not available. I saw the notes in
    snappy-java
    about having the snappy-java.jar only get loaded in once, so I
    cleaned
    it
    out from all the places it was being loaded in, checking via
    running:
    hbase classpath

    but to no luck after removing all of the extra occurrences. Any
    ideas?
    I'll be poking around this more, hopefully I missed something
    obvious.
    thanks,
    -chris

    --



    --
    Alejandro
    --



    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.

    --

    --



    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.

    --


    --




    --


    --
  • John Fang at Jul 22, 2012 at 8:27 am
    Seems it is not just snappy problem, hadoop cannot find any native library
    at all.

    $ hdfs dfs -cat /profile/customer/part-r-00000
    12/07/22 04:21:26 WARN util.NativeCodeLoader: Unable to load native-hadoop
    library for your platform... using builtin-java classes where applicable

    I already added LD_LIBRARY_PATH to point to $HADOOP_HOME/lib/native, but I
    still got the same warning.

    I did see library files in $HADOOP_HOME/lib/native, but why hadoop cannot
    find them?

    On Saturday, July 21, 2012 8:34:46 PM UTC-4, John Fang wrote:

    Could anyone help? This is a road blocker now.

    Thanks in advance,

    John
    On Saturday, July 21, 2012 12:07:52 PM UTC-4, John Fang wrote:

    Already did that. I compiled snappy 1.0.5 and copy the library files to
    $HADOOP_HOME/lib/native and $HBASE_HOME/lib/native/Linux_amd64-64/. I set
    my HBase region to use snappy compression, but the hbase region server
    cannot start because of "no snappy library available".

    Any suggestions?

    Thanks,

    John
    On Sat, Jul 21, 2012 at 7:44 AM, Joey Echeverria wrote:

    Then you shouldn't need to modify a directory owned by root. Just build
    the native libs once, and copy into the directory where you put Hadoop.

    -Joey

    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.

    On Jul 20, 2012, at 10:21 PM, Jian Fang <john.jian.fang@gmail.com>
    wrote:

    I installed it using CDH4.0.1 tarballs to a directory owned by our team.
    On Fri, Jul 20, 2012 at 10:19 PM, Joey Echeverria wrote:

    Where is Hadoop installed? How was it installed?

    -Joey

    On Fri, Jul 20, 2012 at 10:18 PM, Jian Fang <john.jian.fang@gmail.com>
    wrote:
    The problem is I don't have the right to install file on system directory
    owned by root. Would it work if I copy the snappy files from CDH3
    Linux-amd-64 directory to CDH4 hadoop/lib/native directory?

    Thanks,

    John

    On Fri, Jul 20, 2012 at 10:13 PM, Joey Echeverria <joey@cloudera.com>
    wrote:
    You can compile it on one node and transfer the compiled artifact to
    all nodes. If you want to avoid that, you can use one of the OS
    packages available.

    -Joey

    On Fri, Jul 20, 2012 at 10:09 PM, Jian Fang <
    john.jian.fang@gmail.com>
    wrote:
    I have a similar problem. I try to install CDH4.0.1 tar balls, but
    the
    native snappy library is not available. I used CDH3 and it comes
    with
    the
    native snappy library and it works fine.

    So, do you mean I have to download snappy source code and compile
    it on
    every node? That does not really sound a good solution to me and I
    may
    not
    have the root privilege either.

    Thanks,

    John


    On Wednesday, June 27, 2012 11:27:23 AM UTC-4, Alejandro Abdelnur
    wrote:
    Hi Chris,

    The tarball are meant for use without native libraries. If you
    want to
    use
    native libraries you should install from the corresponding RPM/DEB
    packages
    for the linux distribution your are using.

    Thanks and best regards.

    On Wed, Jun 27, 2012 at 12:16 AM, Chris Tarnas <cft@email.com>
    wrote:
    Hello All,

    We use the tarball hadoop and hbase distributions and are having
    some
    trouble upgrading to CDH4 from CDH3, specifically around the
    native
    snappy
    codec.

    It was not included with the tarball (gz was included and
    compiled,
    when
    I moved them to $HADOOP_HOME/lib/native/Linux-amd64-64 it works
    when I
    used
    the hbase compression test). I built the snappy libraries it in
    the
    mrv1
    hadoop release tarball using the same method I used for CDH3:

    In $HADOOP_HOME:

    sh cloudera/do-release-build

    That built the libraries, I then dropped the newly made native
    snappy
    libraries in the same place as the normal native hadoop
    libraries and
    ran
    the hbase test:

    hbase org.apache.hadoop.hbase.util.CompressionTest
    hdfs://namenode:8020/titan/tmp snappy

    and it seems to find the native library and load them, but then
    fails
    with this:

    12/06/27 00:01:21 INFO hfile.CacheConfig: Allocating
    LruBlockCache
    with
    maximum size 246.8m
    12/06/27 00:01:21 WARN snappy.LoadSnappy: Snappy native library
    is
    available
    12/06/27 00:01:21 INFO util.NativeCodeLoader: Loaded the
    native-hadoop
    library
    12/06/27 00:01:21 INFO snappy.LoadSnappy: Snappy native library
    loaded
    12/06/27 00:01:21 WARN snappy.SnappyCompressor:
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs()V
    12/06/27 00:01:21 INFO compress.CodecPool: Got brand-new
    compressor
    [.snappy]
    12/06/27 00:01:21 DEBUG hfile.HFileWriterV2: Initialized with
    CacheConfig:enabled [cacheDataOnRead=true]
    [cacheDataOnWrite=false]
    [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false]
    [cacheEvictOnClose=false] [cacheCompressed=false]
    Exception in thread "main" java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect(Native
    Method)
    at
    org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:146)
    at
    org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:140)
    at
    org.apache.hadoop.hbase.io.hfile.Compression$FinishOnFlushCompressionStream.flush(Compression.java:67)
    ....


    Before I built the native libraries it just errored out saying
    the
    native
    libraries for snappy were not available. I saw the notes in
    snappy-java
    about having the snappy-java.jar only get loaded in once, so I
    cleaned
    it
    out from all the places it was being loaded in, checking via
    running:
    hbase classpath

    but to no luck after removing all of the extra occurrences. Any
    ideas?
    I'll be poking around this more, hopefully I missed something
    obvious.
    thanks,
    -chris

    --



    --
    Alejandro
    --



    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.

    --

    --



    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.

    --


    --




    --


    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcdh-user @
categorieshadoop
postedJun 27, '12 at 12:07p
activeJul 22, '12 at 8:27a
posts17
users5
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase