FAQ
I've been steadily adding more and more shared libraries to the -files option of my pipes command and have had moderate success in that each time I add a new library the app no longer fails on that library, but rather on the next one.

However, I've hit a snag. I'm getting the following error even though this file is being added to the distributed cache:

stderr logs
[CRAZY_LONG_PATH]/MyPipesApp: error while loading shared libraries: libboost_system-gcc41-mt-1_37.so.1.37.0: cannot open shared object file: No such file or directory

Despite the fact that the file clearly does exist:

$ hadoop fs -ls /uwphysics/kwiley/mosaic/c++_bin
Found 14 items
...
-rw-r--r-- 1 kwiley uwphysics 12144 2010-04-12 13:38 /uwphysics/kwiley/mosaic/c++_bin/libboost_system-gcc41-mt-1_37.so.1.37.0
...

Here's my run command:

$ hadoop pipes -D hadoop.pipes.java.recordreader=true -D hadoop.pipes.java.recordwriter=true -files \
...[SEVERAL .so FILES TO DISTRIBUTED CACHE]
hdfs://[HDFS SITE]/uwphysics/kwiley/mosaic/c++_bin/libboost_system-gcc41-mt-1_37.so.1.37.0, \
...[SEVERAL .so FILES TO DISTRIBUTED CACHE]
-input /uwphysics/kwiley/mosaic/input -output /uwphysics/kwiley/mosaic/output -program /uwphysics/kwiley/mosaic/c++_bin/MyPipesApp

Note that this is working for a number of other shared libraries all stored in the same directory on HDFS, so it isn't simply a minor path error. It's more serious than that. I'm open to suggestions as to how to go about fixing this.

Thanks.

________________________________________________________________________________
Keith Wiley kwiley@keithwiley.com www.keithwiley.com

"The easy confidence with which I know another man's religion is folly teaches
me to suspect that my own is also."
-- Mark Twain
________________________________________________________________________________

Search Discussions

  • Keith Wiley at Apr 12, 2010 at 9:13 pm
    I love working on a problem for an hour, sending an email for help, then solving it. Problem was the space after the comma in the -files option:

    $ hadoop pipes -D hadoop.pipes.java.recordreader=true -D hadoop.pipes.java.recordwriter=true -files \
    ...[SEVERAL .so FILES TO DISTRIBUTED CACHE]
    hdfs://[HDFS SITE]/uwphysics/kwiley/mosaic/c++_bin/libboost_system-gcc41-mt-1_37.so.1.37.0, \
    ...[SEVERAL .so FILES TO DISTRIBUTED CACHE]
    -input /uwphysics/kwiley/mosaic/input -output /uwphysics/kwiley/mosaic/output -program /uwphysics/kwiley/mosaic/c++_bin/MyPipesApp

    should have been:

    $ hadoop pipes -D hadoop.pipes.java.recordreader=true -D hadoop.pipes.java.recordwriter=true -files \
    ...[SEVERAL .so FILES TO DISTRIBUTED CACHE]
    hdfs://[HDFS SITE]/uwphysics/kwiley/mosaic/c++_bin/libboost_system-gcc41-mt-1_37.so.1.37.0,\ <==== NO SPACE AFTER THE COMMA
    ...[SEVERAL .so FILES TO DISTRIBUTED CACHE]
    -input /uwphysics/kwiley/mosaic/input -output /uwphysics/kwiley/mosaic/output -program /uwphysics/kwiley/mosaic/c++_bin/MyPipesApp

    Makes perfect sense in hindsight. Perhaps others will benefit from the wisdom this thread imparts.

    Cheers!

    ________________________________________________________________________________
    Keith Wiley kwiley@keithwiley.com www.keithwiley.com

    "It's a fine line between meticulous and obsessive-compulsive and a slippery
    rope between obsessive compulsive and debilitatingly slow."
    -- Keith Wiley
    ________________________________________________________________________________

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedApr 12, '10 at 9:09p
activeApr 12, '10 at 9:13p
posts2
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Keith Wiley: 2 posts

People

Translate

site design / logo © 2022 Grokbase