FAQ
Hi,

I am a new user to HDFS and am trying to following the instruction on:
http://wiki.apache.org/hadoop/MountableHDFS

But fuse-dfs fails to load and spits out the following errors:
fuse_dfs_wrapper.sh dfs://xsn95:19001 /hdfs -d
port=19001,server=xsn95
fuse-dfs didn't recognize /hdfs,-2
fuse-dfs ignoring option -d
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
INIT: 7.8
flags=0x00000003
max_readahead=0x00080000
INIT: 7.8
flags=0x00000001
max_readahead=0x00080000
max_write=0x00020000
unique: 1, error: 0 (Success), outsize: 40
unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 40
Call to JNI_CreateJavaVM option 1 = -Xmx2048m
Calling JNI_CreateJavaVM
There was an error trying to initialize the HPI library.
Call to JNI_CreateJavaVM failed with error: -1
unique: 2, error: -5 (Input/output error), outsize: 16



I've added libnsl.so and libhpi.so to the LD_LIBRARY_PATH and still it won't
load. The error code is from:
Hadoop-0.20.2/src/c++/libhdfs/hdfsJniHelper.c

In getJNIEnv function:

//Create the VM

JavaVMInitArgs vm_args;

JavaVM *vm;

vm_args.version = JNI_VERSION_1_4;

vm_args.options = options;

vm_args.nOptions = noArgs;

vm_args.ignoreUnrecognized = 1;



fprintf(stderr, "Calling JNI_CreateJavaVM\n");

rv = JNI_CreateJavaVM(&vm, (void*)&env, &vm_args);

if (rv != 0) {

fprintf(stderr, "Call to JNI_CreateJavaVM failed "

"with error: %d\n", rv);

UNLOCK_JVM_MUTEX();

return NULL;

}



Anybody out there in HADOOP community can help me out please?

Thanks in advance.


--
View this message in context: http://hadoop-common.472056.n3.nabble.com/Anybody-can-help-with-MountableHDFS-fuse-dfs-please-tp2652043p2652043.html
Sent from the Users mailing list archive at Nabble.com.

Search Discussions

  • Brian Bockelman at Mar 8, 2011 at 6:54 pm
    Hi,

    Sounds like an issue with your Hadoop runtime environment.

    What does "ldd </path/to/libhdfs.so>" say? What happens if you try one of the libhdfs test applications?

    Brian
    On Mar 8, 2011, at 12:50 PM, yxxtdc wrote:

    Hi,

    I am a new user to HDFS and am trying to following the instruction on:
    http://wiki.apache.org/hadoop/MountableHDFS

    But fuse-dfs fails to load and spits out the following errors:
    fuse_dfs_wrapper.sh dfs://xsn95:19001 /hdfs -d
    port=19001,server=xsn95
    fuse-dfs didn't recognize /hdfs,-2
    fuse-dfs ignoring option -d
    unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
    INIT: 7.8
    flags=0x00000003
    max_readahead=0x00080000
    INIT: 7.8
    flags=0x00000001
    max_readahead=0x00080000
    max_write=0x00020000
    unique: 1, error: 0 (Success), outsize: 40
    unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 40
    Call to JNI_CreateJavaVM option 1 = -Xmx2048m
    Calling JNI_CreateJavaVM
    There was an error trying to initialize the HPI library.
    Call to JNI_CreateJavaVM failed with error: -1
    unique: 2, error: -5 (Input/output error), outsize: 16



    I've added libnsl.so and libhpi.so to the LD_LIBRARY_PATH and still it won't
    load. The error code is from:
    Hadoop-0.20.2/src/c++/libhdfs/hdfsJniHelper.c

    In getJNIEnv function:

    //Create the VM

    JavaVMInitArgs vm_args;

    JavaVM *vm;

    vm_args.version = JNI_VERSION_1_4;

    vm_args.options = options;

    vm_args.nOptions = noArgs;

    vm_args.ignoreUnrecognized = 1;



    fprintf(stderr, "Calling JNI_CreateJavaVM\n");

    rv = JNI_CreateJavaVM(&vm, (void*)&env, &vm_args);

    if (rv != 0) {

    fprintf(stderr, "Call to JNI_CreateJavaVM failed "

    "with error: %d\n", rv);

    UNLOCK_JVM_MUTEX();

    return NULL;

    }



    Anybody out there in HADOOP community can help me out please?

    Thanks in advance.


    --
    View this message in context: http://hadoop-common.472056.n3.nabble.com/Anybody-can-help-with-MountableHDFS-fuse-dfs-please-tp2652043p2652043.html
    Sent from the Users mailing list archive at Nabble.com.
  • Yxxtdc at Mar 8, 2011 at 7:05 pm
    Hi,

    xsn95:/ # ldd /root/build_src/hadoop-0.20.2/lib/libhdfs.so.0
    linux-vdso.so.1 => (0x00007ffff413c000)
    libjvm.so => /root/build_src/hadoop-0.20.2/lib/libjvm.so
    (0x00002adbd5442000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00002adbd5f62000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00002adbd6066000)
    libc.so.6 => /lib64/libc.so.6 (0x00002adbd617f000)
    libm.so.6 => /lib64/libm.so.6 (0x00002adbd63c0000)
    /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)


    It does not look like hdfs_test is built along with libhdfs.so. Is there a
    compile flag I need to turn on with ant to compile hdfs_test executable?
    Sorry, I am totally new at this and still learning. Any help would be deeply
    appreciated.

    Thanks,

    Yang


    --
    View this message in context: http://hadoop-common.472056.n3.nabble.com/Anybody-can-help-with-MountableHDFS-fuse-dfs-please-tp2652043p2652121.html
    Sent from the Users mailing list archive at Nabble.com.
  • Brian Bockelman at Mar 8, 2011 at 7:13 pm
    On Mar 8, 2011, at 1:04 PM, yxxtdc wrote:

    Hi,

    xsn95:/ # ldd /root/build_src/hadoop-0.20.2/lib/libhdfs.so.0
    linux-vdso.so.1 => (0x00007ffff413c000)
    libjvm.so => /root/build_src/hadoop-0.20.2/lib/libjvm.so
    (0x00002adbd5442000)
    This is likely problematic. Why is libjvm.so here, instead of the normal place inside the JVM install? I'm wondering if maybe the JVM is looking for libhpi.so relative to your libjvm.
    libdl.so.2 => /lib64/libdl.so.2 (0x00002adbd5f62000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00002adbd6066000)
    libc.so.6 => /lib64/libc.so.6 (0x00002adbd617f000)
    libm.so.6 => /lib64/libm.so.6 (0x00002adbd63c0000)
    /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)


    It does not look like hdfs_test is built along with libhdfs.so. Is there a
    compile flag I need to turn on with ant to compile hdfs_test executable?
    Sorry, I am totally new at this and still learning. Any help would be deeply
    appreciated.

    Thanks,

    Yang


    --
    View this message in context: http://hadoop-common.472056.n3.nabble.com/Anybody-can-help-with-MountableHDFS-fuse-dfs-please-tp2652043p2652121.html
    Sent from the Users mailing list archive at Nabble.com.
  • Yxxtdc at Mar 8, 2011 at 7:38 pm
    I copied the libjvm.so from
    /usr/java/jdk1.6.0_24/jre/lib/amd64/server/libjvm.so.
    I added /usr/java/jdk1.6.0_24/jre/lib/amd64/server to the LD_LIBRARY_PATH
    and it did not work so I made a manual copy to $HADOOP_HOME/lib directory.

    I googled my HPI error someone suggested to use the client version of the
    libjvm.so but jdk 6 only came with the lib/amd64/server/libjvm.so.


    Thanks,

    Yang


    --
    View this message in context: http://hadoop-common.472056.n3.nabble.com/Anybody-can-help-with-MountableHDFS-fuse-dfs-please-tp2652043p2652276.html
    Sent from the Users mailing list archive at Nabble.com.
  • Brian Bockelman at Mar 8, 2011 at 7:42 pm
    I would avoid making a manual copy of libjvm.so - see if you can figure out why adding the JDK copy didn't work. For reference, here's what we use:

    if [ "$LD_LIBRARY_PATH" = "" ]
    then JVM_LIB=`find ${JAVA_HOME}/jre/lib -name libjvm.so |tail -n 1`
    export LD_LIBRARY_PATH=`dirname $JVM_LIB`:/usr/lib/

    fi

    Brian
    On Mar 8, 2011, at 1:38 PM, yxxtdc wrote:

    I copied the libjvm.so from
    /usr/java/jdk1.6.0_24/jre/lib/amd64/server/libjvm.so.
    I added /usr/java/jdk1.6.0_24/jre/lib/amd64/server to the LD_LIBRARY_PATH
    and it did not work so I made a manual copy to $HADOOP_HOME/lib directory.

    I googled my HPI error someone suggested to use the client version of the
    libjvm.so but jdk 6 only came with the lib/amd64/server/libjvm.so.


    Thanks,

    Yang


    --
    View this message in context: http://hadoop-common.472056.n3.nabble.com/Anybody-can-help-with-MountableHDFS-fuse-dfs-please-tp2652043p2652276.html
    Sent from the Users mailing list archive at Nabble.com.
  • Yxxtdc at Mar 8, 2011 at 9:42 pm
    Thanks Brian.

    Got over that by fixing a typo in LD_LIBRARY_PATH and now fuse_dfs is
    mounting albeit very very very slowly, like one inode a minute.

    Yang


    --
    View this message in context: http://hadoop-common.472056.n3.nabble.com/Anybody-can-help-with-MountableHDFS-fuse-dfs-please-tp2652043p2652761.html
    Sent from the Users mailing list archive at Nabble.com.
  • Brian Bockelman at Mar 8, 2011 at 10:10 pm

    On Mar 8, 2011, at 3:41 PM, yxxtdc wrote:

    Thanks Brian.

    Got over that by fixing a typo in LD_LIBRARY_PATH and now fuse_dfs is
    mounting albeit very very very slowly, like one inode a minute.
    What do you mean by "mounting"? Do you mean "listing"?

    Anything on the order of a minute is out of the ordinary. See if you can remove some of the "moving pieces" and try to replicate with the stock "hadoop" client.

    FUSE, especially on Hadoop 0.20, is not a rocketship when it comes to metadata operations. However, it's not horrid. See below for example.

    Brian

    [bbockelm@t3-sl5 ~]$ time ls -1 /mnt/hadoop/dropfiles | wc -l
    22

    real 0m0.135s
    user 0m0.001s
    sys 0m0.003s
    [bbockelm@t3-sl5 ~]$ time ls -1 /tmp | wc -l
    65

    real 0m0.002s
    user 0m0.000s
    sys 0m0.002s

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMar 8, '11 at 6:50p
activeMar 8, '11 at 10:10p
posts8
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Yxxtdc: 4 posts Brian Bockelman: 4 posts

People

Translate

site design / logo © 2022 Grokbase