FAQ
Can anybody point me how to use JNI calls in a map reduce program. My .so
files have other dependencies also , is there a way to load the
LD_LIBRARY_PATH for child processes . Should all the native stuff be in
HDFS?

Thanks,
Utkarsh.

Search Discussions

  • Alex Kozlov at Feb 12, 2010 at 6:02 pm
    All native libraries should be on each of the cluster nodes. You need to
    set "java.library.path" property to point to your libraries (or just put
    them in the default system dirs).
    On Fri, Feb 12, 2010 at 9:12 AM, Utkarsh Agarwal wrote:

    Can anybody point me how to use JNI calls in a map reduce program. My .so
    files have other dependencies also , is there a way to load the
    LD_LIBRARY_PATH for child processes . Should all the native stuff be in
    HDFS?

    Thanks,
    Utkarsh.
  • Allen Wittenauer at Feb 12, 2010 at 6:45 pm
    ... or just use distributed cache.

    On 2/12/10 10:02 AM, "Alex Kozlov" wrote:

    All native libraries should be on each of the cluster nodes. You need to
    set "java.library.path" property to point to your libraries (or just put
    them in the default system dirs).

    On Fri, Feb 12, 2010 at 9:12 AM, Utkarsh Agarwal
    wrote:
    Can anybody point me how to use JNI calls in a map reduce program. My .so
    files have other dependencies also , is there a way to load the
    LD_LIBRARY_PATH for child processes . Should all the native stuff be in
    HDFS?

    Thanks,
    Utkarsh.
  • Jason Rutherglen at Feb 17, 2010 at 1:30 am
    How would this work?

    On Fri, Feb 12, 2010 at 10:45 AM, Allen Wittenauer
    wrote:
    ... or just use distributed cache.

    On 2/12/10 10:02 AM, "Alex Kozlov" wrote:

    All native libraries should be on each of the cluster nodes.  You need to
    set "java.library.path" property to point to your libraries (or just put
    them in the default system dirs).

    On Fri, Feb 12, 2010 at 9:12 AM, Utkarsh Agarwal
    wrote:
    Can anybody point me how to use JNI calls in a map reduce program. My .so
    files have other dependencies also , is there a way to load the
    LD_LIBRARY_PATH for child processes . Should all the native stuff be in
    HDFS?

    Thanks,
    Utkarsh.
  • Allen Wittenauer at Feb 18, 2010 at 10:15 pm
    Like this:

    http://hadoop.apache.org/common/docs/current/native_libraries.html#Loading+n
    ative+libraries+through+DistributedCache


    On 2/16/10 5:29 PM, "Jason Rutherglen" wrote:

    How would this work?

    On Fri, Feb 12, 2010 at 10:45 AM, Allen Wittenauer
    wrote:
    ... or just use distributed cache.

    On 2/12/10 10:02 AM, "Alex Kozlov" wrote:

    All native libraries should be on each of the cluster nodes.  You need to
    set "java.library.path" property to point to your libraries (or just put
    them in the default system dirs).

    On Fri, Feb 12, 2010 at 9:12 AM, Utkarsh Agarwal
    wrote:
    Can anybody point me how to use JNI calls in a map reduce program. My .so
    files have other dependencies also , is there a way to load the
    LD_LIBRARY_PATH for child processes . Should all the native stuff be in
    HDFS?

    Thanks,
    Utkarsh.
  • Utkarsh Agarwal at Feb 19, 2010 at 2:50 am
    My .so file has other .so dependencies , so would I have to add them all in
    the DistributedCache . Also I tried setting LD_LIBRARY_PATH in
    mapred-site.xml as

    <property>
    <name>mapred.child.env</name>
    <value>LD_LIBRARY_PATH=/opt/libs/</value>
    </property>


    doesnt work. the java.library.path is not sufficient to set , have to get
    LD_LIB set.

    -Utkarsh

    On Thu, Feb 18, 2010 at 3:14 PM, Allen Wittenauer
    wrote:

    Like this:


    http://hadoop.apache.org/common/docs/current/native_libraries.html#Loading+n
    ative+libraries+through+DistributedCache


    On 2/16/10 5:29 PM, "Jason Rutherglen" wrote:

    How would this work?

    On Fri, Feb 12, 2010 at 10:45 AM, Allen Wittenauer
    wrote:
    ... or just use distributed cache.

    On 2/12/10 10:02 AM, "Alex Kozlov" wrote:

    All native libraries should be on each of the cluster nodes. You need
    to
    set "java.library.path" property to point to your libraries (or just
    put
    them in the default system dirs).

    On Fri, Feb 12, 2010 at 9:12 AM, Utkarsh Agarwal
    wrote:
    Can anybody point me how to use JNI calls in a map reduce program. My
    .so
    files have other dependencies also , is there a way to load the
    LD_LIBRARY_PATH for child processes . Should all the native stuff be
    in
    HDFS?

    Thanks,
    Utkarsh.
  • Jason Venner at Feb 19, 2010 at 5:04 am
    We used do this all the time at attributor. Now if I can remember how we did
    it.

    If the libraries are constant you can just install them on your nodes to
    save pushing them through the distributed cache, and then setup the
    LD_LIBRARY_PATH correctly.

    The key issue if you push them through the distributed cache is ensuring
    that the directory that the library gets dropped in, is actually in the
    runtime java.library.path
    You can also give explicit paths to System.load

    The -Djava.library.path in the child.options mapred.child.java.opts (if I
    have the param correct) should work also.
    On Thu, Feb 18, 2010 at 6:49 PM, Utkarsh Agarwal wrote:

    My .so file has other .so dependencies , so would I have to add them all in
    the DistributedCache . Also I tried setting LD_LIBRARY_PATH in
    mapred-site.xml as

    <property>
    <name>mapred.child.env</name>
    <value>LD_LIBRARY_PATH=/opt/libs/</value>
    </property>


    doesnt work. the java.library.path is not sufficient to set , have to get
    LD_LIB set.

    -Utkarsh

    On Thu, Feb 18, 2010 at 3:14 PM, Allen Wittenauer
    wrote:

    Like this:


    http://hadoop.apache.org/common/docs/current/native_libraries.html#Loading+n
    ative+libraries+through+DistributedCache


    On 2/16/10 5:29 PM, "Jason Rutherglen" wrote:

    How would this work?

    On Fri, Feb 12, 2010 at 10:45 AM, Allen Wittenauer
    wrote:
    ... or just use distributed cache.

    On 2/12/10 10:02 AM, "Alex Kozlov" wrote:

    All native libraries should be on each of the cluster nodes. You
    need
    to
    set "java.library.path" property to point to your libraries (or just
    put
    them in the default system dirs).

    On Fri, Feb 12, 2010 at 9:12 AM, Utkarsh Agarwal
    wrote:
    Can anybody point me how to use JNI calls in a map reduce program.
    My
    .so
    files have other dependencies also , is there a way to load the
    LD_LIBRARY_PATH for child processes . Should all the native stuff be
    in
    HDFS?

    Thanks,
    Utkarsh.


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals
  • Utkarsh Agarwal at Feb 19, 2010 at 4:48 pm
    How to set the LD_LIBRARY_PATH for the child , configuring mapred-site.xml
    doesn't work. Also setting -Djava.library.path is not good enough since it
    only gets the reference to the lib I am a trying to load(let's say lib.so) ,
    but that lib has dependencies on other libs like lib1.so resulting in
    UnsatisfiedLinkError . Thus, LD_LIBRARY_PATH has to be set.


    On Thu, Feb 18, 2010 at 10:03 PM, Jason Venner wrote:

    We used do this all the time at attributor. Now if I can remember how we
    did
    it.

    If the libraries are constant you can just install them on your nodes to
    save pushing them through the distributed cache, and then setup the
    LD_LIBRARY_PATH correctly.

    The key issue if you push them through the distributed cache is ensuring
    that the directory that the library gets dropped in, is actually in the
    runtime java.library.path
    You can also give explicit paths to System.load

    The -Djava.library.path in the child.options mapred.child.java.opts (if I
    have the param correct) should work also.

    On Thu, Feb 18, 2010 at 6:49 PM, Utkarsh Agarwal <unrealutkarsh@gmail.com
    wrote:
    My .so file has other .so dependencies , so would I have to add them all in
    the DistributedCache . Also I tried setting LD_LIBRARY_PATH in
    mapred-site.xml as

    <property>
    <name>mapred.child.env</name>
    <value>LD_LIBRARY_PATH=/opt/libs/</value>
    </property>


    doesnt work. the java.library.path is not sufficient to set , have to get
    LD_LIB set.

    -Utkarsh

    On Thu, Feb 18, 2010 at 3:14 PM, Allen Wittenauer
    wrote:

    Like this:

    http://hadoop.apache.org/common/docs/current/native_libraries.html#Loading+n
    ative+libraries+through+DistributedCache



    On 2/16/10 5:29 PM, "Jason Rutherglen" <jason.rutherglen@gmail.com>
    wrote:
    How would this work?

    On Fri, Feb 12, 2010 at 10:45 AM, Allen Wittenauer
    wrote:
    ... or just use distributed cache.

    On 2/12/10 10:02 AM, "Alex Kozlov" wrote:

    All native libraries should be on each of the cluster nodes. You
    need
    to
    set "java.library.path" property to point to your libraries (or
    just
    put
    them in the default system dirs).

    On Fri, Feb 12, 2010 at 9:12 AM, Utkarsh Agarwal
    wrote:
    Can anybody point me how to use JNI calls in a map reduce program.
    My
    .so
    files have other dependencies also , is there a way to load the
    LD_LIBRARY_PATH for child processes . Should all the native stuff
    be
    in
    HDFS?

    Thanks,
    Utkarsh.


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals
  • Allen Wittenauer at Feb 19, 2010 at 5:58 pm
    See http://issues.apache.org/jira/browse/HADOOP-2867 (and
    https://issues.apache.org/jira/browse/HADOOP-5980 if you are using 0.21 or
    Y! Hadoop w/LinuxTaskController). What version of Hadoop are you using?

    Also, if this is custom code, what does the runtime link path look like ( -R
    during compile time)? Using $ORIGIN might be useful here.

    On 2/19/10 8:47 AM, "Utkarsh Agarwal" wrote:

    How to set the LD_LIBRARY_PATH for the child , configuring mapred-site.xml
    doesn't work. Also setting -Djava.library.path is not good enough since it
    only gets the reference to the lib I am a trying to load(let's say lib.so) ,
    but that lib has dependencies on other libs like lib1.so resulting in
    UnsatisfiedLinkError . Thus, LD_LIBRARY_PATH has to be set.


    On Thu, Feb 18, 2010 at 10:03 PM, Jason Venner wrote:

    We used do this all the time at attributor. Now if I can remember how we
    did
    it.

    If the libraries are constant you can just install them on your nodes to
    save pushing them through the distributed cache, and then setup the
    LD_LIBRARY_PATH correctly.

    The key issue if you push them through the distributed cache is ensuring
    that the directory that the library gets dropped in, is actually in the
    runtime java.library.path
    You can also give explicit paths to System.load

    The -Djava.library.path in the child.options mapred.child.java.opts (if I
    have the param correct) should work also.

    On Thu, Feb 18, 2010 at 6:49 PM, Utkarsh Agarwal <unrealutkarsh@gmail.com
    wrote:
    My .so file has other .so dependencies , so would I have to add them all in
    the DistributedCache . Also I tried setting LD_LIBRARY_PATH in
    mapred-site.xml as

    <property>
    <name>mapred.child.env</name>
    <value>LD_LIBRARY_PATH=/opt/libs/</value>
    </property>


    doesnt work. the java.library.path is not sufficient to set , have to get
    LD_LIB set.

    -Utkarsh

    On Thu, Feb 18, 2010 at 3:14 PM, Allen Wittenauer
    wrote:

    Like this:

    http://hadoop.apache.org/common/docs/current/native_libraries.html#Loading+n
    ative+libraries+through+DistributedCache



    On 2/16/10 5:29 PM, "Jason Rutherglen" <jason.rutherglen@gmail.com>
    wrote:
    How would this work?

    On Fri, Feb 12, 2010 at 10:45 AM, Allen Wittenauer
    wrote:
    ... or just use distributed cache.

    On 2/12/10 10:02 AM, "Alex Kozlov" wrote:

    All native libraries should be on each of the cluster nodes. You
    need
    to
    set "java.library.path" property to point to your libraries (or
    just
    put
    them in the default system dirs).

    On Fri, Feb 12, 2010 at 9:12 AM, Utkarsh Agarwal
    wrote:
    Can anybody point me how to use JNI calls in a map reduce program.
    My
    .so
    files have other dependencies also , is there a way to load the
    LD_LIBRARY_PATH for child processes . Should all the native stuff
    be
    in
    HDFS?

    Thanks,
    Utkarsh.


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals
  • Utkarsh Agarwal at Feb 19, 2010 at 6:24 pm
    I am using hadoop 0.20.1 , I added the attached patch still child processes
    don't get the path :(
    On Fri, Feb 19, 2010 at 10:57 AM, Allen Wittenauer wrote:


    See http://issues.apache.org/jira/browse/HADOOP-2867 (and
    https://issues.apache.org/jira/browse/HADOOP-5980 if you are using 0.21 or
    Y! Hadoop w/LinuxTaskController). What version of Hadoop are you using?

    Also, if this is custom code, what does the runtime link path look like (
    -R
    during compile time)? Using $ORIGIN might be useful here.

    On 2/19/10 8:47 AM, "Utkarsh Agarwal" wrote:

    How to set the LD_LIBRARY_PATH for the child , configuring
    mapred-site.xml
    doesn't work. Also setting -Djava.library.path is not good enough since it
    only gets the reference to the lib I am a trying to load(let's say lib.so) ,
    but that lib has dependencies on other libs like lib1.so resulting in
    UnsatisfiedLinkError . Thus, LD_LIBRARY_PATH has to be set.



    On Thu, Feb 18, 2010 at 10:03 PM, Jason Venner <jason.hadoop@gmail.com
    wrote:
    We used do this all the time at attributor. Now if I can remember how we
    did
    it.

    If the libraries are constant you can just install them on your nodes to
    save pushing them through the distributed cache, and then setup the
    LD_LIBRARY_PATH correctly.

    The key issue if you push them through the distributed cache is ensuring
    that the directory that the library gets dropped in, is actually in the
    runtime java.library.path
    You can also give explicit paths to System.load

    The -Djava.library.path in the child.options mapred.child.java.opts (if
    I
    have the param correct) should work also.

    On Thu, Feb 18, 2010 at 6:49 PM, Utkarsh Agarwal <
    unrealutkarsh@gmail.com
    wrote:
    My .so file has other .so dependencies , so would I have to add them
    all
    in
    the DistributedCache . Also I tried setting LD_LIBRARY_PATH in
    mapred-site.xml as

    <property>
    <name>mapred.child.env</name>
    <value>LD_LIBRARY_PATH=/opt/libs/</value>
    </property>


    doesnt work. the java.library.path is not sufficient to set , have to
    get
    LD_LIB set.

    -Utkarsh

    On Thu, Feb 18, 2010 at 3:14 PM, Allen Wittenauer
    wrote:

    Like this:

    http://hadoop.apache.org/common/docs/current/native_libraries.html#Loading+n
    ative+libraries+through+DistributedCache



    On 2/16/10 5:29 PM, "Jason Rutherglen" <jason.rutherglen@gmail.com>
    wrote:
    How would this work?

    On Fri, Feb 12, 2010 at 10:45 AM, Allen Wittenauer
    wrote:
    ... or just use distributed cache.

    On 2/12/10 10:02 AM, "Alex Kozlov" wrote:

    All native libraries should be on each of the cluster nodes. You
    need
    to
    set "java.library.path" property to point to your libraries (or
    just
    put
    them in the default system dirs).

    On Fri, Feb 12, 2010 at 9:12 AM, Utkarsh Agarwal
    wrote:
    Can anybody point me how to use JNI calls in a map reduce program.
    My
    .so
    files have other dependencies also , is there a way to load the
    LD_LIBRARY_PATH for child processes . Should all the native stuff
    be
    in
    HDFS?

    Thanks,
    Utkarsh.


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedFeb 12, '10 at 5:12p
activeFeb 19, '10 at 6:24p
posts10
users5
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase