FAQ
Dear all,

I wrote a plugin codes for Hadoop, which calls the interfaces
in Cpp-built .so library. The plugin codes are written in java,
so I prepared a JNI class to encapsulate the C interfaces.

The java codes can be executed successfully when I compiled
it and run it standalone. However, it does not work when I embedded
in Hadoop. The exception shown out is (found in Hadoop logs):


------------ screen dump ---------------------

# grep myClass logs/* -r
logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
thread "JVM Runner jvm_200904261632_0001_m_-1217897050 spawned."
java.lang.UnsatisfiedLinkError:
org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
logs/hadoop-hadoop-tasktracker-testbed0.container.org.out: at
org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
thread "JVM Runner jvm_200904261632_0001_m_-1887898624 spawned."
java.lang.UnsatisfiedLinkError:
org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
logs/hadoop-hadoop-tasktracker-testbed0.container.org.out: at
org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
...

--------------------------------------------------------

It seems the library can not be loaded in Hadoop. My codes
(myClass.java) is like:


--------------- myClass.java ------------------

public class myClass
{

public static final Log LOG =
LogFactory.getLog("org.apache.hadoop.mapred.myClass");


public myClass() {

try {
//System.setProperty("java.library.path",
"/usr/local/lib");

/* The above line does not work, so I have to
do something
* like following line.
*/
addDir(new String("/usr/local/lib"));
System.loadLibrary("myclass");
}
catch(UnsatisfiedLinkError e) {
LOG.info( "Cannot load library:\n " +
e.toString() );
}
catch(IOException ioe) {
LOG.info( "IO error:\n " +
ioe.toString() );
}

}

/* Since the System.setProperty() does not work, I have to add the following
* function to force the path is added in java.library.path
*/
public static void addDir(String s) throws IOException {

try {
Field field =
ClassLoader.class.getDeclaredField("usr_paths");
field.setAccessible(true);
String[] paths = (String[])field.get(null);
for (int i = 0; i < paths.length; i++) {
if (s.equals(paths[i])) {
return;
}
}
String[] tmp = new String[paths.length+1];
System.arraycopy(paths,0,tmp,0,paths.length);
tmp[paths.length] = s;

field.set(null,tmp);
} catch (IllegalAccessException e) {
throw new IOException("Failed to get
permissions to set library path");
} catch (NoSuchFieldException e) {
throw new IOException("Failed to get field
handle to set library path");
}
}

public native int myClassfsMount(String subsys);
public native int myClassfsUmount(String subsys);


}

--------------------------------------------------------


I don't know what missed in my codes and am wondering whether there are any
rules in Hadoop I should obey if I want to achieve my target.

FYI, the myClassfsMount() and myClassfsUmount() will open a socket to call
services from a daemon. I would better if this design did not cause the fail in
my codes.


Any comments?


Thanks in advance,

Ian

Search Discussions

  • Jason hadoop at Apr 30, 2009 at 3:21 am
    You need to make sure that the shared library is available on the
    tasktracker nodes, either by installing it, or by pushing it around via the
    distributed cache


    On Wed, Apr 29, 2009 at 8:19 PM, Ian jonhson wrote:

    Dear all,

    I wrote a plugin codes for Hadoop, which calls the interfaces
    in Cpp-built .so library. The plugin codes are written in java,
    so I prepared a JNI class to encapsulate the C interfaces.

    The java codes can be executed successfully when I compiled
    it and run it standalone. However, it does not work when I embedded
    in Hadoop. The exception shown out is (found in Hadoop logs):


    ------------ screen dump ---------------------

    # grep myClass logs/* -r
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
    thread "JVM Runner jvm_200904261632_0001_m_-1217897050 spawned."
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out: at
    org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
    thread "JVM Runner jvm_200904261632_0001_m_-1887898624 spawned."
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out: at
    org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
    ...

    --------------------------------------------------------

    It seems the library can not be loaded in Hadoop. My codes
    (myClass.java) is like:


    --------------- myClass.java ------------------

    public class myClass
    {

    public static final Log LOG =
    LogFactory.getLog("org.apache.hadoop.mapred.myClass");


    public myClass() {

    try {
    //System.setProperty("java.library.path",
    "/usr/local/lib");

    /* The above line does not work, so I have to
    do something
    * like following line.
    */
    addDir(new String("/usr/local/lib"));
    System.loadLibrary("myclass");
    }
    catch(UnsatisfiedLinkError e) {
    LOG.info( "Cannot load library:\n " +
    e.toString() );
    }
    catch(IOException ioe) {
    LOG.info( "IO error:\n " +
    ioe.toString() );
    }

    }

    /* Since the System.setProperty() does not work, I have to add the
    following
    * function to force the path is added in java.library.path
    */
    public static void addDir(String s) throws IOException {

    try {
    Field field =
    ClassLoader.class.getDeclaredField("usr_paths");
    field.setAccessible(true);
    String[] paths = (String[])field.get(null);
    for (int i = 0; i < paths.length; i++) {
    if (s.equals(paths[i])) {
    return;
    }
    }
    String[] tmp = new String[paths.length+1];
    System.arraycopy(paths,0,tmp,0,paths.length);
    tmp[paths.length] = s;

    field.set(null,tmp);
    } catch (IllegalAccessException e) {
    throw new IOException("Failed to get
    permissions to set library path");
    } catch (NoSuchFieldException e) {
    throw new IOException("Failed to get field
    handle to set library path");
    }
    }

    public native int myClassfsMount(String subsys);
    public native int myClassfsUmount(String subsys);


    }

    --------------------------------------------------------


    I don't know what missed in my codes and am wondering whether there are any
    rules in Hadoop I should obey if I want to achieve my target.

    FYI, the myClassfsMount() and myClassfsUmount() will open a socket to call
    services from a daemon. I would better if this design did not cause the
    fail in
    my codes.


    Any comments?


    Thanks in advance,

    Ian


    --
    Alpha Chapters of my book on Hadoop are available
    http://www.apress.com/book/view/9781430219422
  • Ian jonhson at Apr 30, 2009 at 8:03 am
    Thanks for answering.

    I run my Hadoop in single node, not cluster mode.


    On Thu, Apr 30, 2009 at 11:21 AM, jason hadoop wrote:
    You need to make sure that the shared library is available on the
    tasktracker nodes, either by installing it, or by pushing it around via the
    distributed cache


    On Wed, Apr 29, 2009 at 8:19 PM, Ian jonhson wrote:

    Dear all,

    I wrote a plugin codes for Hadoop, which calls the interfaces
    in Cpp-built .so library. The plugin codes are written in java,
    so I prepared a JNI class to encapsulate the C interfaces.

    The java codes can be executed successfully when I compiled
    it and run it standalone. However, it does not work when I embedded
    in Hadoop. The exception shown out is (found in Hadoop logs):


    ------------  screen dump  ---------------------

    # grep myClass logs/* -r
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
    thread "JVM Runner jvm_200904261632_0001_m_-1217897050 spawned."
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:      at
    org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
    thread "JVM Runner jvm_200904261632_0001_m_-1887898624 spawned."
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:      at
    org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
    ...

    --------------------------------------------------------

    It seems the library can not be loaded in Hadoop. My codes
    (myClass.java) is like:


    ---------------  myClass.java  ------------------

    public class myClass
    {

    public static final Log LOG =
    LogFactory.getLog("org.apache.hadoop.mapred.myClass");


    public myClass()   {

    try {
    //System.setProperty("java.library.path",
    "/usr/local/lib");

    /* The above line does not work, so I have to
    do something
    * like following line.
    */
    addDir(new String("/usr/local/lib"));
    System.loadLibrary("myclass");
    }
    catch(UnsatisfiedLinkError e) {
    LOG.info( "Cannot load library:\n " +
    e.toString() );
    }
    catch(IOException ioe) {
    LOG.info( "IO error:\n " +
    ioe.toString() );
    }

    }

    /* Since the System.setProperty() does not work, I have to add the
    following
    * function to force the path is added in java.library.path
    */
    public static void addDir(String s) throws IOException {

    try {
    Field field =
    ClassLoader.class.getDeclaredField("usr_paths");
    field.setAccessible(true);
    String[] paths = (String[])field.get(null);
    for (int i = 0; i < paths.length; i++) {
    if (s.equals(paths[i])) {
    return;
    }
    }
    String[] tmp = new String[paths.length+1];
    System.arraycopy(paths,0,tmp,0,paths.length);
    tmp[paths.length] = s;

    field.set(null,tmp);
    } catch (IllegalAccessException e) {
    throw new IOException("Failed to get
    permissions to set library path");
    } catch (NoSuchFieldException e) {
    throw new IOException("Failed to get field
    handle to set library path");
    }
    }

    public native int myClassfsMount(String subsys);
    public native int myClassfsUmount(String subsys);


    }

    --------------------------------------------------------


    I don't know what missed in my codes and am wondering whether there are any
    rules in Hadoop I should obey if I want to  achieve my target.

    FYI, the myClassfsMount() and myClassfsUmount() will open a socket to call
    services from a daemon. I would better if this design did not cause the
    fail in
    my codes.


    Any comments?


    Thanks in advance,

    Ian


    --
    Alpha Chapters of my book on Hadoop are available
    http://www.apress.com/book/view/9781430219422
  • He Yongqiang at Apr 30, 2009 at 8:43 am
    Does hadoop now support jni calls in Mappers or Reducers? If yes, how? If
    not, I think we should create a jira issue for supporting that.

    On 09-4-30 下午4:02, "Ian jonhson" wrote:

    Thanks for answering.

    I run my Hadoop in single node, not cluster mode.


    On Thu, Apr 30, 2009 at 11:21 AM, jason hadoop wrote:
    You need to make sure that the shared library is available on the
    tasktracker nodes, either by installing it, or by pushing it around via the
    distributed cache


    On Wed, Apr 29, 2009 at 8:19 PM, Ian jonhson wrote:

    Dear all,

    I wrote a plugin codes for Hadoop, which calls the interfaces
    in Cpp-built .so library. The plugin codes are written in java,
    so I prepared a JNI class to encapsulate the C interfaces.

    The java codes can be executed successfully when I compiled
    it and run it standalone. However, it does not work when I embedded
    in Hadoop. The exception shown out is (found in Hadoop logs):


    ------------ screen dump ---------------------

    # grep myClass logs/* -r
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
    thread "JVM Runner jvm_200904261632_0001_m_-1217897050 spawned."
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out: at
    org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
    thread "JVM Runner jvm_200904261632_0001_m_-1887898624 spawned."
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out: at
    org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
    ...

    --------------------------------------------------------

    It seems the library can not be loaded in Hadoop. My codes
    (myClass.java) is like:


    --------------- myClass.java ------------------

    public class myClass
    {

    public static final Log LOG =
    LogFactory.getLog("org.apache.hadoop.mapred.myClass");


    public myClass() {

    try {
    //System.setProperty("java.library.path",
    "/usr/local/lib");

    /* The above line does not work, so I have to
    do something
    * like following line.
    */
    addDir(new String("/usr/local/lib"));
    System.loadLibrary("myclass");
    }
    catch(UnsatisfiedLinkError e) {
    LOG.info( "Cannot load library:\n " +
    e.toString() );
    }
    catch(IOException ioe) {
    LOG.info( "IO error:\n " +
    ioe.toString() );
    }

    }

    /* Since the System.setProperty() does not work, I have to add the
    following
    * function to force the path is added in java.library.path
    */
    public static void addDir(String s) throws IOException {

    try {
    Field field =
    ClassLoader.class.getDeclaredField("usr_paths");
    field.setAccessible(true);
    String[] paths = (String[])field.get(null);
    for (int i = 0; i < paths.length; i++) {
    if (s.equals(paths[i])) {
    return;
    }
    }
    String[] tmp = new String[paths.length+1];
    System.arraycopy(paths,0,tmp,0,paths.length);
    tmp[paths.length] = s;

    field.set(null,tmp);
    } catch (IllegalAccessException e) {
    throw new IOException("Failed to get
    permissions to set library path");
    } catch (NoSuchFieldException e) {
    throw new IOException("Failed to get field
    handle to set library path");
    }
    }

    public native int myClassfsMount(String subsys);
    public native int myClassfsUmount(String subsys);


    }

    --------------------------------------------------------


    I don't know what missed in my codes and am wondering whether there are any
    rules in Hadoop I should obey if I want to achieve my target.

    FYI, the myClassfsMount() and myClassfsUmount() will open a socket to call
    services from a daemon. I would better if this design did not cause the
    fail in
    my codes.


    Any comments?


    Thanks in advance,

    Ian


    --
    Alpha Chapters of my book on Hadoop are available
    http://www.apress.com/book/view/9781430219422
  • Ian jonhson at Apr 30, 2009 at 9:15 am
    You mean that the current hadoop does not support JNI calls, right?
    Are there any solution to achieve the calls from C interfaces?

    2009/4/30 He Yongqiang <heyongqiang@software.ict.ac.cn>:
    Does hadoop now support jni calls in Mappers or Reducers? If yes, how? If
    not, I think we should create a jira issue for supporting that.

    On 09-4-30 下午4:02, "Ian jonhson" wrote:

    Thanks for answering.

    I run my Hadoop in single node, not cluster mode.


    On Thu, Apr 30, 2009 at 11:21 AM, jason hadoop wrote:
    You need to make sure that the shared library is available on the
    tasktracker nodes, either by installing it, or by pushing it around via the
    distributed cache


    On Wed, Apr 29, 2009 at 8:19 PM, Ian jonhson wrote:

    Dear all,

    I wrote a plugin codes for Hadoop, which calls the interfaces
    in Cpp-built .so library. The plugin codes are written in java,
    so I prepared a JNI class to encapsulate the C interfaces.

    The java codes can be executed successfully when I compiled
    it and run it standalone. However, it does not work when I embedded
    in Hadoop. The exception shown out is (found in Hadoop logs):


    ------------ screen dump ---------------------

    # grep myClass logs/* -r
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
    thread "JVM Runner jvm_200904261632_0001_m_-1217897050 spawned."
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out: at
    org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
    thread "JVM Runner jvm_200904261632_0001_m_-1887898624 spawned."
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out: at
    org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
    ...

    --------------------------------------------------------

    It seems the library can not be loaded in Hadoop. My codes
    (myClass.java) is like:


    --------------- myClass.java ------------------

    public class myClass
    {

    public static final Log LOG =
    LogFactory.getLog("org.apache.hadoop.mapred.myClass");


    public myClass() {

    try {
    //System.setProperty("java.library.path",
    "/usr/local/lib");

    /* The above line does not work, so I have to
    do something
    * like following line.
    */
    addDir(new String("/usr/local/lib"));
    System.loadLibrary("myclass");
    }
    catch(UnsatisfiedLinkError e) {
    LOG.info( "Cannot load library:\n " +
    e.toString() );
    }
    catch(IOException ioe) {
    LOG.info( "IO error:\n " +
    ioe.toString() );
    }

    }

    /* Since the System.setProperty() does not work, I have to add the
    following
    * function to force the path is added in java.library.path
    */
    public static void addDir(String s) throws IOException {

    try {
    Field field =
    ClassLoader.class.getDeclaredField("usr_paths");
    field.setAccessible(true);
    String[] paths = (String[])field.get(null);
    for (int i = 0; i < paths.length; i++) {
    if (s.equals(paths[i])) {
    return;
    }
    }
    String[] tmp = new String[paths.length+1];
    System.arraycopy(paths,0,tmp,0,paths.length);
    tmp[paths.length] = s;

    field.set(null,tmp);
    } catch (IllegalAccessException e) {
    throw new IOException("Failed to get
    permissions to set library path");
    } catch (NoSuchFieldException e) {
    throw new IOException("Failed to get field
    handle to set library path");
    }
    }

    public native int myClassfsMount(String subsys);
    public native int myClassfsUmount(String subsys);


    }

    --------------------------------------------------------


    I don't know what missed in my codes and am wondering whether there are any
    rules in Hadoop I should obey if I want to achieve my target.

    FYI, the myClassfsMount() and myClassfsUmount() will open a socket to call
    services from a daemon. I would better if this design did not cause the
    fail in
    my codes.


    Any comments?


    Thanks in advance,

    Ian


    --
    Alpha Chapters of my book on Hadoop are available
    http://www.apress.com/book/view/9781430219422
  • He Yongqiang at Apr 30, 2009 at 9:50 am
    put your .so file in every traker's Hadoop-install/lib/native/Linux-xxx-xx/

    Or

    In your code,try to do

    String oldPath=System.getProperty("java.library.path");
    System.setProperty("java.library.path", oldPath==null?
    local_path_of_lib_file:oldPath+pathSeparator +local_path_of_lib_file))
    System.loadLibrary("XXX");

    However, you also need to fetch the library to local through
    DistributedCache( like jason said) or putting and getting it from hdfs by
    yourself.
    On 09-4-30 下午5:14, "Ian jonhson" wrote:

    You mean that the current hadoop does not support JNI calls, right?
    Are there any solution to achieve the calls from C interfaces?

    2009/4/30 He Yongqiang <heyongqiang@software.ict.ac.cn>:
    Does hadoop now support jni calls in Mappers or Reducers? If yes, how? If
    not, I think we should create a jira issue for supporting that.

    On 09-4-30 下午4:02, "Ian jonhson" wrote:

    Thanks for answering.

    I run my Hadoop in single node, not cluster mode.



    On Thu, Apr 30, 2009 at 11:21 AM, jason hadoop <jason.hadoop@gmail.com>
    wrote:
    You need to make sure that the shared library is available on the
    tasktracker nodes, either by installing it, or by pushing it around via the
    distributed cache


    On Wed, Apr 29, 2009 at 8:19 PM, Ian jonhson wrote:

    Dear all,

    I wrote a plugin codes for Hadoop, which calls the interfaces
    in Cpp-built .so library. The plugin codes are written in java,
    so I prepared a JNI class to encapsulate the C interfaces.

    The java codes can be executed successfully when I compiled
    it and run it standalone. However, it does not work when I embedded
    in Hadoop. The exception shown out is (found in Hadoop logs):


    ------------ screen dump ---------------------

    # grep myClass logs/* -r
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
    thread "JVM Runner jvm_200904261632_0001_m_-1217897050 spawned."
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out: at
    org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
    thread "JVM Runner jvm_200904261632_0001_m_-1887898624 spawned."
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out: at
    org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
    ...

    --------------------------------------------------------

    It seems the library can not be loaded in Hadoop. My codes
    (myClass.java) is like:


    --------------- myClass.java ------------------

    public class myClass
    {

    public static final Log LOG =
    LogFactory.getLog("org.apache.hadoop.mapred.myClass");


    public myClass() {

    try {
    //System.setProperty("java.library.path",
    "/usr/local/lib");

    /* The above line does not work, so I have to
    do something
    * like following line.
    */
    addDir(new String("/usr/local/lib"));
    System.loadLibrary("myclass");
    }
    catch(UnsatisfiedLinkError e) {
    LOG.info( "Cannot load library:\n " +
    e.toString() );
    }
    catch(IOException ioe) {
    LOG.info( "IO error:\n " +
    ioe.toString() );
    }

    }

    /* Since the System.setProperty() does not work, I have to add the
    following
    * function to force the path is added in java.library.path
    */
    public static void addDir(String s) throws IOException {

    try {
    Field field =
    ClassLoader.class.getDeclaredField("usr_paths");
    field.setAccessible(true);
    String[] paths = (String[])field.get(null);
    for (int i = 0; i < paths.length; i++) {
    if (s.equals(paths[i])) {
    return;
    }
    }
    String[] tmp = new String[paths.length+1];
    System.arraycopy(paths,0,tmp,0,paths.length);
    tmp[paths.length] = s;

    field.set(null,tmp);
    } catch (IllegalAccessException e) {
    throw new IOException("Failed to get
    permissions to set library path");
    } catch (NoSuchFieldException e) {
    throw new IOException("Failed to get field
    handle to set library path");
    }
    }

    public native int myClassfsMount(String subsys);
    public native int myClassfsUmount(String subsys);


    }

    --------------------------------------------------------


    I don't know what missed in my codes and am wondering whether there are
    any
    rules in Hadoop I should obey if I want to achieve my target.

    FYI, the myClassfsMount() and myClassfsUmount() will open a socket to call
    services from a daemon. I would better if this design did not cause the
    fail in
    my codes.


    Any comments?


    Thanks in advance,

    Ian


    --
    Alpha Chapters of my book on Hadoop are available
    http://www.apress.com/book/view/9781430219422
  • Ian jonhson at Apr 30, 2009 at 11:46 am

    2009/4/30 He Yongqiang <heyongqiang@software.ict.ac.cn>:
    put your .so file in every traker's Hadoop-install/lib/native/Linux-xxx-xx/

    Or

    In your code,try to do

    String oldPath=System.getProperty("java.library.path");
    System.setProperty("java.library.path", oldPath==null?
    local_path_of_lib_file:oldPath+pathSeparator +local_path_of_lib_file))
    System.loadLibrary("XXX");

    I have copied .so and .a files to Hadoop-install/lib/native/Linux-xxx-xx/
    and called System.loadLibrary("XXX"); in my codes, but nothing happens.

    Then, I tried the second solution mentioned above, same problem is
    occurred (the .so files have been in native directory).


    However, you also need to fetch the library to local through
    DistributedCache( like jason said) or putting and getting it from hdfs by
    yourself.
    Does I need to copy libraries in local machine since I run the Hadoop in
    single node?

    How can I do it either by fetching or putting from hdfs?

    On 09-4-30 下午5:14, "Ian jonhson" wrote:

    You mean that the current hadoop does not support JNI calls, right?
    Are there any solution to achieve the calls from C interfaces?

    2009/4/30 He Yongqiang <heyongqiang@software.ict.ac.cn>:
    Does hadoop now support jni calls in Mappers or Reducers? If yes, how? If
    not, I think we should create a jira issue for supporting that.
  • Rakhi Khatwani at Apr 30, 2009 at 9:44 am
    Hi Jason,
    when will the full version of your book be available??
    On Thu, Apr 30, 2009 at 8:51 AM, jason hadoop wrote:

    You need to make sure that the shared library is available on the
    tasktracker nodes, either by installing it, or by pushing it around via the
    distributed cache


    On Wed, Apr 29, 2009 at 8:19 PM, Ian jonhson wrote:

    Dear all,

    I wrote a plugin codes for Hadoop, which calls the interfaces
    in Cpp-built .so library. The plugin codes are written in java,
    so I prepared a JNI class to encapsulate the C interfaces.

    The java codes can be executed successfully when I compiled
    it and run it standalone. However, it does not work when I embedded
    in Hadoop. The exception shown out is (found in Hadoop logs):


    ------------ screen dump ---------------------

    # grep myClass logs/* -r
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
    thread "JVM Runner jvm_200904261632_0001_m_-1217897050 spawned."
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out: at
    org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
    thread "JVM Runner jvm_200904261632_0001_m_-1887898624 spawned."
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out: at
    org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
    ...

    --------------------------------------------------------

    It seems the library can not be loaded in Hadoop. My codes
    (myClass.java) is like:


    --------------- myClass.java ------------------

    public class myClass
    {

    public static final Log LOG =
    LogFactory.getLog("org.apache.hadoop.mapred.myClass");


    public myClass() {

    try {
    //System.setProperty("java.library.path",
    "/usr/local/lib");

    /* The above line does not work, so I have to
    do something
    * like following line.
    */
    addDir(new String("/usr/local/lib"));
    System.loadLibrary("myclass");
    }
    catch(UnsatisfiedLinkError e) {
    LOG.info( "Cannot load library:\n " +
    e.toString() );
    }
    catch(IOException ioe) {
    LOG.info( "IO error:\n " +
    ioe.toString() );
    }

    }

    /* Since the System.setProperty() does not work, I have to add the
    following
    * function to force the path is added in java.library.path
    */
    public static void addDir(String s) throws IOException {

    try {
    Field field =
    ClassLoader.class.getDeclaredField("usr_paths");
    field.setAccessible(true);
    String[] paths = (String[])field.get(null);
    for (int i = 0; i < paths.length; i++) {
    if (s.equals(paths[i])) {
    return;
    }
    }
    String[] tmp = new String[paths.length+1];
    System.arraycopy(paths,0,tmp,0,paths.length);
    tmp[paths.length] = s;

    field.set(null,tmp);
    } catch (IllegalAccessException e) {
    throw new IOException("Failed to get
    permissions to set library path");
    } catch (NoSuchFieldException e) {
    throw new IOException("Failed to get field
    handle to set library path");
    }
    }

    public native int myClassfsMount(String subsys);
    public native int myClassfsUmount(String subsys);


    }

    --------------------------------------------------------


    I don't know what missed in my codes and am wondering whether there are any
    rules in Hadoop I should obey if I want to achieve my target.

    FYI, the myClassfsMount() and myClassfsUmount() will open a socket to call
    services from a daemon. I would better if this design did not cause the
    fail in
    my codes.


    Any comments?


    Thanks in advance,

    Ian


    --
    Alpha Chapters of my book on Hadoop are available
    http://www.apress.com/book/view/9781430219422
  • Jason hadoop at May 1, 2009 at 2:39 am
    I believe it is early to mid june for the paper version.
    I think all the mainline chapters are available as alphas right now, and
    pretty close to the final versions.
    On Thu, Apr 30, 2009 at 2:43 AM, Rakhi Khatwani wrote:

    Hi Jason,
    when will the full version of your book be available??

    On Thu, Apr 30, 2009 at 8:51 AM, jason hadoop <jason.hadoop@gmail.com
    wrote:
    You need to make sure that the shared library is available on the
    tasktracker nodes, either by installing it, or by pushing it around via the
    distributed cache



    On Wed, Apr 29, 2009 at 8:19 PM, Ian jonhson <jonhson.ian@gmail.com>
    wrote:
    Dear all,

    I wrote a plugin codes for Hadoop, which calls the interfaces
    in Cpp-built .so library. The plugin codes are written in java,
    so I prepared a JNI class to encapsulate the C interfaces.

    The java codes can be executed successfully when I compiled
    it and run it standalone. However, it does not work when I embedded
    in Hadoop. The exception shown out is (found in Hadoop logs):


    ------------ screen dump ---------------------

    # grep myClass logs/* -r
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
    thread "JVM Runner jvm_200904261632_0001_m_-1217897050 spawned."
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out: at
    org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out:Exception in
    thread "JVM Runner jvm_200904261632_0001_m_-1887898624 spawned."
    java.lang.UnsatisfiedLinkError:
    org.apache.hadoop.mapred.myClass.myClassfsMount(Ljava/lang/String;)I
    logs/hadoop-hadoop-tasktracker-testbed0.container.org.out: at
    org.apache.hadoop.mapred.myClass.myClassfsMount(Native Method)
    ...

    --------------------------------------------------------

    It seems the library can not be loaded in Hadoop. My codes
    (myClass.java) is like:


    --------------- myClass.java ------------------

    public class myClass
    {

    public static final Log LOG =
    LogFactory.getLog("org.apache.hadoop.mapred.myClass");

    public myClass() {

    try {
    //System.setProperty("java.library.path",
    "/usr/local/lib");

    /* The above line does not work, so I have to
    do something
    * like following line.
    */
    addDir(new String("/usr/local/lib"));
    System.loadLibrary("myclass");
    }
    catch(UnsatisfiedLinkError e) {
    LOG.info( "Cannot load library:\n " +
    e.toString() );
    }
    catch(IOException ioe) {
    LOG.info( "IO error:\n " +
    ioe.toString() );
    }

    }

    /* Since the System.setProperty() does not work, I have to add
    the
    following
    * function to force the path is added in java.library.path
    */
    public static void addDir(String s) throws IOException {

    try {
    Field field =
    ClassLoader.class.getDeclaredField("usr_paths");
    field.setAccessible(true);
    String[] paths = (String[])field.get(null);
    for (int i = 0; i < paths.length; i++) {
    if (s.equals(paths[i])) {
    return;
    }
    }
    String[] tmp = new String[paths.length+1];
    System.arraycopy(paths,0,tmp,0,paths.length);
    tmp[paths.length] = s;

    field.set(null,tmp);
    } catch (IllegalAccessException e) {
    throw new IOException("Failed to get
    permissions to set library path");
    } catch (NoSuchFieldException e) {
    throw new IOException("Failed to get field
    handle to set library path");
    }
    }

    public native int myClassfsMount(String subsys);
    public native int myClassfsUmount(String subsys);


    }

    --------------------------------------------------------


    I don't know what missed in my codes and am wondering whether there are any
    rules in Hadoop I should obey if I want to achieve my target.

    FYI, the myClassfsMount() and myClassfsUmount() will open a socket to call
    services from a daemon. I would better if this design did not cause the
    fail in
    my codes.


    Any comments?


    Thanks in advance,

    Ian


    --
    Alpha Chapters of my book on Hadoop are available
    http://www.apress.com/book/view/9781430219422


    --
    Alpha Chapters of my book on Hadoop are available
    http://www.apress.com/book/view/9781430219422

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedApr 30, '09 at 3:19a
activeMay 1, '09 at 2:39a
posts9
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase