FAQ
Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
------------------------------------------------------------------------------------------

Key: HADOOP-2838
URL: https://issues.apache.org/jira/browse/HADOOP-2838
Project: Hadoop Core
Issue Type: Improvement
Components: scripts
Affects Versions: 0.16.0
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Fix For: 0.17.0


Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Robert Chansler (JIRA) at Mar 25, 2008 at 3:06 am
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Robert Chansler updated HADOOP-2838:
    ------------------------------------

    Fix Version/s: (was: 0.17.0)
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: scripts
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Owen O'Malley

    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Arun C Murthy (JIRA) at May 16, 2008 at 4:20 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12597525#action_12597525 ]

    acmurthy edited comment on HADOOP-2838 at 5/16/08 9:18 AM:
    ----------------------------------------------------------------

    I propose we generalise this feature and let users set _any_ environment variable for their tasks... maybe we should add a mapred.child.env configuration variable which takes in a comma-separated list of environment variables:

    {noformat}
    <property>
    <name>mapred.child.env</name>
    <value>LD_LIBRARY_PATH=/foo,PATH=/blah</value>
    </property>
    {noformat}

    Of course we should ensure that each of the env-vars in that list is appended to the env inherited from the parent TT.

    Thoughts?

    was (Author: acmurthy):
    I propose we generalise this feature and let users set _any_ environment variable for their tasks... maybe we should add a mapred.child.env configuration variable which takes in a comma-separated list of environment variables:

    {noformat}
    <property>
    <name>mapred.child.env</name>
    <value>LD_LIBRARY_PATH=/foo,PATH=/blah</value>
    </property>

    Of course we should ensure that each of the env-vars in that list is appended to the env inherited from the parent TT.

    Thoughts?
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: scripts
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Owen O'Malley

    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Arun C Murthy (JIRA) at May 16, 2008 at 4:20 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12597525#action_12597525 ]

    Arun C Murthy commented on HADOOP-2838:
    ---------------------------------------

    I propose we generalise this feature and let users set _any_ environment variable for their tasks... maybe we should add a mapred.child.env configuration variable which takes in a comma-separated list of environment variables:

    {noformat}
    <property>
    <name>mapred.child.env</name>
    <value>LD_LIBRARY_PATH=/foo,PATH=/blah</value>
    </property>

    Of course we should ensure that each of the env-vars in that list is appended to the env inherited from the parent TT.

    Thoughts?
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: scripts
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Owen O'Malley

    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Christian Kunz (JIRA) at Oct 20, 2008 at 10:47 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641195#action_12641195 ]

    Christian Kunz commented on HADOOP-2838:
    ----------------------------------------

    I see a use case for this proposal.

    HADOOP-2867 allows to load shared libraries via symbolic links pointing to localized cache files.
    This would apply to libhdfs library

    On the other hand, libhdfs is part of the Hadoop release and is likely locally available on all the nodes. By being able to set LD_LIBRARY_PATH to some static location, the need to use DistributedCache for dynamically loading libhdfs could be avoided.
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: scripts
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Owen O'Malley

    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at May 12, 2009 at 1:16 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708428#action_12708428 ]

    Amar Kamat commented on HADOOP-2838:
    ------------------------------------

    +1 for mapred.child.env which will be a comma separated.
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: scripts
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Owen O'Malley

    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 1, 2009 at 9:20 am
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Amar Kamat updated HADOOP-2838:
    -------------------------------

    Attachment: HADOOP-2838-v1.0.patch

    Attaching a patch that provides the facility to add/set/append env variables for child processes. Result of test-patch
    {code}
    [exec] +1 overall.
    [exec]
    [exec] +1 @author. The patch does not contain any @author tags.
    [exec]
    [exec] +1 tests included. The patch appears to include 3 new or modified tests.
    [exec]
    [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
    [exec]
    [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
    [exec]
    [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
    [exec]
    [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
    [exec]
    [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
    {code}

    Todo:
    # ant test
    # optimize and re-iterate over the patch.
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: scripts
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Owen O'Malley
    Attachments: HADOOP-2838-v1.0.patch


    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 1, 2009 at 9:28 am
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Amar Kamat updated HADOOP-2838:
    -------------------------------

    Component/s: (was: scripts)
    mapred
    Assignee: Amar Kamat (was: Owen O'Malley)
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: mapred
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Amar Kamat
    Attachments: HADOOP-2838-v1.0.patch


    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 1, 2009 at 11:48 am
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Amar Kamat updated HADOOP-2838:
    -------------------------------

    Attachment: HADOOP-2838-v1.1.patch

    Attaching a new patch with a bug fix. Result of test-patch
    {code}
    [exec] +1 overall.
    [exec]
    [exec] +1 @author. The patch does not contain any @author tags.
    [exec]
    [exec] +1 tests included. The patch appears to include 3 new or modified tests.
    [exec]
    [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
    [exec]
    [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
    [exec]
    [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
    [exec]
    [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
    [exec]
    [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
    {code}

    Ant tests passed on my box. Currently trying to optimize the patch.
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: mapred
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Amar Kamat
    Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch


    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Sharad Agarwal (JIRA) at Jun 2, 2009 at 11:05 am
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715477#action_12715477 ]

    Sharad Agarwal commented on HADOOP-2838:
    ----------------------------------------

    Adding a new MiniDFSCluster based test for this looks to be an overkill. Instead can we write a test by separating this functionality in a method and just testing that? Alternatively we can add to one of the existing MiniDFSCluster test.
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: mapred
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Amar Kamat
    Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch


    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Todd Lipcon (JIRA) at Jun 2, 2009 at 3:57 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715563#action_12715563 ]

    Todd Lipcon commented on HADOOP-2838:
    -------------------------------------

    Using a comma separated conf variable here seems slightly dangerous. I imagine there must be some cases where people want to pass commas as part of their environment. I don't have a concrete example, but it seems it's only a matter of time until some streaming application comes along and really needs a comma in its environment, and then we're stuck dealing with escaping, etc.

    What about using the entire mapred.child.env namespace, and iterating over it? So mapred.child.env.LD_LIBRARY_PATH would hold the value for that environment variable.

    Unfortunately this would either involve iterating over and filtering all of the conf vars, or switching the "Properties" class used for storage inside Configuration to be a SortedMap so we can do a prefix match. I don't particularly like the former, and the latter is a bigger change, but the interface does seem simpler and also more robust. For example, a sysadmin could set a final property of LD_LIBRARY_PATH="path/to/some.so" to provide some kind of security sandboxing while still allowing the user to modify other variables in the environment.
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: mapred
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Amar Kamat
    Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch


    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 3, 2009 at 3:16 am
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715785#action_12715785 ]

    Amar Kamat commented on HADOOP-2838:
    ------------------------------------

    Todd, I think for now we can keep this simple and get the basic functionality (i.e for paths) working. We can change/add if needed. Thoughts?
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: mapred
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Amar Kamat
    Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch


    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Todd Lipcon (JIRA) at Jun 3, 2009 at 3:28 am
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715787#action_12715787 ]

    Todd Lipcon commented on HADOOP-2838:
    -------------------------------------

    Yep, that seems reasonable to me, Amar.
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: mapred
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Amar Kamat
    Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch


    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 4, 2009 at 10:21 am
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Amar Kamat updated HADOOP-2838:
    -------------------------------

    Attachment: HADOOP-2838-v2.2.patch

    Attaching a patch incorporating Sharad's comments. Result of test-patch
    {code}
    [exec] +1 overall.
    [exec]
    [exec] +1 @author. The patch does not contain any @author tags.
    [exec]
    [exec] +1 tests included. The patch appears to include 6 new or modified tests.
    [exec]
    [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
    [exec]
    [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
    [exec]
    [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
    [exec]
    [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
    [exec]
    [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
    {code}

    Changes :
    # Improved the test case
    # Merged TestMiniMRTaskTempDir with env checks.
    Running ant tests now.
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: mapred
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Amar Kamat
    Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2.patch


    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 4, 2009 at 12:45 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716262#action_12716262 ]

    Amar Kamat commented on HADOOP-2838:
    ------------------------------------

    Ant tests passed on my box.
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: mapred
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Amar Kamat
    Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2.patch


    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Sharad Agarwal (JIRA) at Jun 4, 2009 at 1:41 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716275#action_12716275 ]

    Sharad Agarwal commented on HADOOP-2838:
    ----------------------------------------

    +1 patch looks fine to me.
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: mapred
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Amar Kamat
    Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2.patch


    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Sharad Agarwal (JIRA) at Jun 4, 2009 at 2:07 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Sharad Agarwal resolved HADOOP-2838.
    ------------------------------------

    Resolution: Fixed
    Fix Version/s: 0.21.0
    Hadoop Flags: [Reviewed]

    I just committed this. Thanks Amar!
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: mapred
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2.patch


    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 4, 2009 at 5:03 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Amar Kamat updated HADOOP-2838:
    -------------------------------

    Attachment: HADOOP-2838-v2.2-branch-20-example.patch

    Not to be committed to branch 20. Just an example patch.
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: mapred
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2-branch-20-example.patch, HADOOP-2838-v2.2.patch


    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hemanth Yamijala (JIRA) at Jun 5, 2009 at 10:02 am
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716558#action_12716558 ]

    Hemanth Yamijala commented on HADOOP-2838:
    ------------------------------------------

    Stumbled on this patch.. what happens if the value of mapred.child.env does not have a well formed string - like doesn't have "key=value" but only key ? Has this been taken care of ?
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: mapred
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2-branch-20-example.patch, HADOOP-2838-v2.2.patch


    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 5, 2009 at 10:36 am
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716571#action_12716571 ]

    Amar Kamat commented on HADOOP-2838:
    ------------------------------------

    bq. Stumbled on this patch.. what happens if the value of mapred.child.env does not have a well formed string - like doesn't have "key=value" but only key ? Has this been taken care of ?
    TaskRunner will fail in child launch leading to task failure ultimately leading to job failure. I see it as something similar to bad mapred code.
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: mapred
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2-branch-20-example.patch, HADOOP-2838-v2.2.patch


    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 5, 2009 at 10:46 am
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716574#action_12716574 ]

    Amar Kamat commented on HADOOP-2838:
    ------------------------------------

    I tried that with the testcase and the job fails. Wondering if this requires to be added in the testcase.
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: mapred
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2-branch-20-example.patch, HADOOP-2838-v2.2.patch


    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hemanth Yamijala (JIRA) at Jun 8, 2009 at 4:24 am
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717140#action_12717140 ]

    Hemanth Yamijala commented on HADOOP-2838:
    ------------------------------------------

    Will it fail mysteriously or will it be easy to know that the failure is because of the invalid syntax ? If not, I think it would be useful to make it easy to know this is the cause in some way. Maybe through the diagnostic string.
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: mapred
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2-branch-20-example.patch, HADOOP-2838-v2.2.patch


    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 10, 2009 at 6:21 am
    [ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Amar Kamat updated HADOOP-2838:
    -------------------------------

    Release Note:
    Now the users can set environment variables using mapred.child.env. They can do the following
    X=Y : set X to Y
    X=$X:Y : Append Y to X (which should be taken from the tasktracker)
    Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
    ------------------------------------------------------------------------------------------

    Key: HADOOP-2838
    URL: https://issues.apache.org/jira/browse/HADOOP-2838
    Project: Hadoop Core
    Issue Type: Improvement
    Components: mapred
    Affects Versions: 0.16.0
    Reporter: Owen O'Malley
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2-branch-20-example.patch, HADOOP-2838-v2.2.patch


    Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedFeb 15, '08 at 6:02p
activeJun 10, '09 at 6:21a
posts23
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Amar Kamat (JIRA): 23 posts

People

Translate

site design / logo © 2022 Grokbase