[Hadoop-common-dev] [jira] Created: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Chansler updated HADOOP-2838:
------------------------------------

Fix Version/s: (was: 0.17.0)

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12597525#action_12597525 ]

acmurthy edited comment on HADOOP-2838 at 5/16/08 9:18 AM:
----------------------------------------------------------------

I propose we generalise this feature and let users set _any_ environment variable for their tasks... maybe we should add a mapred.child.env configuration variable which takes in a comma-separated list of environment variables:

{noformat}
<property>
<name>mapred.child.env</name>
<value>LD_LIBRARY_PATH=/foo,PATH=/blah</value>
</property>
{noformat}

Of course we should ensure that each of the env-vars in that list is appended to the env inherited from the parent TT.

Thoughts?

was (Author: acmurthy):
I propose we generalise this feature and let users set _any_ environment variable for their tasks... maybe we should add a mapred.child.env configuration variable which takes in a comma-separated list of environment variables:

{noformat}
<property>
<name>mapred.child.env</name>
<value>LD_LIBRARY_PATH=/foo,PATH=/blah</value>
</property>

Of course we should ensure that each of the env-vars in that list is appended to the env inherited from the parent TT.

Thoughts?

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12597525#action_12597525 ]

Arun C Murthy commented on HADOOP-2838:
---------------------------------------

I propose we generalise this feature and let users set _any_ environment variable for their tasks... maybe we should add a mapred.child.env configuration variable which takes in a comma-separated list of environment variables:

{noformat}
<property>
<name>mapred.child.env</name>
<value>LD_LIBRARY_PATH=/foo,PATH=/blah</value>
</property>

Of course we should ensure that each of the env-vars in that list is appended to the env inherited from the parent TT.

Thoughts?

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641195#action_12641195 ]

Christian Kunz commented on HADOOP-2838:
----------------------------------------

I see a use case for this proposal.

HADOOP-2867 allows to load shared libraries via symbolic links pointing to localized cache files.
This would apply to libhdfs library

On the other hand, libhdfs is part of the Hadoop release and is likely locally available on all the nodes. By being able to set LD_LIBRARY_PATH to some static location, the need to use DistributedCache for dynamically loading libhdfs could be avoided.

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708428#action_12708428 ]

Amar Kamat commented on HADOOP-2838:
------------------------------------

+1 for mapred.child.env which will be a comma separated.

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-2838:
-------------------------------

Attachment: HADOOP-2838-v1.0.patch

Attaching a patch that provides the facility to add/set/append env variables for child processes. Result of test-patch
{code}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}

Todo:
# ant test
# optimize and re-iterate over the patch.

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-2838:
-------------------------------

Component/s: (was: scripts)
mapred
Assignee: Amar Kamat (was: Owen O'Malley)

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-2838:
-------------------------------

Attachment: HADOOP-2838-v1.1.patch

Attaching a new patch with a bug fix. Result of test-patch
{code}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}

Ant tests passed on my box. Currently trying to optimize the patch.

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715477#action_12715477 ]

Sharad Agarwal commented on HADOOP-2838:
----------------------------------------

Adding a new MiniDFSCluster based test for this looks to be an overkill. Instead can we write a test by separating this functionality in a method and just testing that? Alternatively we can add to one of the existing MiniDFSCluster test.

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715563#action_12715563 ]

Todd Lipcon commented on HADOOP-2838:
-------------------------------------

Using a comma separated conf variable here seems slightly dangerous. I imagine there must be some cases where people want to pass commas as part of their environment. I don't have a concrete example, but it seems it's only a matter of time until some streaming application comes along and really needs a comma in its environment, and then we're stuck dealing with escaping, etc.

What about using the entire mapred.child.env namespace, and iterating over it? So mapred.child.env.LD_LIBRARY_PATH would hold the value for that environment variable.

Unfortunately this would either involve iterating over and filtering all of the conf vars, or switching the "Properties" class used for storage inside Configuration to be a SortedMap so we can do a prefix match. I don't particularly like the former, and the latter is a bigger change, but the interface does seem simpler and also more robust. For example, a sysadmin could set a final property of LD_LIBRARY_PATH="path/to/some.so" to provide some kind of security sandboxing while still allowing the user to modify other variables in the environment.

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715785#action_12715785 ]

Amar Kamat commented on HADOOP-2838:
------------------------------------

Todd, I think for now we can keep this simple and get the basic functionality (i.e for paths) working. We can change/add if needed. Thoughts?

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715787#action_12715787 ]

Todd Lipcon commented on HADOOP-2838:
-------------------------------------

Yep, that seems reasonable to me, Amar.

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-2838:
-------------------------------

Attachment: HADOOP-2838-v2.2.patch

Attaching a patch incorporating Sharad's comments. Result of test-patch
{code}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 6 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}

Changes :
# Improved the test case
# Merged TestMiniMRTaskTempDir with env checks.
Running ant tests now.

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716262#action_12716262 ]

Amar Kamat commented on HADOOP-2838:
------------------------------------

Ant tests passed on my box.

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716275#action_12716275 ]

Sharad Agarwal commented on HADOOP-2838:
----------------------------------------

+1 patch looks fine to me.

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal resolved HADOOP-2838.
------------------------------------

Resolution: Fixed
Fix Version/s: 0.21.0
Hadoop Flags: [Reviewed]

I just committed this. Thanks Amar!

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-2838:
-------------------------------

Attachment: HADOOP-2838-v2.2-branch-20-example.patch

Not to be committed to branch 20. Just an example patch.

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716558#action_12716558 ]

Hemanth Yamijala commented on HADOOP-2838:
------------------------------------------

Stumbled on this patch.. what happens if the value of mapred.child.env does not have a well formed string - like doesn't have "key=value" but only key ? Has this been taken care of ?

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716571#action_12716571 ]

Amar Kamat commented on HADOOP-2838:
------------------------------------

bq. Stumbled on this patch.. what happens if the value of mapred.child.env does not have a well formed string - like doesn't have "key=value" but only key ? Has this been taken care of ?
TaskRunner will fail in child launch leading to task failure ultimately leading to job failure. I see it as something similar to bad mapred code.

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716574#action_12716574 ]

Amar Kamat commented on HADOOP-2838:
------------------------------------

I tried that with the testcase and the job fails. Wondering if this requires to be added in the testcase.

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717140#action_12717140 ]

Hemanth Yamijala commented on HADOOP-2838:
------------------------------------------

Will it fail mysteriously or will it be easy to know that the failure is because of the invalid syntax ? If not, I think it would be useful to make it easy to know this is the cause in some way. Maybe through the diagnostic string.

[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-2838:
-------------------------------

Release Note:
Now the users can set environment variables using mapred.child.env. They can do the following
X=Y : set X to Y
X=$X:Y : Append Y to X (which should be taken from the tasktracker)