[Hadoop-common-dev] [jira] Created: (HADOOP-5981) HADOOP-2838 doesnt work as expected

[ https://issues.apache.org/jira/browse/HADOOP-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-5981:
-------------------------------

Attachment: HADOOP-5981-v1.1.patch
HADOOP-5981-v1.1-branch-20.patch

Attaching a patch [HADOOP-5981-v1.1.patch] that fixes the issue. The issue got masked because of the improper merge of the two testcases. Tested this patch to see if it works. The testcase now passes. Running test-patch now.

Note that [HADOOP-5981-v1.1-branch-20.patch] is an example patch not to be committed to branch 20.

[ https://issues.apache.org/jira/browse/HADOOP-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716602#action_12716602 ]

Amar Kamat commented on HADOOP-5981:
------------------------------------

Result of test-patch
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

[ https://issues.apache.org/jira/browse/HADOOP-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716614#action_12716614 ]

Sreekanth Ramakrishnan commented on HADOOP-5981:
------------------------------------------------

The changes look fine to me.

[ https://issues.apache.org/jira/browse/HADOOP-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716654#action_12716654 ]

Amar Kamat commented on HADOOP-5981:
------------------------------------

Ant tests passed on my box.

[ https://issues.apache.org/jira/browse/HADOOP-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717225#action_12717225 ]

Hemanth Yamijala commented on HADOOP-5981:
------------------------------------------

Hmm. I don't fully understand this patch.

So, what is the requirement with respect to variable substitution in mapred.child.env ? Currently, it looks like unless the key has LD_LIBRARY_PATH, it is not going to do anything useful.

For e.g. if I have mapred.child.env with the value as "key=$value", where 'value' is in the environment of the Tasktracker, since 'key' is not added to the 'env' variable in the taskrunner method, it will not work, right ? I thought the use case would be that if any thing in the value should be picked from the tasktracker's environment, it should be replaced. Isn't this the use case ?

[ https://issues.apache.org/jira/browse/HADOOP-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717272#action_12717272 ]

Amar Kamat commented on HADOOP-5981:
------------------------------------

Hemanth,
This is just a bugfix to do with HADOOP-2838. Currently we can pass
# x=:/tmp [sets x to :/tmp]
# x=$x:/tmp [appends :/tmp to x]

We can support x=$y:/tmp but I didnt have any usecase for that. Hemanth, if you think thats a valid usecase then plz file a jira. Was this your concern?

[ https://issues.apache.org/jira/browse/HADOOP-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-5981:
-------------------------------

Attachment: HADOOP-5981-v2.0.patch

Attaching a new patch that
# inherits the tastracker's env variable value for substitution
# fixes the substitution code
# fixes the testcase
# adds meaningful diagnostic information upon faulty env

Result of test-patch
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

Tested this patch on a single node cluster and works as expected. Here is what I did

[ https://issues.apache.org/jira/browse/HADOOP-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717672#action_12717672 ]

Sreekanth Ramakrishnan commented on HADOOP-5981:
------------------------------------------------

Changes look fine with me, with respect to the blocked issue HADOOP-5980, verified the fix with respect to {{LinuxTaskController}}

[ https://issues.apache.org/jira/browse/HADOOP-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718012#action_12718012 ]

Amar Kamat commented on HADOOP-5981:
------------------------------------

Ant tests passed on my box.

[ https://issues.apache.org/jira/browse/HADOOP-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal resolved HADOOP-5981.
------------------------------------

Resolution: Fixed
Hadoop Flags: [Reviewed]

I just committed this. Thanks Amar!

[ https://issues.apache.org/jira/browse/HADOOP-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-5981:
-------------------------------

Attachment: hadoop5981-branch-20-example.patch

Attaching an example patch for 20 branch (not to be committed).

[ https://issues.apache.org/jira/browse/HADOOP-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-5981:
-------------------------------

Release Note:
This issue completes the feature mentioned in HADOOP-2838. HADOOP-2838 provided a way to set env variables in child process. This issue provides a way to inherit tt's env variables and append or reset it. So now
X=$X:y will inherit X (if there) and append y to it.