[Hadoop-common-dev] [jira] Created: (HADOOP-3251) hod generated hadoop-site.xml causes hadoop to throw a WARM message on command line when a hadoop command is executed

[ https://issues.apache.org/jira/browse/HADOOP-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-3251:
-------------------------------------

Component/s: (was: contrib/hod)
mapred
Assignee: Devaraj Das (was: Hemanth Yamijala)

Reassiging to mapred to check where the warning message is coming from.

[ https://issues.apache.org/jira/browse/HADOOP-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588902#action_12588902 ]

Hemanth Yamijala commented on HADOOP-3251:
------------------------------------------

Presumably you are getting this issue when running a jar, like the examples jar for e.g.

However, HOD does generate hdfs://namenode in the hadoop-site.xml in hadoop 0.17.0. This was addressed in HADOOP-2936. So, if you notice hadoop dfs commands do not give this warning with HOD generated hadoop-site.xml. So, I will reassign the component appropriately.

[ https://issues.apache.org/jira/browse/HADOOP-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589030#action_12589030 ]

Amar Kamat commented on HADOOP-3251:
------------------------------------

It seems that {{FileSystem#getName()}} is the cause of the problem. {{FileSystem#getName()}} is a deprecated api and should ideally call {{FileSystem#getUri().toString()}} which is not the case. There are two ways to overcome this :
1) Replace all the occurrences of {{FileSystem.getName()}} with {{FileSystem.getUri().toString()}} OR
2) Fix the deprecated {{FileSystem.getName()}} api to call {{getUri().toString()}}.

(1) will result into a big patch while (2) will be a small one.

I think we should to go for (2) since this is a blocker for 0.17, comments?

[ https://issues.apache.org/jira/browse/HADOOP-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley updated HADOOP-3251:
--------------------------------

Fix Version/s: 0.17.0

My vote is for (2). Will this change the return value or possible exceptions thrown by this method?

[ https://issues.apache.org/jira/browse/HADOOP-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589373#action_12589373 ]

Mukund Madhugiri commented on HADOOP-3251:
------------------------------------------

Sorry, I was using an old version of HoD. The latest version of HoD generates the hadoop-site.xml correctly (as pointed to, by Hemanth), and the DFS commands do not show the WARN message.

The WARN message is thrown when you run a job with a jar file.

[ https://issues.apache.org/jira/browse/HADOOP-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589399#action_12589399 ]

Sameer Paranjpye commented on HADOOP-3251:
------------------------------------------

So is this a 'won'tfix' then?

[ https://issues.apache.org/jira/browse/HADOOP-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589406#action_12589406 ]

Amar Kamat commented on HADOOP-3251:
------------------------------------

bq. My vote is for (2). Will this change the return value or possible exceptions thrown by this method?
Now the return value is {{hostname:port}} but should be {{hdfs://hostname:port/}}.

[ https://issues.apache.org/jira/browse/HADOOP-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley updated HADOOP-3251:
--------------------------------

Component/s: (was: mapred)
dfs
Description:
When a job is run, a WARN message is given on the command line. This does not affect the cluster functionality, but is bad user experience. Hence, marking this a blocker for 0.17.0

The message is:
08/04/14 20:53:46 WARN fs.FileSystem: "namenode:59440" is a deprecated filesystem name. Use "hdfs://namenode:59440/" instead.


was:
When a cluster is allocated thru HoD, dfs commands throw a WARN message on the command line. This does not affect the cluster functionality, but is bad user experience. Hence, marking this a blocker for 0.17.0

The message is:
08/04/14 20:53:46 WARN fs.FileSystem: "namenode:59440" is a deprecated filesystem name. Use "hdfs://namenode:59440/" instead.

The entries in HoD generated hadoop-site.xml should be changed to use *hdfs://namenode*, instead of just *namenode*

Assignee: Konstantin Shvachko (was: Devaraj Das)
Summary: WARN message on command line when a hadoop jar command is executed (was: hod generated hadoop-site.xml causes hadoop to throw a WARM message on command line when a hadoop command is executed)

This WARN message started to appear after the fix for HADOOP-1967. Assigning to dfs for further investigation. See Amar's 1st comment. Thinking more about this, solution (1) from Amar (Replace all the occurrences of FileSystem.getName() with FileSystem.getUri().toString()) seems like the right approach, but seems like too big a change for 0.17 this late in the release.

How about removing the WARN message from FileSystem for 0.17 and then fix this issue properly for 0.18 (putting the WARN message back in) and then remove the deprecated method in 0.19. Thoughts?

[ https://issues.apache.org/jira/browse/HADOOP-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy reassigned HADOOP-3251:
-------------------------------------

Assignee: Arun C Murthy (was: Konstantin Shvachko)

[ https://issues.apache.org/jira/browse/HADOOP-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589792#action_12589792 ]

Arun C Murthy commented on HADOOP-3251:
---------------------------------------

This particular problem seems to stem from the fact that JobTracker.getFileSystemName returns the value of FileSystem.getName which is defined by the DistributedFileSystem as:

{code:title=DistributedFileSystem.java|borderStyle=solid}
/** @deprecated */
public String getName() { return uri.getAuthority(); }
{code}

The simple fix for now is to reimplement JobTracker.getFileSystemName as:
{code:title=JobTracker.java|borderStyle=solid}
/**
* Grab the local fs name
*/
public synchronized String getFilesystemName() throws IOException {
if (fs == null) {
throw new IllegalStateException("FileSystem object not available yet");
}
return fs.getUri().toString();
}
{code}

[ https://issues.apache.org/jira/browse/HADOOP-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589791#action_12589791 ]

Tsz Wo (Nicholas), SZE commented on HADOOP-3251:
------------------------------------------------

Here is a stack trace showing how the warning message was generated.
{noformat}
$ ./bin/hadoop jar build/hadoop-0.18.0-dev-examples.jar wordcount input o/1
08/04/16 16:43:39 WARN fs.FileSystem: "hostname:9000" is a deprecated filesystem name. Use "hdfs://hostname:9000/" instead.
java.lang.Exception
at org.apache.hadoop.fs.FileSystem.fixName(FileSystem.java:165)
at org.apache.hadoop.fs.FileSystem.getNamed(FileSystem.java:152)
at org.apache.hadoop.mapred.JobClient.getFs(JobClient.java:437)
at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:543)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:700)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973)
at org.apache.hadoop.examples.WordCount.run(WordCount.java:149)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:155)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:53)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220)
{noformat}

[ https://issues.apache.org/jira/browse/HADOOP-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy updated HADOOP-3251:
----------------------------------

Attachment: HADOOP-3251_0_20080416.patch

Simple fix to {JobTracker|LocalJobRunner}.getFileSystemName to not use FileSystem.getName() and use FileSystem.getUri().

[ https://issues.apache.org/jira/browse/HADOOP-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy updated HADOOP-3251:
----------------------------------

Status: Patch Available (was: Open)

[ https://issues.apache.org/jira/browse/HADOOP-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589796#action_12589796 ]

Arun C Murthy commented on HADOOP-3251:
---------------------------------------

Clearly we need to remove FIleSystem.getName and change all the code using it, however I propose we postpone that to 0.18.

The reason this patch is necessary is due to the fact that it causes the WARN message _even if_ fs.default.name is configured *correctly*.

[ https://issues.apache.org/jira/browse/HADOOP-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-3251:
-------------------------------------------

Hadoop Flags: [Reviewed]
+1

Also, +1 for the patch.

[ https://issues.apache.org/jira/browse/HADOOP-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589815#action_12589815 ]

Hadoop QA commented on HADOOP-3251:
-----------------------------------

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12380350/HADOOP-3251_0_20080416.patch
against trunk revision 645773.

@author +1. The patch does not contain any @author tags.

tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests +1. The patch passed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2258/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2258/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2258/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2258/console

This message is automatically generated.

[ https://issues.apache.org/jira/browse/HADOOP-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-3251:
--------------------------------

Resolution: Fixed
Status: Resolved (was: Patch Available)

I just committed this. Thanks Arun and Amar!

[ https://issues.apache.org/jira/browse/HADOOP-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589991#action_12589991 ]

Hudson commented on HADOOP-3251:
--------------------------------

Integrated in Hadoop-trunk #463 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/463/])