When connecting to HDFS using an IP Task MapRed gets confused when checking the output path.

Key: HADOOP-5610
URL: https://issues.apache.org/jira/browse/HADOOP-5610
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.19.1, 0.19.0, 0.18.3
Environment: AIX 6.1 and Solaris
Reporter: Bill Habermaas
Priority: Minor

The exception appearing below happens if you are using an IP address to reference a HDFS file system path. Although the URI authority for either path (whether referenced by hostname:port or IP-address:port) refer to the same filesystem, Task throws an exception because the stringified comparisons of the authority in the URIs are not equal. The FileSystem holds the network location of the namenode as a host name regardless as to whether an IP address or hostname is used when the file system is connected (refer to HADOOP-5191 for an earlier fix specifically for HDFS only).

2009-03-27 04:15:45,045 WARN [Thread-145] org.apache.hadoop.mapred.LocalJobRunner: job_local_0002
java.io.IOException: Can not get the relative path: base = hdfs:// child = hdfs://p520aix61.mydomain.com:9000/mydata/2009/03/27/0bab100a-1bf1-499a-935d-bc4b4e94f44c/_temporary/_attempt_local_0002_r_000000_0/part-00000
at org.apache.hadoop.mapred.Task.getFinalPath(Task.java:586)
at org.apache.hadoop.mapred.Task.moveTaskOutputs(Task.java:599)
at org.apache.hadoop.mapred.Task.moveTaskOutputs(Task.java:617)
at org.apache.hadoop.mapred.Task.saveTaskOutput(Task.java:561)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:202)

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
postedApr 1, '09 at 7:06p
activeApr 1, '09 at 7:06p

1 user in discussion

Bill Habermaas (JIRA): 1 post



site design / logo © 2022 Grokbase