FAQ
GenericOptionsParser should ignore symlinks from archive filenames
------------------------------------------------------------------

Key: HADOOP-6178
URL: https://issues.apache.org/jira/browse/HADOOP-6178
Project: Hadoop Common
Issue Type: Bug
Components: util
Reporter: Amar Kamat


http://hadoop.apache.org/common/docs/r0.20.0/streaming.html#Large+files+and+archives+in+Hadoop+Streaming example doesnt work as expected.
Stack trace :
{code}
trunk$ ./bin/hadoop jar build/contrib/streaming/hadoop-0.21.0-dev-streaming.jar -D mapred.map.tasks=1 \
-D mapred.reduce.tasks=1 -D mapred.job.name="Experiment" \
-archives 'hdfs://namenode:port/user/me/cachedir.jar#testlink' -input input -output output -mapper "xargs cat" -reducer "cat"
java.io.FileNotFoundException: File hdfs://namenode:port/user/me/cachedir.jar#testlink does not exist.
at org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:349)
at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:275)
at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:375)
at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:138)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
{code}

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Amareshwari Sriramadasu (JIRA) at Feb 22, 2010 at 3:48 am
    [ https://issues.apache.org/jira/browse/HADOOP-6178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Amareshwari Sriramadasu resolved HADOOP-6178.
    ---------------------------------------------

    Resolution: Duplicate

    Fixed in MAPREDUCE-787
    GenericOptionsParser should ignore symlinks from archive filenames
    ------------------------------------------------------------------

    Key: HADOOP-6178
    URL: https://issues.apache.org/jira/browse/HADOOP-6178
    Project: Hadoop Common
    Issue Type: Bug
    Components: util
    Reporter: Amar Kamat

    http://hadoop.apache.org/common/docs/r0.20.0/streaming.html#Large+files+and+archives+in+Hadoop+Streaming example doesnt work as expected.
    Stack trace :
    {code}
    trunk$ ./bin/hadoop jar build/contrib/streaming/hadoop-0.21.0-dev-streaming.jar -D mapred.map.tasks=1 \
    -D mapred.reduce.tasks=1 -D mapred.job.name="Experiment" \
    -archives 'hdfs://namenode:port/user/me/cachedir.jar#testlink' -input input -output output -mapper "xargs cat" -reducer "cat"
    java.io.FileNotFoundException: File hdfs://namenode:port/user/me/cachedir.jar#testlink does not exist.
    at org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:349)
    at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:275)
    at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:375)
    at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
    at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:138)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
    at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:50)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedAug 7, '09 at 9:49a
activeFeb 22, '10 at 3:48a
posts2
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Amareshwari Sriramadasu (JIRA): 2 posts

People

Translate

site design / logo © 2022 Grokbase