FAQ
IOException when running map reduce on S3 filesystem
----------------------------------------------------

Key: HADOOP-857
URL: https://issues.apache.org/jira/browse/HADOOP-857
Project: Hadoop
Issue Type: Bug
Components: fs
Affects Versions: 0.10.0
Reporter: Tom White


Setting fs.default.name to be an S3 path causes the following exception:

java.io.IOException: Cannot create file /tmp/hadoop-tom/mapred/system/submit_habo0/job.jar since parent directory does not exist.
at org.apache.hadoop.fs.s3.S3FileSystem.createRaw(S3FileSystem.java:150)
at org.apache.hadoop.fs.FSDataOutputStream$Summer.(FSDataOutputStream.java:47)
at org.apache.hadoop.fs.FSDataOutputStream.(FileSystem.java:369)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:276)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:94)
at org.apache.hadoop.fs.s3.S3FileSystem.copyFromLocalFile(S3FileSystem.java:290)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:294)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:371)
at org.apache.hadoop.examples.Grep.main(Grep.java:69)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:143)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:41)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.util.RunJar.main(RunJar.java:151)


--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

Search Discussions

  • Tom White (JIRA) at Jan 4, 2007 at 9:45 pm
    [ https://issues.apache.org/jira/browse/HADOOP-857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12462331 ]

    Tom White commented on HADOOP-857:
    ----------------------------------

    This failure occurs because S3FileSystem treats lack of a parent directory as an error in createRaw, whereas LocalFileSystem (and presumably DistributedFileSystem too) creates parent directories if they don't exist.

    I will create a patch to fix this.
    IOException when running map reduce on S3 filesystem
    ----------------------------------------------------

    Key: HADOOP-857
    URL: https://issues.apache.org/jira/browse/HADOOP-857
    Project: Hadoop
    Issue Type: Bug
    Components: fs
    Affects Versions: 0.10.0
    Reporter: Tom White

    Setting fs.default.name to be an S3 path causes the following exception:
    java.io.IOException: Cannot create file /tmp/hadoop-tom/mapred/system/submit_habo0/job.jar since parent directory does not exist.
    at org.apache.hadoop.fs.s3.S3FileSystem.createRaw(S3FileSystem.java:150)
    at org.apache.hadoop.fs.FSDataOutputStream$Summer.<init>(FSDataOutputStream.java:58)
    at org.apache.hadoop.fs.FSDataOutputStream$Summer.<init>(FSDataOutputStream.java:47)
    at org.apache.hadoop.fs.FSDataOutputStream.<init>(FSDataOutputStream.java:148)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:369)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:276)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:94)
    at org.apache.hadoop.fs.s3.S3FileSystem.copyFromLocalFile(S3FileSystem.java:290)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:294)
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:371)
    at org.apache.hadoop.examples.Grep.main(Grep.java:69)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:143)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:41)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:151)
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Tom White (JIRA) at Jan 7, 2007 at 9:46 pm
    [ https://issues.apache.org/jira/browse/HADOOP-857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Tom White updated HADOOP-857:
    -----------------------------

    Attachment: hadoop-857.patch
    IOException when running map reduce on S3 filesystem
    ----------------------------------------------------

    Key: HADOOP-857
    URL: https://issues.apache.org/jira/browse/HADOOP-857
    Project: Hadoop
    Issue Type: Bug
    Components: fs
    Affects Versions: 0.10.0
    Reporter: Tom White
    Attachments: hadoop-857.patch


    Setting fs.default.name to be an S3 path causes the following exception:
    java.io.IOException: Cannot create file /tmp/hadoop-tom/mapred/system/submit_habo0/job.jar since parent directory does not exist.
    at org.apache.hadoop.fs.s3.S3FileSystem.createRaw(S3FileSystem.java:150)
    at org.apache.hadoop.fs.FSDataOutputStream$Summer.<init>(FSDataOutputStream.java:58)
    at org.apache.hadoop.fs.FSDataOutputStream$Summer.<init>(FSDataOutputStream.java:47)
    at org.apache.hadoop.fs.FSDataOutputStream.<init>(FSDataOutputStream.java:148)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:369)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:276)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:94)
    at org.apache.hadoop.fs.s3.S3FileSystem.copyFromLocalFile(S3FileSystem.java:290)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:294)
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:371)
    at org.apache.hadoop.examples.Grep.main(Grep.java:69)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:143)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:41)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:151)
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Tom White (JIRA) at Jan 7, 2007 at 10:13 pm
    [ https://issues.apache.org/jira/browse/HADOOP-857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Tom White updated HADOOP-857:
    -----------------------------

    Status: Patch Available (was: Open)

    This patch makes it possible to run Map/Reduce using S3 instead of HDFS. I've written a wiki page explaining how to do it here: http://wiki.apache.org/lucene-hadoop/AmazonS3.
    IOException when running map reduce on S3 filesystem
    ----------------------------------------------------

    Key: HADOOP-857
    URL: https://issues.apache.org/jira/browse/HADOOP-857
    Project: Hadoop
    Issue Type: Bug
    Components: fs
    Affects Versions: 0.10.0
    Reporter: Tom White
    Attachments: hadoop-857.patch


    Setting fs.default.name to be an S3 path causes the following exception:
    java.io.IOException: Cannot create file /tmp/hadoop-tom/mapred/system/submit_habo0/job.jar since parent directory does not exist.
    at org.apache.hadoop.fs.s3.S3FileSystem.createRaw(S3FileSystem.java:150)
    at org.apache.hadoop.fs.FSDataOutputStream$Summer.<init>(FSDataOutputStream.java:58)
    at org.apache.hadoop.fs.FSDataOutputStream$Summer.<init>(FSDataOutputStream.java:47)
    at org.apache.hadoop.fs.FSDataOutputStream.<init>(FSDataOutputStream.java:148)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:369)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:276)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:94)
    at org.apache.hadoop.fs.s3.S3FileSystem.copyFromLocalFile(S3FileSystem.java:290)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:294)
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:371)
    at org.apache.hadoop.examples.Grep.main(Grep.java:69)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:143)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:41)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:151)
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Hadoop QA (JIRA) at Jan 7, 2007 at 10:29 pm
    [ https://issues.apache.org/jira/browse/HADOOP-857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12462878 ]

    Hadoop QA commented on HADOOP-857:
    ----------------------------------

    +1, because http://issues.apache.org/jira/secure/attachment/12348453/hadoop-857.patch applied and successfully tested against trunk revision r493236.
    IOException when running map reduce on S3 filesystem
    ----------------------------------------------------

    Key: HADOOP-857
    URL: https://issues.apache.org/jira/browse/HADOOP-857
    Project: Hadoop
    Issue Type: Bug
    Components: fs
    Affects Versions: 0.10.0
    Reporter: Tom White
    Attachments: hadoop-857.patch


    Setting fs.default.name to be an S3 path causes the following exception:
    java.io.IOException: Cannot create file /tmp/hadoop-tom/mapred/system/submit_habo0/job.jar since parent directory does not exist.
    at org.apache.hadoop.fs.s3.S3FileSystem.createRaw(S3FileSystem.java:150)
    at org.apache.hadoop.fs.FSDataOutputStream$Summer.<init>(FSDataOutputStream.java:58)
    at org.apache.hadoop.fs.FSDataOutputStream$Summer.<init>(FSDataOutputStream.java:47)
    at org.apache.hadoop.fs.FSDataOutputStream.<init>(FSDataOutputStream.java:148)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:369)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:276)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:94)
    at org.apache.hadoop.fs.s3.S3FileSystem.copyFromLocalFile(S3FileSystem.java:290)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:294)
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:371)
    at org.apache.hadoop.examples.Grep.main(Grep.java:69)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:143)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:41)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:151)
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Doug Cutting (JIRA) at Jan 8, 2007 at 6:23 pm
    [ https://issues.apache.org/jira/browse/HADOOP-857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Doug Cutting updated HADOOP-857:
    --------------------------------

    Resolution: Fixed
    Fix Version/s: 0.10.1
    Status: Resolved (was: Patch Available)

    I just committed this. Thanks, Tom.
    IOException when running map reduce on S3 filesystem
    ----------------------------------------------------

    Key: HADOOP-857
    URL: https://issues.apache.org/jira/browse/HADOOP-857
    Project: Hadoop
    Issue Type: Bug
    Components: fs
    Affects Versions: 0.10.0
    Reporter: Tom White
    Fix For: 0.10.1

    Attachments: hadoop-857.patch


    Setting fs.default.name to be an S3 path causes the following exception:
    java.io.IOException: Cannot create file /tmp/hadoop-tom/mapred/system/submit_habo0/job.jar since parent directory does not exist.
    at org.apache.hadoop.fs.s3.S3FileSystem.createRaw(S3FileSystem.java:150)
    at org.apache.hadoop.fs.FSDataOutputStream$Summer.<init>(FSDataOutputStream.java:58)
    at org.apache.hadoop.fs.FSDataOutputStream$Summer.<init>(FSDataOutputStream.java:47)
    at org.apache.hadoop.fs.FSDataOutputStream.<init>(FSDataOutputStream.java:148)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:369)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:276)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:94)
    at org.apache.hadoop.fs.s3.S3FileSystem.copyFromLocalFile(S3FileSystem.java:290)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:294)
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:371)
    at org.apache.hadoop.examples.Grep.main(Grep.java:69)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:143)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:41)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:151)
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedJan 4, '07 at 9:25p
activeJan 8, '07 at 6:23p
posts6
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Doug Cutting (JIRA): 6 posts

People

Translate

site design / logo © 2022 Grokbase