Hi,

Apologies for cross-posting.

We're in the process of migrating data from an Apache Hadoop 0.20.203.0
cluster to a 0.22.0 cluster using distcp with a hftp source and hdfs dest as
described in the manual. During the copy a handful of the following
cryptic IOExceptions occured and the job finally failed completely. Almost
20GB of data was not copied at all.

11/12/18 22:50:53 INFO mapreduce.Job: Task Id :
attempt_201112181848_0007_m_000097_0, Status : FAILED
java.io.IOException: Copied: 251 Skipped: 0 Failed: 16
at
org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:694)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:389)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
at org.apache.hadoop.mapred.Child$4.run(Child.java:223)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
at org.apache.hadoop.mapred.Child.main(Child.java:217)

The files being copied do not contain any fancy filenames and are all map or
sequence files with an occasional _SUCCESS file here and there. The cryptic
stack trace is similar to some CDH issue [1] which is still open. Any
thoughts?

[1]: https://issues.cloudera.org/browse/DISTRO-347


Thanks

Search Discussions

  • Markus Jelsma at Dec 19, 2011 at 12:30 pm
    It seems the files cannot be validated for some reason. The source files are
    fine, not corrupt and can be read without issues.

    java.io.IOException: Validation of copy of file
    hftp://namenode01.openindex.io:50070/user/systems/segments/index/20111021161228/crawl_parse/part-00011
    failed.
    at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:540)
    at
    org.apache.hadoop.tools.DistCp$CopyFilesMapper.copyWithRetries(DistCp.java:595)
    at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:651)
    at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:325)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:389)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:223)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
    at org.apache.hadoop.mapred.Child.main(Child.java:217)

    When i retry some will be copied but most will fail. At a third retry all will
    fail or be skipped. I also found MAPREDUE-2406 [1] and gave -pb a try, i can
    confirm this solves the problem.

    [1] : https://issues.apache.org/jira/browse/MAPREDUCE-2406
    On Monday 19 December 2011 09:22:39 Markus Jelsma wrote:
    Hi,

    Apologies for cross-posting.

    We're in the process of migrating data from an Apache Hadoop 0.20.203.0
    cluster to a 0.22.0 cluster using distcp with a hftp source and hdfs dest
    as described in the manual. During the copy a handful of the following
    cryptic IOExceptions occured and the job finally failed completely. Almost
    20GB of data was not copied at all.

    11/12/18 22:50:53 INFO mapreduce.Job: Task Id :
    attempt_201112181848_0007_m_000097_0, Status : FAILED
    java.io.IOException: Copied: 251 Skipped: 0 Failed: 16
    at
    org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:694)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:389)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:223)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j
    ava:1153) at org.apache.hadoop.mapred.Child.main(Child.java:217)

    The files being copied do not contain any fancy filenames and are all map
    or sequence files with an occasional _SUCCESS file here and there. The
    cryptic stack trace is similar to some CDH issue [1] which is still open.
    Any thoughts?

    [1]: https://issues.cloudera.org/browse/DISTRO-347


    Thanks

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedDec 19, '11 at 8:24a
activeDec 19, '11 at 12:30p
posts2
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Markus Jelsma: 2 posts

People

Translate

site design / logo © 2022 Grokbase