FAQ
Hi,

In Pseudo mode, when I have set no number of reducer to 2,  I get the following error from a reduce task.
11/07/06 21:15:59 INFO mapreduce.Job: Task Id : attempt_201107062043_0002_r_000000_2, Status : FAILED
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in OnDiskMerger - Thread to merge on-disk map-outputs
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.RuntimeException: java.io.EOFException
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:132)
at org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:530)
at org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:141)
at org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:108)
at org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Mer


But when I set number of reducer to 1, the job completes without the above error.
I am using Hadoop 0.21.0.

Thanks in advance for any assistance!

Shing

Search Discussions

  • Shing Hing Man at Jul 10, 2011 at 6:55 pm
    After setting   mapred.child.java.opts=-Xmx512m (the default was 200m),   I no longer get the exception.

    Shing




    ________________________________
    From: Shing Hing Man <matmsh@yahoo.com>
    To: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
    Sent: Wednesday, 6 July 2011, 21:37
    Subject: Pseudo mode :Reduce task failed when there are more than one reducers


    Hi,

    In Pseudo mode, when I have set no number of reducer to 2,  I get the following error from a reduce task.
    11/07/06 21:15:59 INFO mapreduce.Job: Task Id : attempt_201107062043_0002_r_000000_2, Status : FAILED
    org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in OnDiskMerger - Thread to merge on-disk map-outputs
    at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
    at java.security.AccessController.doPrivileged(Native
    Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
    at org.apache.hadoop.mapred.Child.main(Child.java:211)
    Caused by: java.lang.RuntimeException: java.io.EOFException
    at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:132)
    at org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:530)
    at org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:141)
    at org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:108)
    at
    org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Mer


    But when I set number of reducer to 1, the job completes without the above error.
    I am using Hadoop 0.21.0.

    Thanks in advance for any assistance!

    Shing

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedJul 6, '11 at 8:37p
activeJul 10, '11 at 6:55p
posts2
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Shing Hing Man: 2 posts

People

Translate

site design / logo © 2022 Grokbase