FAQ
Hi,

I'm testing some of our jobs on 0.21 (built last week) which are still
written against the old API and I get across a weird issue. Simply, we
don't want reducers so we set job.setNumReduceTasks(0) but when the
job starts it does get reduce tasks and they all fail the same way:

2010-01-26 19:17:23,942 WARN org.apache.hadoop.mapred.Child: Exception
running child :
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
shuffle in fetcher#5
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:119)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:358)
at org.apache.hadoop.mapred.Child.main(Child.java:165)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 219
at org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.getMapsForHost(ShuffleScheduler.java:319)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:167)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:144)

See the a full log here: http://pastebin.com/m7359f8ce

Relevant line from JT: 2010-01-26 19:12:27,625 INFO
org.apache.hadoop.mapred.JobInProgress: Job job_201001261911_0001
initialized successfully with 1001 map tasks and 113 reduce tasks.

In mapred-site.xml mapred.reduce.tasks is set to 113.

I searched the jiras, didn't find anything obviously relevant. Is
there something we are overlooking?

J-D

Search Discussions

  • Jean-Daniel Cryans at Jan 29, 2010 at 5:55 pm
    Thanks for the pointer, before seeing that email I was able to fix it
    by setting mapreduce.reduce.tasks instead of mapred. It seems that
    6439 is what I really want.

    Thx!

    J-D
    On Thu, Jan 28, 2010 at 5:31 PM, Chris Douglas wrote:
    You might want to check HADOOP-6439, which is tracking some issues
    with old config params causing issues with the shuffle in 0.21. -C
    On Thu, Jan 28, 2010 at 5:08 PM, Jean-Daniel Cryans wrote:
    Hi,

    I'm testing some of our jobs on 0.21 (built last week) which are still
    written against the old API and I get across a weird issue. Simply, we
    don't want reducers so we set job.setNumReduceTasks(0) but when the
    job starts it does get reduce tasks and they all fail the same way:

    2010-01-26 19:17:23,942 WARN org.apache.hadoop.mapred.Child: Exception
    running child :
    org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
    shuffle in fetcher#5
    at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:119)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:358)
    at org.apache.hadoop.mapred.Child.main(Child.java:165)
    Caused by: java.lang.ArrayIndexOutOfBoundsException: 219
    at org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.getMapsForHost(ShuffleScheduler.java:319)
    at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:167)
    at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:144)

    See the a full log here: http://pastebin.com/m7359f8ce

    Relevant line from JT: 2010-01-26 19:12:27,625 INFO
    org.apache.hadoop.mapred.JobInProgress: Job job_201001261911_0001
    initialized successfully with 1001 map tasks and 113 reduce tasks.

    In mapred-site.xml mapred.reduce.tasks is set to 113.

    I searched the jiras, didn't find anything obviously relevant. Is
    there something we are overlooking?

    J-D

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-dev @
categorieshadoop
postedJan 29, '10 at 1:08a
activeJan 29, '10 at 5:55p
posts2
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Jean-Daniel Cryans: 2 posts

People

Translate

site design / logo © 2022 Grokbase