FAQ
Hi friends,

I have made a cluster of 3 machines, one of them is master, and other 2
slaves. I executed a mapreduce job on master but after Map, the execution
terminates and Reduce doesn't happen. I have checked dfs and no output
folder gets created.

this is the error I see

08/03/10 10:35:21 INFO mapred.JobClient: Task Id :
task_200803101001_0001_m_000064_0, Status : FAILED
java.lang.OutOfMemoryError: Java heap space
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
:95)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.Text.write(Text.java:243)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
MapTask.java:347)
at org.apache.hadoop.examples.WordCount$MapClass.map(WordCount.java
:72)
at org.apache.hadoop.examples.WordCount$MapClass.map(WordCount.java
:59)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java
:1787)

08/03/10 10:35:22 INFO mapred.JobClient: map 55% reduce 17%
08/03/10 10:35:31 INFO mapred.JobClient: map 56% reduce 17%
08/03/10 10:35:51 INFO mapred.JobClient: map 57% reduce 17%
08/03/10 10:36:04 INFO mapred.JobClient: map 58% reduce 17%
08/03/10 10:36:07 INFO mapred.JobClient: map 57% reduce 17%
08/03/10 10:36:07 INFO mapred.JobClient: Task Id :
task_200803101001_0001_m_000071_0, Status : FAILED
java.lang.OutOfMemoryError: Java heap space
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
:95)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.Text.write(Text.java:243)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
MapTask.java:347)
at org.apache.hadoop.examples.WordCount$MapClass.map(WordCount.java
:72)
at org.apache.hadoop.examples.WordCount$MapClass.map(WordCount.java
:59)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java
:1787)

though it tries to overcome this problem but the mapreduce application
doesn't create output, can anyone tell me why is this happening?

Thanks

Search Discussions

  • Amar Kamat at Mar 10, 2008 at 5:30 am
    What is the heap size you are using for your tasks? Check
    'mapred.child.java.opts' in your hadoop-default.xml. Try increasing it.
    This will happen if you try running the random-writer + sort examples with
    default parameters. The maps are not able to spill the data to the disk.
    Btw what version of HADOOP are you using?
    Amar
    On Mon, 10 Mar 2008, Ved Prakash
    wrote:
    Hi friends,

    I have made a cluster of 3 machines, one of them is master, and other 2
    slaves. I executed a mapreduce job on master but after Map, the execution
    terminates and Reduce doesn't happen. I have checked dfs and no output
    folder gets created.

    this is the error I see

    08/03/10 10:35:21 INFO mapred.JobClient: Task Id :
    task_200803101001_0001_m_000064_0, Status : FAILED
    java.lang.OutOfMemoryError: Java heap space
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
    :95)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at org.apache.hadoop.io.Text.write(Text.java:243)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
    MapTask.java:347)
    at org.apache.hadoop.examples.WordCount$MapClass.map(WordCount.java
    :72)
    at org.apache.hadoop.examples.WordCount$MapClass.map(WordCount.java
    :59)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
    at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java
    :1787)

    08/03/10 10:35:22 INFO mapred.JobClient: map 55% reduce 17%
    08/03/10 10:35:31 INFO mapred.JobClient: map 56% reduce 17%
    08/03/10 10:35:51 INFO mapred.JobClient: map 57% reduce 17%
    08/03/10 10:36:04 INFO mapred.JobClient: map 58% reduce 17%
    08/03/10 10:36:07 INFO mapred.JobClient: map 57% reduce 17%
    08/03/10 10:36:07 INFO mapred.JobClient: Task Id :
    task_200803101001_0001_m_000071_0, Status : FAILED
    java.lang.OutOfMemoryError: Java heap space
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
    :95)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at org.apache.hadoop.io.Text.write(Text.java:243)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
    MapTask.java:347)
    at org.apache.hadoop.examples.WordCount$MapClass.map(WordCount.java
    :72)
    at org.apache.hadoop.examples.WordCount$MapClass.map(WordCount.java
    :59)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
    at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java
    :1787)

    though it tries to overcome this problem but the mapreduce application
    doesn't create output, can anyone tell me why is this happening?

    Thanks
  • Stefan Groschupf at Mar 16, 2008 at 3:38 am
    Hi there,

    we see the same situation and browsing the posts there are quite a lot
    of people running into this OOM problem.
    We run a own Mapper and our mapred.child.java.opts is -Xmx3048m, I
    think that should be more then enough.
    Also I changed io.sort.mb to 10, which had also no impact.

    Any ideas what might cause the OutOfMemoryError ?
    Thanks.
    Stefan



    On Mar 9, 2008, at 10:28 PM, Amar Kamat wrote:

    What is the heap size you are using for your tasks? Check
    'mapred.child.java.opts' in your hadoop-default.xml. Try increasing
    it. This will happen if you try running the random-writer + sort
    examples with default parameters. The maps are not able to spill the
    data to the disk. Btw what version of HADOOP are you using?
    Amar
    On Mon, 10 Mar 2008, Ved Prakash wrote:

    Hi friends,

    I have made a cluster of 3 machines, one of them is master, and
    other 2
    slaves. I executed a mapreduce job on master but after Map, the
    execution
    terminates and Reduce doesn't happen. I have checked dfs and no
    output
    folder gets created.

    this is the error I see

    08/03/10 10:35:21 INFO mapred.JobClient: Task Id :
    task_200803101001_0001_m_000064_0, Status : FAILED
    java.lang.OutOfMemoryError: Java heap space
    at
    java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
    :95)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at org.apache.hadoop.io.Text.write(Text.java:243)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
    MapTask.java:347)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :72)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :59)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
    at org.apache.hadoop.mapred.TaskTracker
    $Child.main(TaskTracker.java
    :1787)

    08/03/10 10:35:22 INFO mapred.JobClient: map 55% reduce 17%
    08/03/10 10:35:31 INFO mapred.JobClient: map 56% reduce 17%
    08/03/10 10:35:51 INFO mapred.JobClient: map 57% reduce 17%
    08/03/10 10:36:04 INFO mapred.JobClient: map 58% reduce 17%
    08/03/10 10:36:07 INFO mapred.JobClient: map 57% reduce 17%
    08/03/10 10:36:07 INFO mapred.JobClient: Task Id :
    task_200803101001_0001_m_000071_0, Status : FAILED
    java.lang.OutOfMemoryError: Java heap space
    at
    java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
    :95)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at org.apache.hadoop.io.Text.write(Text.java:243)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
    MapTask.java:347)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :72)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :59)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
    at org.apache.hadoop.mapred.TaskTracker
    $Child.main(TaskTracker.java
    :1787)

    though it tries to overcome this problem but the mapreduce
    application
    doesn't create output, can anyone tell me why is this happening?

    Thanks
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    101tec Inc.
    Menlo Park, California, USA
    http://www.101tec.com
  • Devaraj Das at Mar 16, 2008 at 6:02 am
    It might have something to do with your application itself. By any chance
    are you doing a lot of huge object allocation (directly or indirectly)
    within the map method? Which version of hadoop are you on?
    -----Original Message-----
    From: Stefan Groschupf
    Sent: Sunday, March 16, 2008 9:07 AM
    To: core-user@hadoop.apache.org
    Subject: [memory leak?] Re: MapReduce failure

    Hi there,

    we see the same situation and browsing the posts there are
    quite a lot of people running into this OOM problem.
    We run a own Mapper and our mapred.child.java.opts is
    -Xmx3048m, I think that should be more then enough.
    Also I changed io.sort.mb to 10, which had also no impact.

    Any ideas what might cause the OutOfMemoryError ?
    Thanks.
    Stefan



    On Mar 9, 2008, at 10:28 PM, Amar Kamat wrote:

    What is the heap size you are using for your tasks? Check
    'mapred.child.java.opts' in your hadoop-default.xml. Try increasing
    it. This will happen if you try running the random-writer + sort
    examples with default parameters. The maps are not able to spill the
    data to the disk. Btw what version of HADOOP are you using?
    Amar
    On Mon, 10 Mar 2008, Ved Prakash wrote:

    Hi friends,

    I have made a cluster of 3 machines, one of them is
    master, and other
    2 slaves. I executed a mapreduce job on master but after Map, the
    execution terminates and Reduce doesn't happen. I have checked dfs
    and no output folder gets created.

    this is the error I see

    08/03/10 10:35:21 INFO mapred.JobClient: Task Id :
    task_200803101001_0001_m_000064_0, Status : FAILED
    java.lang.OutOfMemoryError: Java heap space
    at
    java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
    :95)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at org.apache.hadoop.io.Text.write(Text.java:243)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
    MapTask.java:347)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :72)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :59)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
    at org.apache.hadoop.mapred.TaskTracker
    $Child.main(TaskTracker.java
    :1787)

    08/03/10 10:35:22 INFO mapred.JobClient: map 55% reduce
    17% 08/03/10
    10:35:31 INFO mapred.JobClient: map 56% reduce 17%
    08/03/10 10:35:51
    INFO mapred.JobClient: map 57% reduce 17% 08/03/10 10:36:04 INFO
    mapred.JobClient: map 58% reduce 17% 08/03/10 10:36:07 INFO
    mapred.JobClient: map 57% reduce 17% 08/03/10 10:36:07 INFO
    mapred.JobClient: Task Id :
    task_200803101001_0001_m_000071_0, Status : FAILED
    java.lang.OutOfMemoryError: Java heap space
    at
    java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
    :95)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at org.apache.hadoop.io.Text.write(Text.java:243)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
    MapTask.java:347)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :72)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :59)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
    at org.apache.hadoop.mapred.TaskTracker
    $Child.main(TaskTracker.java
    :1787)

    though it tries to overcome this problem but the mapreduce
    application doesn't create output, can anyone tell me why is this
    happening?

    Thanks
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    101tec Inc.
    Menlo Park, California, USA
    http://www.101tec.com

  • Stefan Groschupf at Mar 16, 2008 at 7:16 am
    I do not instantiate 3 GB of objects, that is for sure. The wordcount
    example does not run anymore so I dont think this is something
    releated to my code, beside the wordcount example many other users
    report the same problem:
    See:
    http://markmail.org/search/?q=org.apache.hadoop.mapred.MapTask%24MapOutputBuffer.collect+order%3Adate-backward
    Thanks for your help!

    Stefan

    On Mar 15, 2008, at 11:02 PM, Devaraj Das wrote:

    It might have something to do with your application itself. By any
    chance
    are you doing a lot of huge object allocation (directly or indirectly)
    within the map method? Which version of hadoop are you on?
    -----Original Message-----
    From: Stefan Groschupf
    Sent: Sunday, March 16, 2008 9:07 AM
    To: core-user@hadoop.apache.org
    Subject: [memory leak?] Re: MapReduce failure

    Hi there,

    we see the same situation and browsing the posts there are
    quite a lot of people running into this OOM problem.
    We run a own Mapper and our mapred.child.java.opts is
    -Xmx3048m, I think that should be more then enough.
    Also I changed io.sort.mb to 10, which had also no impact.

    Any ideas what might cause the OutOfMemoryError ?
    Thanks.
    Stefan



    On Mar 9, 2008, at 10:28 PM, Amar Kamat wrote:

    What is the heap size you are using for your tasks? Check
    'mapred.child.java.opts' in your hadoop-default.xml. Try increasing
    it. This will happen if you try running the random-writer + sort
    examples with default parameters. The maps are not able to spill the
    data to the disk. Btw what version of HADOOP are you using?
    Amar
    On Mon, 10 Mar 2008, Ved Prakash wrote:

    Hi friends,

    I have made a cluster of 3 machines, one of them is
    master, and other
    2 slaves. I executed a mapreduce job on master but after Map, the
    execution terminates and Reduce doesn't happen. I have checked dfs
    and no output folder gets created.

    this is the error I see

    08/03/10 10:35:21 INFO mapred.JobClient: Task Id :
    task_200803101001_0001_m_000064_0, Status : FAILED
    java.lang.OutOfMemoryError: Java heap space
    at
    java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
    :95)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at org.apache.hadoop.io.Text.write(Text.java:243)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
    MapTask.java:347)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :72)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :59)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
    at org.apache.hadoop.mapred.TaskTracker
    $Child.main(TaskTracker.java
    :1787)

    08/03/10 10:35:22 INFO mapred.JobClient: map 55% reduce
    17% 08/03/10
    10:35:31 INFO mapred.JobClient: map 56% reduce 17%
    08/03/10 10:35:51
    INFO mapred.JobClient: map 57% reduce 17% 08/03/10 10:36:04 INFO
    mapred.JobClient: map 58% reduce 17% 08/03/10 10:36:07 INFO
    mapred.JobClient: map 57% reduce 17% 08/03/10 10:36:07 INFO
    mapred.JobClient: Task Id :
    task_200803101001_0001_m_000071_0, Status : FAILED
    java.lang.OutOfMemoryError: Java heap space
    at
    java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
    :95)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at org.apache.hadoop.io.Text.write(Text.java:243)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
    MapTask.java:347)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :72)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :59)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
    at org.apache.hadoop.mapred.TaskTracker
    $Child.main(TaskTracker.java
    :1787)

    though it tries to overcome this problem but the mapreduce
    application doesn't create output, can anyone tell me why is this
    happening?

    Thanks
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    101tec Inc.
    Menlo Park, California, USA
    http://www.101tec.com

    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    101tec Inc.
    Menlo Park, California, USA
    http://www.101tec.com
  • Devaraj Das at Mar 16, 2008 at 10:57 am
    Wordcount doesn't work? Which platform are you on? I just ran the wordcount
    on a RHEL linux machine with hadoop-0.16.0 and that worked, and my -Xmx is
    512m.
    Could you pls share your config file (hadoop-site.xml) as well?
    -----Original Message-----
    From: Stefan Groschupf
    Sent: Sunday, March 16, 2008 12:46 PM
    To: core-user@hadoop.apache.org
    Subject: Re: [memory leak?] Re: MapReduce failure

    I do not instantiate 3 GB of objects, that is for sure. The
    wordcount example does not run anymore so I dont think this
    is something releated to my code, beside the wordcount
    example many other users report the same problem:
    See:
    http://markmail.org/search/?q=org.apache.hadoop.mapred.MapTask
    %24MapOutputBuffer.collect+order%3Adate-backward
    Thanks for your help!

    Stefan

    On Mar 15, 2008, at 11:02 PM, Devaraj Das wrote:

    It might have something to do with your application itself. By any
    chance are you doing a lot of huge object allocation (directly or
    indirectly) within the map method? Which version of hadoop
    are you on?
    -----Original Message-----
    From: Stefan Groschupf
    Sent: Sunday, March 16, 2008 9:07 AM
    To: core-user@hadoop.apache.org
    Subject: [memory leak?] Re: MapReduce failure

    Hi there,

    we see the same situation and browsing the posts there are quite a
    lot of people running into this OOM problem.
    We run a own Mapper and our mapred.child.java.opts is -Xmx3048m, I
    think that should be more then enough.
    Also I changed io.sort.mb to 10, which had also no impact.

    Any ideas what might cause the OutOfMemoryError ?
    Thanks.
    Stefan



    On Mar 9, 2008, at 10:28 PM, Amar Kamat wrote:

    What is the heap size you are using for your tasks? Check
    'mapred.child.java.opts' in your hadoop-default.xml. Try
    increasing
    it. This will happen if you try running the random-writer + sort
    examples with default parameters. The maps are not able to spill the
    data to the disk. Btw what version of HADOOP are you using?
    Amar
    On Mon, 10 Mar 2008, Ved Prakash wrote:

    Hi friends,

    I have made a cluster of 3 machines, one of them is
    master, and other
    2 slaves. I executed a mapreduce job on master but after
    Map, the
    execution terminates and Reduce doesn't happen. I have
    checked dfs
    and no output folder gets created.

    this is the error I see

    08/03/10 10:35:21 INFO mapred.JobClient: Task Id :
    task_200803101001_0001_m_000064_0, Status : FAILED
    java.lang.OutOfMemoryError: Java heap space
    at
    java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
    :95)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at org.apache.hadoop.io.Text.write(Text.java:243)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
    MapTask.java:347)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :72)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :59)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
    at org.apache.hadoop.mapred.TaskTracker
    $Child.main(TaskTracker.java
    :1787)

    08/03/10 10:35:22 INFO mapred.JobClient: map 55% reduce
    17% 08/03/10
    10:35:31 INFO mapred.JobClient: map 56% reduce 17%
    08/03/10 10:35:51
    INFO mapred.JobClient: map 57% reduce 17% 08/03/10 10:36:04 INFO
    mapred.JobClient: map 58% reduce 17% 08/03/10 10:36:07 INFO
    mapred.JobClient: map 57% reduce 17% 08/03/10 10:36:07 INFO
    mapred.JobClient: Task Id :
    task_200803101001_0001_m_000071_0, Status : FAILED
    java.lang.OutOfMemoryError: Java heap space
    at
    java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
    :95)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at org.apache.hadoop.io.Text.write(Text.java:243)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
    MapTask.java:347)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :72)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :59)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
    at org.apache.hadoop.mapred.TaskTracker
    $Child.main(TaskTracker.java
    :1787)

    though it tries to overcome this problem but the mapreduce
    application doesn't create output, can anyone tell me
    why is this
    happening?

    Thanks
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    101tec Inc.
    Menlo Park, California, USA
    http://www.101tec.com

    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    101tec Inc.
    Menlo Park, California, USA
    http://www.101tec.com

  • Stefan Groschupf at Mar 16, 2008 at 7:18 am
    ups sorry I forgot to mention I use 0.16.0. I will try to update to
    16.1 tomorrow and see if this helps, but i couldn't find an closed
    issue in jira that might be related.
    On Mar 15, 2008, at 8:37 PM, Stefan Groschupf wrote:

    Hi there,

    we see the same situation and browsing the posts there are quite a
    lot of people running into this OOM problem.
    We run a own Mapper and our mapred.child.java.opts is -Xmx3048m, I
    think that should be more then enough.
    Also I changed io.sort.mb to 10, which had also no impact.

    Any ideas what might cause the OutOfMemoryError ?
    Thanks.
    Stefan



    On Mar 9, 2008, at 10:28 PM, Amar Kamat wrote:

    What is the heap size you are using for your tasks? Check
    'mapred.child.java.opts' in your hadoop-default.xml. Try increasing
    it. This will happen if you try running the random-writer + sort
    examples with default parameters. The maps are not able to spill
    the data to the disk. Btw what version of HADOOP are you using?
    Amar
    On Mon, 10 Mar 2008, Ved Prakash wrote:

    Hi friends,

    I have made a cluster of 3 machines, one of them is master, and
    other 2
    slaves. I executed a mapreduce job on master but after Map, the
    execution
    terminates and Reduce doesn't happen. I have checked dfs and no
    output
    folder gets created.

    this is the error I see

    08/03/10 10:35:21 INFO mapred.JobClient: Task Id :
    task_200803101001_0001_m_000064_0, Status : FAILED
    java.lang.OutOfMemoryError: Java heap space
    at
    java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
    :95)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at org.apache.hadoop.io.Text.write(Text.java:243)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
    MapTask.java:347)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :72)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :59)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
    at org.apache.hadoop.mapred.TaskTracker
    $Child.main(TaskTracker.java
    :1787)

    08/03/10 10:35:22 INFO mapred.JobClient: map 55% reduce 17%
    08/03/10 10:35:31 INFO mapred.JobClient: map 56% reduce 17%
    08/03/10 10:35:51 INFO mapred.JobClient: map 57% reduce 17%
    08/03/10 10:36:04 INFO mapred.JobClient: map 58% reduce 17%
    08/03/10 10:36:07 INFO mapred.JobClient: map 57% reduce 17%
    08/03/10 10:36:07 INFO mapred.JobClient: Task Id :
    task_200803101001_0001_m_000071_0, Status : FAILED
    java.lang.OutOfMemoryError: Java heap space
    at
    java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
    :95)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at org.apache.hadoop.io.Text.write(Text.java:243)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
    MapTask.java:347)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :72)
    at org.apache.hadoop.examples.WordCount
    $MapClass.map(WordCount.java
    :59)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
    at org.apache.hadoop.mapred.TaskTracker
    $Child.main(TaskTracker.java
    :1787)

    though it tries to overcome this problem but the mapreduce
    application
    doesn't create output, can anyone tell me why is this happening?

    Thanks
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    101tec Inc.
    Menlo Park, California, USA
    http://www.101tec.com

    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    101tec Inc.
    Menlo Park, California, USA
    http://www.101tec.com
  • Ved Prakash at Mar 18, 2008 at 7:49 am
    I increased the heap size as you have suggested, and I could run a map
    reduce job on it.

    thanks
    On Mon, Mar 10, 2008 at 10:58 AM, Amar Kamat wrote:

    What is the heap size you are using for your tasks? Check
    'mapred.child.java.opts' in your hadoop-default.xml. Try increasing it.
    This will happen if you try running the random-writer + sort examples with
    default parameters. The maps are not able to spill the data to the disk.
    Btw what version of HADOOP are you using?
    Amar
    On Mon, 10 Mar 2008, Ved Prakash
    wrote:
    Hi friends,

    I have made a cluster of 3 machines, one of them is master, and other 2
    slaves. I executed a mapreduce job on master but after Map, the execution
    terminates and Reduce doesn't happen. I have checked dfs and no output
    folder gets created.

    this is the error I see

    08/03/10 10:35:21 INFO mapred.JobClient: Task Id :
    task_200803101001_0001_m_000064_0, Status : FAILED
    java.lang.OutOfMemoryError: Java heap space
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
    :95)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at org.apache.hadoop.io.Text.write(Text.java:243)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
    MapTask.java:347)
    at org.apache.hadoop.examples.WordCount$MapClass.map(
    WordCount.java
    :72)
    at org.apache.hadoop.examples.WordCount$MapClass.map(
    WordCount.java
    :59)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
    at org.apache.hadoop.mapred.TaskTracker$Child.main(
    TaskTracker.java
    :1787)

    08/03/10 10:35:22 INFO mapred.JobClient: map 55% reduce 17%
    08/03/10 10:35:31 INFO mapred.JobClient: map 56% reduce 17%
    08/03/10 10:35:51 INFO mapred.JobClient: map 57% reduce 17%
    08/03/10 10:36:04 INFO mapred.JobClient: map 58% reduce 17%
    08/03/10 10:36:07 INFO mapred.JobClient: map 57% reduce 17%
    08/03/10 10:36:07 INFO mapred.JobClient: Task Id :
    task_200803101001_0001_m_000071_0, Status : FAILED
    java.lang.OutOfMemoryError: Java heap space
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
    :95)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at org.apache.hadoop.io.Text.write(Text.java:243)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
    MapTask.java:347)
    at org.apache.hadoop.examples.WordCount$MapClass.map(
    WordCount.java
    :72)
    at org.apache.hadoop.examples.WordCount$MapClass.map(
    WordCount.java
    :59)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
    at org.apache.hadoop.mapred.TaskTracker$Child.main(
    TaskTracker.java
    :1787)

    though it tries to overcome this problem but the mapreduce application
    doesn't create output, can anyone tell me why is this happening?

    Thanks

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMar 10, '08 at 5:10a
activeMar 18, '08 at 7:49a
posts8
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2021 Grokbase