By the way it happens on Yarn not on MRv1
each container gets 1GB at the moment.
can you try increasing memory per reducer ?


On Wed, Oct 31, 2012 at 9:15 PM, Eduard Skaley <e.v.skaley@gmail.com
wrote:

Hello,

I'm getting this Error through job execution:

16:20:26 INFO [main] Job - map 100% reduce 46%
16:20:27 INFO [main] Job - map 100% reduce 51%
16:20:29 INFO [main] Job - map 100% reduce 62%
16:20:30 INFO [main] Job - map 100% reduce 64%
16:20:32 INFO [main] Job - Task Id :
attempt_1351680008718_0018_r_000006_0, Status : FAILED
Error:
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError:
error in shuffle in fetcher#2
at
org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:123)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:371)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:58)
at
org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:45)
at
org.apache.hadoop.mapreduce.task.reduce.MapOutput.<init>(MapOutput.java:97)
at
org.apache.hadoop.mapreduce.task.reduce.MergeManager.unconditionalReserve(MergeManager.java:286)
at
org.apache.hadoop.mapreduce.task.reduce.MergeManager.reserve(MergeManager.java:276)
at
org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:384)
at
org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:319)
at
org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:179)

16:20:33 INFO [main] Job - map 100% reduce 65%
16:20:36 INFO [main] Job - map 100% reduce 67%
16:20:39 INFO [main] Job - map 100% reduce 69%
16:20:41 INFO [main] Job - map 100% reduce 70%
16:20:43 INFO [main] Job - map 100% reduce 71%

I have no clue what the issue could be for this. I googled this
issue and checked several sources of possible solutions but
nothing does fit.

I saw this jira entry which could fit:
https://issues.apache.org/jira/browse/MAPREDUCE-4655.

Here somebody recommends to increase the value for the property
dfs.datanode.max.xcievers / dfs.datanode.max.receiver.threads to
4096, but this is the value for our cluster.
http://yaseminavcular.blogspot.de/2011/04/common-hadoop-hdfs-exceptions-with.html

The issue with the to small input files doesn't fit I think,
because the map phase reads 137 files with each 130MB. Block Size
is 128MB.

The cluster uses version 2.0.0-cdh4.1.1,
581959ba23e4af85afd8db98b7687662fe9c5f20.

Thx









--
Nitin Pawar

Search Discussions

  • Eduard Skaley at Nov 5, 2012 at 1:21 pm
    We increased mapreduce.reduce.memory.mb to 2GB and
    mapreduce.reduce.java.opts to 1.5GB.

    Now we are getting livelocks for our jobs, map jobs don't start.

    We are using CapacityScheduler because we had LiveLocks with FifoScheduler.

    Does anybody have a clue ?
    By the way it happens on Yarn not on MRv1
    each container gets 1GB at the moment.
    can you try increasing memory per reducer ?


    On Wed, Oct 31, 2012 at 9:15 PM, Eduard Skaley <e.v.skaley@gmail.com
    wrote:

    Hello,

    I'm getting this Error through job execution:

    16:20:26 INFO [main] Job - map 100% reduce 46%
    16:20:27 INFO [main] Job - map 100% reduce 51%
    16:20:29 INFO [main] Job - map 100% reduce 62%
    16:20:30 INFO [main] Job - map 100% reduce 64%
    16:20:32 INFO [main] Job - Task Id :
    attempt_1351680008718_0018_r_000006_0, Status : FAILED
    Error:
    org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError:
    error in shuffle in fetcher#2
    at
    org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:123)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:371)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
    Caused by: java.lang.OutOfMemoryError: Java heap space
    at
    org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:58)
    at
    org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:45)
    at
    org.apache.hadoop.mapreduce.task.reduce.MapOutput.<init>(MapOutput.java:97)
    at
    org.apache.hadoop.mapreduce.task.reduce.MergeManager.unconditionalReserve(MergeManager.java:286)
    at
    org.apache.hadoop.mapreduce.task.reduce.MergeManager.reserve(MergeManager.java:276)
    at
    org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:384)
    at
    org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:319)
    at
    org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:179)

    16:20:33 INFO [main] Job - map 100% reduce 65%
    16:20:36 INFO [main] Job - map 100% reduce 67%
    16:20:39 INFO [main] Job - map 100% reduce 69%
    16:20:41 INFO [main] Job - map 100% reduce 70%
    16:20:43 INFO [main] Job - map 100% reduce 71%

    I have no clue what the issue could be for this. I googled this
    issue and checked several sources of possible solutions but
    nothing does fit.

    I saw this jira entry which could fit:
    https://issues.apache.org/jira/browse/MAPREDUCE-4655.

    Here somebody recommends to increase the value for the property
    dfs.datanode.max.xcievers / dfs.datanode.max.receiver.threads to
    4096, but this is the value for our cluster.
    http://yaseminavcular.blogspot.de/2011/04/common-hadoop-hdfs-exceptions-with.html

    The issue with the to small input files doesn't fit I think,
    because the map phase reads 137 files with each 130MB. Block
    Size is 128MB.

    The cluster uses version 2.0.0-cdh4.1.1,
    581959ba23e4af85afd8db98b7687662fe9c5f20.

    Thx









    --
    Nitin Pawar
  • Kartashov, Andy at Nov 5, 2012 at 3:43 pm
    Your error takes place during reduce task, when temporary files are written to memory/disk. You are clearly running low on resources. Check your memory "$ free -m" and disk space "$ df -H" as well as "$hadoop fs -df"

    I remember it took me a couple of days to figure out why I was getting heap size error and nothing wporked! Becaue, I tried to write 7Gb output file onto a disk (in pseudo distr mode) that only had 4Gb of free space.

    p.s. Always test your jobs on small input first (few lines of inputs) .

    p.p.s. follow your job execution through web: http://<fully-qualified-hostan-name<http://%3cfully-qualified-hostan-name> of your job tracker>:50030


    From: Eduard Skaley
    Sent: Monday, November 05, 2012 4:10 AM
    To: user@hadoop.apache.org
    Cc: Nitin Pawar
    Subject: Re: Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError Java Heap Space

    By the way it happens on Yarn not on MRv1
    each container gets 1GB at the moment.
    can you try increasing memory per reducer ?

    On Wed, Oct 31, 2012 at 9:15 PM, Eduard Skaley wrote:
    Hello,

    I'm getting this Error through job execution:

    16:20:26 INFO [main] Job - map 100% reduce 46%
    16:20:27 INFO [main] Job - map 100% reduce 51%
    16:20:29 INFO [main] Job - map 100% reduce 62%
    16:20:30 INFO [main] Job - map 100% reduce 64%
    16:20:32 INFO [main] Job - Task Id : attempt_1351680008718_0018_r_000006_0, Status : FAILED
    Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#2
    at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:123)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:371)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
    Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.io.BoundedByteArrayOutputStream.(BoundedByteArrayOutputStream.java:45)
    at org.apache.hadoop.mapreduce.task.reduce.MapOutput.(MergeManager.java:286)
    at org.apache.hadoop.mapreduce.task.reduce.MergeManager.reserve(MergeManager.java:276)
    at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:384)
    at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:319)
    at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:179)

    16:20:33 INFO [main] Job - map 100% reduce 65%
    16:20:36 INFO [main] Job - map 100% reduce 67%
    16:20:39 INFO [main] Job - map 100% reduce 69%
    16:20:41 INFO [main] Job - map 100% reduce 70%
    16:20:43 INFO [main] Job - map 100% reduce 71%

    I have no clue what the issue could be for this. I googled this issue and checked several sources of possible solutions but nothing does fit.

    I saw this jira entry which could fit: https://issues.apache.org/jira/browse/MAPREDUCE-4655.

    Here somebody recommends to increase the value for the property dfs.datanode.max.xcievers / dfs.datanode.max.receiver.threads to 4096, but this is the value for our cluster.
    http://yaseminavcular.blogspot.de/2011/04/common-hadoop-hdfs-exceptions-with.html

    The issue with the to small input files doesn't fit I think, because the map phase reads 137 files with each 130MB. Block Size is 128MB.

    The cluster uses version 2.0.0-cdh4.1.1, 581959ba23e4af85afd8db98b7687662fe9c5f20.

    Thx








    --
    Nitin Pawar


    NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel
  • Alejandro Abdelnur at Nov 5, 2012 at 4:28 pm
    Eduard,

    Would you try using the following properties in your job invocation?

    -D mapreduce.map.java.opts=-Xmx768m -D
    mapreduce.reduce.java.opts=-Xmx768m -D mapreduce.map.memory.mb=2000 -D
    mapreduce.reduce.memory.mb=3000

    Thx

    On Mon, Nov 5, 2012 at 7:43 AM, Kartashov, Andy wrote:
    Your error takes place during reduce task, when temporary files are written
    to memory/disk. You are clearly running low on resources. Check your memory
    “$ free –m” and disk space “$ df –H” as well as “$hadoop fs -df”



    I remember it took me a couple of days to figure out why I was getting heap
    size error and nothing wporked! Becaue, I tried to write 7Gb output file
    onto a disk (in pseudo distr mode) that only had 4Gb of free space.



    p.s. Always test your jobs on small input first (few lines of inputs) .



    p.p.s. follow your job execution through web:
    http://<fully-qualified-hostan-name of your job tracker>:50030





    From: Eduard Skaley
    Sent: Monday, November 05, 2012 4:10 AM
    To: user@hadoop.apache.org
    Cc: Nitin Pawar
    Subject: Re: Error:
    org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError Java Heap Space



    By the way it happens on Yarn not on MRv1

    each container gets 1GB at the moment.

    can you try increasing memory per reducer ?



    On Wed, Oct 31, 2012 at 9:15 PM, Eduard Skaley wrote:

    Hello,

    I'm getting this Error through job execution:

    16:20:26 INFO [main] Job - map 100% reduce 46%
    16:20:27 INFO [main] Job - map 100% reduce 51%
    16:20:29 INFO [main] Job - map 100% reduce 62%
    16:20:30 INFO [main] Job - map 100% reduce 64%
    16:20:32 INFO [main] Job - Task Id :
    attempt_1351680008718_0018_r_000006_0, Status : FAILED
    Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error
    in shuffle in fetcher#2
    at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:123)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:371)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
    Caused by: java.lang.OutOfMemoryError: Java heap space
    at
    org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:58)
    at
    org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:45)
    at
    org.apache.hadoop.mapreduce.task.reduce.MapOutput.<init>(MapOutput.java:97)
    at
    org.apache.hadoop.mapreduce.task.reduce.MergeManager.unconditionalReserve(MergeManager.java:286)
    at
    org.apache.hadoop.mapreduce.task.reduce.MergeManager.reserve(MergeManager.java:276)
    at
    org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:384)
    at
    org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:319)
    at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:179)

    16:20:33 INFO [main] Job - map 100% reduce 65%
    16:20:36 INFO [main] Job - map 100% reduce 67%
    16:20:39 INFO [main] Job - map 100% reduce 69%
    16:20:41 INFO [main] Job - map 100% reduce 70%
    16:20:43 INFO [main] Job - map 100% reduce 71%

    I have no clue what the issue could be for this. I googled this issue and
    checked several sources of possible solutions but nothing does fit.

    I saw this jira entry which could fit:
    https://issues.apache.org/jira/browse/MAPREDUCE-4655.

    Here somebody recommends to increase the value for the property
    dfs.datanode.max.xcievers / dfs.datanode.max.receiver.threads to 4096, but
    this is the value for our cluster.
    http://yaseminavcular.blogspot.de/2011/04/common-hadoop-hdfs-exceptions-with.html

    The issue with the to small input files doesn't fit I think, because the map
    phase reads 137 files with each 130MB. Block Size is 128MB.

    The cluster uses version 2.0.0-cdh4.1.1,
    581959ba23e4af85afd8db98b7687662fe9c5f20.

    Thx









    --
    Nitin Pawar





    NOTICE: This e-mail message and any attachments are confidential, subject to
    copyright and may be privileged. Any unauthorized use, copying or disclosure
    is prohibited. If you are not the intended recipient, please delete and
    contact the sender immediately. Please consider the environment before
    printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui
    l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent
    être couverts par le secret professionnel. Toute utilisation, copie ou
    divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire
    prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur.
    Veuillez penser à l'environnement avant d'imprimer le présent courriel


    --
    Alejandro

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedNov 5, '12 at 9:11a
activeNov 5, '12 at 4:28p
posts4
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase