Hi to all,
first many thanks for the quality of the work you are doing : thanks a lot

I am facing a bug with the memory management at shuffle time, I regularly get

Map output copy failure : java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1612)


reading the code in org.apache.hadoop.mapred.ReduceTask.java file

the "ShuffleRamManager" is limiting the maximum of RAM allocation to Integer.MAX_VALUE * maxInMemCopyUse ?

maxSize = (int)(conf.getInt("mapred.job.reduce.total.mem.bytes",
(int)Math.min(Runtime.getRuntime().maxMemory(), Integer.MAX_VALUE))
* maxInMemCopyUse);

Why is is so ?
And why is it concatened to an Integer as its raw type is long ?

Does it mean that you can not have a Reduce Task taking advantage of more than 2Gb of memory ?

To explain a little bit my use case,
I am processing some 2700 maps (each working on 128 MB block of data), and when the reduce phase starts, it sometimes stumbles with java heap memory issues.

configuration is : java 1.6.0-27
hadoop 0.20.2
-Xmx1400m
io.sort.mb 400
io.sort.factor 25
io.sort.spill.percent 0.80
mapred.job.shuffle.input.buffer.percent 0.70
ShuffleRamManager: MemoryLimit=913466944, MaxSingleShuffleLimit=228366736

I will decrease
mapred.job.shuffle.input.buffer.percent to limit the errors, but I am not fully confident for the scalability of the process.

Any help would be welcomed

once again, many thanks
Olivier


P.S: sorry if I misunderstood the code, any explanation would be really welcomed

--

Search Discussions

  • Olivier Varene - echo at Dec 6, 2012 at 4:35 pm
    anyone ?

    Début du message réexpédié :
    De : Olivier Varene - echo <varene@echo.fr>
    Objet : ReduceTask > ShuffleRamManager : Java Heap memory error
    Date : 4 décembre 2012 09:34:06 HNEC
    À : mapreduce-user@hadoop.apache.org
    Répondre à : mapreduce-user@hadoop.apache.org


    Hi to all,
    first many thanks for the quality of the work you are doing : thanks a lot

    I am facing a bug with the memory management at shuffle time, I regularly get

    Map output copy failure : java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1612)


    reading the code in org.apache.hadoop.mapred.ReduceTask.java file

    the "ShuffleRamManager" is limiting the maximum of RAM allocation to Integer.MAX_VALUE * maxInMemCopyUse ?

    maxSize = (int)(conf.getInt("mapred.job.reduce.total.mem.bytes",
    (int)Math.min(Runtime.getRuntime().maxMemory(), Integer.MAX_VALUE))
    * maxInMemCopyUse);

    Why is is so ?
    And why is it concatened to an Integer as its raw type is long ?

    Does it mean that you can not have a Reduce Task taking advantage of more than 2Gb of memory ?

    To explain a little bit my use case,
    I am processing some 2700 maps (each working on 128 MB block of data), and when the reduce phase starts, it sometimes stumbles with java heap memory issues.

    configuration is : java 1.6.0-27
    hadoop 0.20.2
    -Xmx1400m
    io.sort.mb 400
    io.sort.factor 25
    io.sort.spill.percent 0.80
    mapred.job.shuffle.input.buffer.percent 0.70
    ShuffleRamManager: MemoryLimit=913466944, MaxSingleShuffleLimit=228366736

    I will decrease
    mapred.job.shuffle.input.buffer.percent to limit the errors, but I am not fully confident for the scalability of the process.

    Any help would be welcomed

    once again, many thanks
    Olivier


    P.S: sorry if I misunderstood the code, any explanation would be really welcomed

    --



  • Arun C Murthy at Dec 6, 2012 at 6:41 pm
    Oliver,

    Sorry, missed this.

    The historical reason, if I remember right, is that we used to have a single byte buffer and hence the limit.

    We should definitely remove it now since we don't use a single buffer. Mind opening a jira?

    http://wiki.apache.org/hadoop/HowToContribute

    thanks!
    Arun
    On Dec 6, 2012, at 8:01 AM, Olivier Varene - echo wrote:

    anyone ?

    Début du message réexpédié :
    De : Olivier Varene - echo <varene@echo.fr>
    Objet : ReduceTask > ShuffleRamManager : Java Heap memory error
    Date : 4 décembre 2012 09:34:06 HNEC
    À : mapreduce-user@hadoop.apache.org
    Répondre à : mapreduce-user@hadoop.apache.org


    Hi to all,
    first many thanks for the quality of the work you are doing : thanks a lot

    I am facing a bug with the memory management at shuffle time, I regularly get

    Map output copy failure : java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1612)


    reading the code in org.apache.hadoop.mapred.ReduceTask.java file

    the "ShuffleRamManager" is limiting the maximum of RAM allocation to Integer.MAX_VALUE * maxInMemCopyUse ?

    maxSize = (int)(conf.getInt("mapred.job.reduce.total.mem.bytes",
    (int)Math.min(Runtime.getRuntime().maxMemory(), Integer.MAX_VALUE))
    * maxInMemCopyUse);

    Why is is so ?
    And why is it concatened to an Integer as its raw type is long ?

    Does it mean that you can not have a Reduce Task taking advantage of more than 2Gb of memory ?

    To explain a little bit my use case,
    I am processing some 2700 maps (each working on 128 MB block of data), and when the reduce phase starts, it sometimes stumbles with java heap memory issues.

    configuration is : java 1.6.0-27
    hadoop 0.20.2
    -Xmx1400m
    io.sort.mb 400
    io.sort.factor 25
    io.sort.spill.percent 0.80
    mapred.job.shuffle.input.buffer.percent 0.70
    ShuffleRamManager: MemoryLimit=913466944, MaxSingleShuffleLimit=228366736

    I will decrease
    mapred.job.shuffle.input.buffer.percent to limit the errors, but I am not fully confident for the scalability of the process.

    Any help would be welcomed

    once again, many thanks
    Olivier


    P.S: sorry if I misunderstood the code, any explanation would be really welcomed

    --



    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/
  • Olivier Varene - echo at Dec 6, 2012 at 10:15 pm
    Yes I will

    thanks for the answer

    regards
    Olivier

    Le 6 déc. 2012 à 19:41, Arun C Murthy a écrit :
    Oliver,

    Sorry, missed this.

    The historical reason, if I remember right, is that we used to have a single byte buffer and hence the limit.

    We should definitely remove it now since we don't use a single buffer. Mind opening a jira?

    http://wiki.apache.org/hadoop/HowToContribute

    thanks!
    Arun
    On Dec 6, 2012, at 8:01 AM, Olivier Varene - echo wrote:

    anyone ?

    Début du message réexpédié :
    De : Olivier Varene - echo <varene@echo.fr>
    Objet : ReduceTask > ShuffleRamManager : Java Heap memory error
    Date : 4 décembre 2012 09:34:06 HNEC
    À : mapreduce-user@hadoop.apache.org
    Répondre à : mapreduce-user@hadoop.apache.org


    Hi to all,
    first many thanks for the quality of the work you are doing : thanks a lot

    I am facing a bug with the memory management at shuffle time, I regularly get

    Map output copy failure : java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1612)


    reading the code in org.apache.hadoop.mapred.ReduceTask.java file

    the "ShuffleRamManager" is limiting the maximum of RAM allocation to Integer.MAX_VALUE * maxInMemCopyUse ?

    maxSize = (int)(conf.getInt("mapred.job.reduce.total.mem.bytes",
    (int)Math.min(Runtime.getRuntime().maxMemory(), Integer.MAX_VALUE))
    * maxInMemCopyUse);

    Why is is so ?
    And why is it concatened to an Integer as its raw type is long ?

    Does it mean that you can not have a Reduce Task taking advantage of more than 2Gb of memory ?

    To explain a little bit my use case,
    I am processing some 2700 maps (each working on 128 MB block of data), and when the reduce phase starts, it sometimes stumbles with java heap memory issues.

    configuration is : java 1.6.0-27
    hadoop 0.20.2
    -Xmx1400m
    io.sort.mb 400
    io.sort.factor 25
    io.sort.spill.percent 0.80
    mapred.job.shuffle.input.buffer.percent 0.70
    ShuffleRamManager: MemoryLimit=913466944, MaxSingleShuffleLimit=228366736

    I will decrease
    mapred.job.shuffle.input.buffer.percent to limit the errors, but I am not fully confident for the scalability of the process.

    Any help would be welcomed

    once again, many thanks
    Olivier


    P.S: sorry if I misunderstood the code, any explanation would be really welcomed

    --



    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/
  • Olivier Varene - echo at Dec 10, 2012 at 2:23 pm
    here is the Jira issue,

    and the beginning of a patch


    https://issues.apache.org/jira/browse/MAPREDUCE-4866


    there is indeed a limitation on the byte array size (around Integer.MAX_VALUE)

    Maybe we could use BigArrays to overcome this limitation ?

    What do you think ?

    regards
    Olivier

    Le 6 déc. 2012 à 19:41, Arun C Murthy a écrit :
    Oliver,

    Sorry, missed this.

    The historical reason, if I remember right, is that we used to have a single byte buffer and hence the limit.

    We should definitely remove it now since we don't use a single buffer. Mind opening a jira?

    http://wiki.apache.org/hadoop/HowToContribute

    thanks!
    Arun
    On Dec 6, 2012, at 8:01 AM, Olivier Varene - echo wrote:

    anyone ?

    Début du message réexpédié :
    De : Olivier Varene - echo <varene@echo.fr>
    Objet : ReduceTask > ShuffleRamManager : Java Heap memory error
    Date : 4 décembre 2012 09:34:06 HNEC
    À : mapreduce-user@hadoop.apache.org
    Répondre à : mapreduce-user@hadoop.apache.org


    Hi to all,
    first many thanks for the quality of the work you are doing : thanks a lot

    I am facing a bug with the memory management at shuffle time, I regularly get

    Map output copy failure : java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1612)


    reading the code in org.apache.hadoop.mapred.ReduceTask.java file

    the "ShuffleRamManager" is limiting the maximum of RAM allocation to Integer.MAX_VALUE * maxInMemCopyUse ?

    maxSize = (int)(conf.getInt("mapred.job.reduce.total.mem.bytes",
    (int)Math.min(Runtime.getRuntime().maxMemory(), Integer.MAX_VALUE))
    * maxInMemCopyUse);

    Why is is so ?
    And why is it concatened to an Integer as its raw type is long ?

    Does it mean that you can not have a Reduce Task taking advantage of more than 2Gb of memory ?

    To explain a little bit my use case,
    I am processing some 2700 maps (each working on 128 MB block of data), and when the reduce phase starts, it sometimes stumbles with java heap memory issues.

    configuration is : java 1.6.0-27
    hadoop 0.20.2
    -Xmx1400m
    io.sort.mb 400
    io.sort.factor 25
    io.sort.spill.percent 0.80
    mapred.job.shuffle.input.buffer.percent 0.70
    ShuffleRamManager: MemoryLimit=913466944, MaxSingleShuffleLimit=228366736

    I will decrease
    mapred.job.shuffle.input.buffer.percent to limit the errors, but I am not fully confident for the scalability of the process.

    Any help would be welcomed

    once again, many thanks
    Olivier


    P.S: sorry if I misunderstood the code, any explanation would be really welcomed

    --



    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedDec 4, '12 at 8:34a
activeDec 10, '12 at 2:23p
posts5
users2
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase