FAQ
Hi to all,
first many thansk for the quality of the work you are doing : thanks a lot

I am facing a bug with the memory management at shuffle time, I regularly
get

Map output copy failure : java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1612)



reading the code in org.apache.hadoop.mapred.ReduceTask.java file

the "ShuffleRamManager" is limiting the maximum of RAM allocation to
Integer.MAX_VALUE * maxInMemCopyUse ?

maxSize = (int)(conf.getInt("mapred.job.reduce.total.mem.bytes",
(int)Math.min(Runtime.getRuntime().maxMemory(),
Integer.MAX_VALUE))
* maxInMemCopyUse);

Why is is so ?
And why is it concatened to an Integer as its raw type is long ?

Does it mean that you can not have a Reduce Task taking advantage of more
than 2Gb of memory ?

To explain a little bit my use case,
I am processing some 2700 maps (each working on 128 MB block of data), and
when the reduce phase starts, it sometimes stumbles with java heap memory
issues.

configuration is : java 1.6.0-27
cloudera cdh3u5 418
-Xmx1400m

io.sort.mb 400
io.sort.factor 25
io.sort.spill.percent 0.80
mapred.job.shuffle.input.buffer.percent 0.70
ShuffleRamManager: MemoryLimit=913466944, MaxSingleShuffleLimit=228366736

map output is compressed with Snappy

I will decrease
mapred.job.shuffle.input.buffer.percent to limit the errors, but I am not fully confident for the scalability of the process.

Any help would be welcomed

once again, many thanks
Olivier




sorry if I misunderstood the code, any explanation would be really welcomed

--

Search Discussions

  • Sandy Ryza at Dec 11, 2012 at 12:34 am
    Hi Olivier,

    That is correct that the reduce side of shuffling can not take advantage of
    more than 2 Gb if memory.

    What is set for mapred.reduce.parallel.copies? This is the number of
    threads used to pull data from mappers concurrently. Also, how many
    reducers are you running?

    -Sandy
    On Mon, Dec 3, 2012 at 7:26 AM, ov wrote:

    Hi to all,
    first many thansk for the quality of the work you are doing : thanks a lot

    I am facing a bug with the memory management at shuffle time, I regularly
    get

    Map output copy failure : java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1612)



    reading the code in org.apache.hadoop.mapred.ReduceTask.java file

    the "ShuffleRamManager" is limiting the maximum of RAM allocation to
    Integer.MAX_VALUE * maxInMemCopyUse ?

    maxSize = (int)(conf.getInt("mapred.job.reduce.total.mem.bytes",
    (int)Math.min(Runtime.getRuntime().maxMemory(),
    Integer.MAX_VALUE))
    * maxInMemCopyUse);

    Why is is so ?
    And why is it concatened to an Integer as its raw type is long ?

    Does it mean that you can not have a Reduce Task taking advantage of more
    than 2Gb of memory ?

    To explain a little bit my use case,
    I am processing some 2700 maps (each working on 128 MB block of data), and
    when the reduce phase starts, it sometimes stumbles with java heap memory
    issues.

    configuration is : java 1.6.0-27
    cloudera cdh3u5 418
    -Xmx1400m

    io.sort.mb 400
    io.sort.factor 25
    io.sort.spill.percent 0.80
    mapred.job.shuffle.input.buffer.percent 0.70
    ShuffleRamManager: MemoryLimit=913466944, MaxSingleShuffleLimit=228366736

    map output is compressed with Snappy

    I will decrease
    mapred.job.shuffle.input.buffer.percent to limit the errors, but I am not fully confident for the scalability of the process.

    Any help would be welcomed

    once again, many thanks
    Olivier




    sorry if I misunderstood the code, any explanation would be really welcomed

    --


    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcdh-user @
categorieshadoop
postedDec 3, '12 at 3:34p
activeDec 11, '12 at 12:34a
posts2
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Sandy Ryza: 1 post Ov: 1 post

People

Translate

site design / logo © 2018 Grokbase