Hi to all,
first many thansk for the quality of the work you are doing : thanks a lot
I am facing a bug with the memory management at shuffle time, I regularly
Map output copy failure : java.lang.OutOfMemoryError: Java heap space
reading the code in org.apache.hadoop.mapred.ReduceTask.java file
the "ShuffleRamManager" is limiting the maximum of RAM allocation to
Integer.MAX_VALUE * maxInMemCopyUse ?
maxSize = (int)(conf.getInt("mapred.job.reduce.total.mem.bytes",
Why is is so ?
And why is it concatened to an Integer as its raw type is long ?
Does it mean that you can not have a Reduce Task taking advantage of more
than 2Gb of memory ?
To explain a little bit my use case,
I am processing some 2700 maps (each working on 128 MB block of data), and
when the reduce phase starts, it sometimes stumbles with java heap memory
configuration is : java 1.6.0-27
cloudera cdh3u5 418
ShuffleRamManager: MemoryLimit=913466944, MaxSingleShuffleLimit=228366736
map output is compressed with Snappy
I will decrease
mapred.job.shuffle.input.buffer.percent to limit the errors, but I am not fully confident for the scalability of the process.
Any help would be welcomed
once again, many thanks
sorry if I misunderstood the code, any explanation would be really welcomed