[Hadoop-common-user] java.lang.OutOfMemoryError: GC overhead limit exceeded

I'm going to try running it on high-RAM boxes with -Xmx4096m or so,
see if that helps.

On Sun, Sep 26, 2010 at 12:55 AM, Bradford Stephens
wrote:

Nope, that didn't seem to help.

On Sun, Sep 26, 2010 at 1:00 AM, Bradford Stephens
wrote:

Have you tried lowering mapred.job.reuse.jvm.num.tasks ?

Hrm.... no. I've lowered it to -1, but I can try 1 again.

-1 means there is no limit to reusing.
At the same time, you can generate heap dump from OOME and analyze with
YourKit, etc.

Cheers

fwiw

I run m2.xlarge slaves, using the default mappers/reducers (4/2 i think).

with swap
--bootstrap-action s3://elasticmapreduce/bootstrap-actions/create-swap-file.rb --args "-E,/mnt/swap,1000"

historically i'v run this property with no issues, but should probably re-research the gc setting (comments please)
"mapred.child.java.opts", "-server -Xmx2000m -XX:+UseParallelOldGC"

i haven't co-installed ganglia to look at utilization lately, but any more mappers than 4 or more than 2 reducers have always given me headaches.

ckw

The old GC routinely gives me lower performance than modern GC. The default
is now quite good for batch programs.

My feeling is that you have some kind of leak going on in your mappers or
reducers and that reducing the number of times the jvm is re-used would
improve matters.

GC overhead limit indicates that your (tiny) heap is full and collection is
not reducing that.

Sadly, making Chris's changes didn't help.

Here's the Cascading code, it's pretty simple but uses the new
"combiner"-like functionality:

http://pastebin.com/ccvDmLSX

Try using a lower threshold value (the num of values in the LRU to cache). this is the tradeoff of this approach.

ckw

Yup, I've turned it down to 1,000. Let's see if that helps!

Hi Bradford,

Sometimes the reducers do not handle merging large chunks of data too well:
How many reducers do you have? Try to increase the # of reducers (you can
always merge the data later if you are worried about too many output files).

One of the problems with this data set is that I'm grouping by a
category that has only, say, 20 different values. Then I'm doing a
unique count of Facebook user IDs per group. I imagine that's not
pleasant for the reducers.

If there are combiners, the reducers shouldn't get any lists longer than a
small multiple of the number of maps.

It turned out to be a deployment issue of an old version. Ted and
Chris's suggestions were spot-on.

I can't believe how BRILLIANT these combiners from Cascading are. It's
cut my processing time down from 20 hours to 50 minutes. AND I cut out
about 80% of my hand-crafted code.

Bravo. I look smart now. (Almost).

-B

Hi,

"[..]if more than 98% of the total time is spent in garbage collection
and less than 2% of the heap is recovered, an OutOfMemoryError will be
thrown. This feature is designed to prevent applications from running
for an extended period of time while making little or no progress
because the heap is too small. If necessary, this feature can be
disabled by adding the option -XX:-UseGCOverheadLimit to the command
line."

This is what often happens in MapReduce operations when u process a lot of data.
I recommend to try
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx1024m -XX:-UseGCOverheadLimit</value>
</property>


also from my personal experience when process a lot of data often it
is much cheaper to kill JVM rather than wait for GC.
For that reason if you have a lot of BIG tasks rather than tons of
small tasks do not reuse JVM, killing JVM and starting it again often
much cheaper than trying to GC 1GB of ram(don't know why, it just
tuned out in my tests).
<property>
<name>mapred.job.reuse.jvm.num.tasks</name>
<value>1</value>
</description>

Regards,
Vitaliy S

On Sun, Sep 26, 2010 at 11:55 AM, Bradford Stephens
wrote:

Couple of things you can try.
1. Increase the Heap Size for the tasks.

2. Since, your OOM happening randomly, try setting -XX:+HeapDumpOnOutOfMemoryError for your child JVM parameters. Atleast you can detect, why your heap growing -is it due to a leak ? or if you need to increase the heap size for your mappers or reduces from this heap dump analysis.

3. Other reason is due to poor JVM GC tuning. Sometimes, default can't catchup with the garbage created. This needs some GC tuning.

-Bharath




From: [email protected]
To: [email protected]; [email protected]
Cc:
Sent: Sunday, September 26, 2010 12:55:15 AM
Subject: java.lang.OutOfMemoryError: GC overhead limit exceeded

Greetings,

I'm running into a brain-numbing problem on Elastic MapReduce. I'm
running a decent-size task (22,000 mappers, a ton of GZipped input
blocks, ~1TB of data) on 40 c1.xlarge nodes (7 gb RAM, ~8 "cores").

I get failures randomly --- sometimes at the end of my 6-step process,
sometimes at the first reducer phase, sometimes in the mapper. It
seems to fail in multiple areas. Mostly in the reducers. Any ideas?

Here's the settings I've changed:
-Xmx400m
6 max mappers
1 max reducer
1GB swap partition
mapred.job.reuse.jvm.num.tasks=50
mapred.reduce.parallel.copies=3


java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.nio.CharBuffer.wrap(CharBuffer.java:350)
at java.nio.CharBuffer.wrap(CharBuffer.java:373)
at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:138)
at java.lang.StringCoding.decode(StringCoding.java:173)
at java.lang.String.(String.java:443)
at java.lang.String.(String.java:515)
at org.apache.hadoop.io.WritableUtils.readString(WritableUtils.java:116)
at cascading.tuple.TupleInputStream.readString(TupleInputStream.java:144)
at cascading.tuple.TupleInputStream.readType(TupleInputStream.java:154)
at cascading.tuple.TupleInputStream.getNextElement(TupleInputStream.java:101)
at cascading.tuple.hadoop.TupleElementComparator.compare(TupleElementComparator.java:75)
at cascading.tuple.hadoop.TupleElementComparator.compare(TupleElementComparator.java:33)
at cascading.tuple.hadoop.DelegatingTupleElementComparator.compare(DelegatingTupleElementComparator.java:74)
at cascading.tuple.hadoop.DelegatingTupleElementComparator.compare(DelegatingTupleElementComparator.java:34)
at cascading.tuple.hadoop.DeserializerComparator.compareTuples(DeserializerComparator.java:142)
at cascading.tuple.hadoop.GroupingSortingComparator.compare(GroupingSortingComparator.java:55)
at org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:373)
at org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:136)
at org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:103)
at org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:335)
at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:350)
at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:156)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2645)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2586)