FAQ
This is technically a java issue but I thought other hadoop users would find
it interesting.

When we upgraded from old jvm to a newer (32bit) jvm of 1.6.0_21, we started
seeing users' tasks having issues at random places

1. Tasks running 2-3 times slower
2. Tasks failing with OutOfMemory
3. Reducer failing with OutOfMemory while pulling map outpus

Our first response was to increase the heapsize but what actually was
happening was that NewRatio has changed from 8 to 2 after we upgraded jvm to
1.6.0_21.

http://forums.oracle.com/forums/thread.jspa?threadID=2167755
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6887571

So with -Xmx640m, older jvm (1.6.0_17 or before)

old generation went up to

PS Old Generation
capacity = 596574208 (568.9375MB)
used = 596574200 (568.9374923706055MB)
free = 8 (7.62939453125E-6MB)
99.99999865901007% used

but newer jvm(1.6.0_21) only went up to

PS Old Generation
capacity = 447414272 (426.6875MB)
used = 447414264 (426.68749237060547MB)
free = 8 (7.62939453125E-6MB)
99.99999821194797% used


causing all sorts of issues. To bring back the old behavior, we needed to
add -XX:NewRatio=8 to jvm param.

If you have MAPREDUCE-1207 in your version, you can set
<property>
<name>mapreduce.admin.map.child.java.opts</name>
<value>-XX:NewRatio=8</value>
</property>

<property>
<name>mapreduce.admin.reduce.child.java.opts</name>
<value>-XX:NewRatio=8</value>
</property>

Otherwise, you can set it on 'mapred.child.java.opts' and ask users to also
set it whenever they overwrite that conf.

Maybe this is a known issue to everyone but thought I would share it since
we spent so much time on this simple(?) issue :)

You can check the NewRatio value by running jmap -histo on the jvm.

Koji

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedApr 8, '11 at 9:11p
activeApr 8, '11 at 9:11p
posts1
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Koji Noguchi: 1 post

People

Translate

site design / logo © 2022 Grokbase