FAQ
After deep investigation of specific OOMs, it seems I can't set -Xmx1024m
for MapRed tasks; memory analysis (for Mappers & Reducers) always shows
magic
-Xmx (maximal heap size) is 536870912 bytes

At the same time, for TaskTracker it shows
-Xmx (maximal heap size) is 1048576000 bytes


Strange! Hadoop 0.19.2, with hadoop-site.xml properly replicated in
cluster...
<property>
<name>mapred.child.java.opts</name>
<value>-server -Xms1g -Xmx1g</value>


Has anyone had anything like that? I can't set -Xmx!

-Fuad
http://www.tokenizer.ca

Search Discussions

  • Fuad Efendi at Jan 10, 2010 at 6:42 am
    Sorry guys, I found it... it was hard-coded config in current BIXO framework
    (Cascading-based web crawler)


    Fuad Efendi
    +1 416-993-2060
    http://www.linkedin.com/in/liferay

    Tokenizer Inc.
    http://www.tokenizer.ca/
    Data Mining, Vertical Search

    -----Original Message-----
    From: Fuad Efendi
    Sent: January-10-10 12:31 AM
    To: mapreduce-user@hadoop.apache.org
    Subject: OOM: I can't set -Xmx!


    After deep investigation of specific OOMs, it seems I can't set -
    Xmx1024m
    for MapRed tasks; memory analysis (for Mappers & Reducers) always shows
    magic
    -Xmx (maximal heap size) is 536870912 bytes

    At the same time, for TaskTracker it shows
    -Xmx (maximal heap size) is 1048576000 bytes


    Strange! Hadoop 0.19.2, with hadoop-site.xml properly replicated in
    cluster...
    <property>
    <name>mapred.child.java.opts</name>
    <value>-server -Xms1g -Xmx1g</value>


    Has anyone had anything like that? I can't set -Xmx!

    -Fuad
    http://www.tokenizer.ca
  • Aaron Kimball at Jan 15, 2010 at 6:41 am
    The default mapred.child.java.opts sets to -Xmx200m.

    If you're seeing 512m, then that means that somewhere else you've manually
    configured that setting. Maybe your application itself is calling
    conf.set("mapred.child.java.opts", "-Xmx512m") and overriding your site
    file? (Also, note that this setting will be drawn from the hadoop-site.xml
    file on the client node that submits the job, not on the TT nodes
    themselves.)

    - Aaron
    On Sat, Jan 9, 2010 at 9:31 PM, Fuad Efendi wrote:


    After deep investigation of specific OOMs, it seems I can't set -Xmx1024m
    for MapRed tasks; memory analysis (for Mappers & Reducers) always shows
    magic
    -Xmx (maximal heap size) is 536870912 bytes

    At the same time, for TaskTracker it shows
    -Xmx (maximal heap size) is 1048576000 bytes


    Strange! Hadoop 0.19.2, with hadoop-site.xml properly replicated in
    cluster...
    <property>
    <name>mapred.child.java.opts</name>
    <value>-server -Xms1g -Xmx1g</value>


    Has anyone had anything like that? I can't set -Xmx!

    -Fuad
    http://www.tokenizer.ca


Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedJan 10, '10 at 5:33a
activeJan 15, '10 at 6:41a
posts3
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Fuad Efendi: 2 posts Aaron Kimball: 1 post

People

Translate

site design / logo © 2022 Grokbase