Grokbase › Groups › Hadoop › common-user › April 2009
FAQ
I'm trying to set a System Property in the Hadoop config, so my jobs will know which cluster they are running on. I think I should be able to do this with -Dname=value in "mapred.child.java.opts" (example below), but the setting is ignored.
In hadoop-site.xml I have:
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx200m -Dfoo=test</value>
</property>
But the job conf through the web server indicates:
mapred.child.java.opts -Xmx1024M -Duser.timezone=UTC

I'm using Hadoop-0.17.2.1.
Any tips on why my setting is not picked up?

Marc

________________________________
PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE, DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.

Search Discussions

  • Aaron Kimball at Apr 30, 2009 at 2:26 pm ⇧
    So you want a different "-Dfoo=test" on each node? It's probably grabbing
    the setting from the node where the job was submitted, and this overrides
    the settings on each task node.

    Try adding <final>true</final> to the <property> block on the tasktrackers,
    then restart Hadoop and try again. This will prevent the job from overriding
    the setting.

    - Aaron
    On Thu, Apr 30, 2009 at 9:25 AM, Marc Limotte wrote:

    I'm trying to set a System Property in the Hadoop config, so my jobs will
    know which cluster they are running on. I think I should be able to do this
    with -Dname=value in "mapred.child.java.opts" (example below), but the
    setting is ignored.
    In hadoop-site.xml I have:
    <property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx200m -Dfoo=test</value>
    </property>
    But the job conf through the web server indicates:
    mapred.child.java.opts -Xmx1024M -Duser.timezone=UTC

    I'm using Hadoop-0.17.2.1.
    Any tips on why my setting is not picked up?

    Marc

    ________________________________
    PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR
    ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION
    PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE,
    DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY
    PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND
    PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.
  • Tom White at Apr 30, 2009 at 2:34 pm ⇧
    Another way to do this would be to set a property in the Hadoop config itself.

    In the job launcher you would have something like:

    JobConf conf = ...
    conf.setProperty("foo", "test");

    Then you can read the property in your map or reduce task.

    Tom
    On Thu, Apr 30, 2009 at 3:25 PM, Aaron Kimball wrote:
    So you want a different "-Dfoo=test" on each node? It's probably grabbing
    the setting from the node where the job was submitted, and this overrides
    the settings on each task node.

    Try adding <final>true</final> to the <property> block on the tasktrackers,
    then restart Hadoop and try again. This will prevent the job from overriding
    the setting.

    - Aaron
    On Thu, Apr 30, 2009 at 9:25 AM, Marc Limotte wrote:

    I'm trying to set a System Property in the Hadoop config, so my jobs will
    know which cluster they are running on.  I think I should be able to do this
    with -Dname=value in "mapred.child.java.opts" (example below), but the
    setting is ignored.
    In hadoop-site.xml I have:
    <property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx200m -Dfoo=test</value>
    </property>
    But the job conf through the web server indicates:
    mapred.child.java.opts -Xmx1024M -Duser.timezone=UTC

    I'm using Hadoop-0.17.2.1.
    Any tips on why my setting is not picked up?

    Marc

    ________________________________
    PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR
    ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION
    PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE,
    DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY
    PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND
    PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedApr 30, '09 at 12:26a
activeApr 30, '09 at 2:34p
posts3
users3
websitehadoop.apache.org...
irc#hadoop

3 users in discussion

Aaron Kimball: 1 post Marc Limotte: 1 post Tom White: 1 post

Content

People

Support

Translate

site design / logo © 2023 Grokbase