FAQ
Another way to do this would be to set a property in the Hadoop config itself.

In the job launcher you would have something like:

JobConf conf = ...
conf.setProperty("foo", "test");

Then you can read the property in your map or reduce task.

Tom
On Thu, Apr 30, 2009 at 3:25 PM, Aaron Kimball wrote:
So you want a different "-Dfoo=test" on each node? It's probably grabbing
the setting from the node where the job was submitted, and this overrides
the settings on each task node.

Try adding <final>true</final> to the <property> block on the tasktrackers,
then restart Hadoop and try again. This will prevent the job from overriding
the setting.

- Aaron
On Thu, Apr 30, 2009 at 9:25 AM, Marc Limotte wrote:

I'm trying to set a System Property in the Hadoop config, so my jobs will
know which cluster they are running on.  I think I should be able to do this
with -Dname=value in "mapred.child.java.opts" (example below), but the
setting is ignored.
In hadoop-site.xml I have:
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx200m -Dfoo=test</value>
</property>
But the job conf through the web server indicates:
mapred.child.java.opts -Xmx1024M -Duser.timezone=UTC

I'm using Hadoop-0.17.2.1.
Any tips on why my setting is not picked up?

Marc

________________________________
PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR
ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION
PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE,
DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY
PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND
PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.

Search Discussions

Discussion Posts

Previous

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 3 of 3 | next ›
Discussion Overview
groupcommon-user @
categorieshadoop
postedApr 30, '09 at 12:26a
activeApr 30, '09 at 2:34p
posts3
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase