I've run into a situation where it would be helpful to set specific configuration variables local to a data/task node. I've got a solution, but I'm curious if there is a best practice around this and if I'm doing it in a reasonable way.

Basically what we've got is a number of machines that have 2 Cores/4GB of memory. For those boxes we have some options configured higher than the default in mapred-site.xml ( specifically mapred.child.java.opts set to -Xmx1024m ). We recently added a few additional boxes that have 2 Cores/2GB of memory and the mapred.child.java.opts causes those boxes to swap so we'd like to configure those boxes to set mapred.child.java.opts to -Xmx512m. What I found is that if on the data/task node that if I change the value in mapred-site.xml it is overridden, but if I set the parameter using <final>true</final> it is used. So effectively now I've got a different mapred-site.xml on each of the nodes.

My question is, is this a reasonable way of going about this? Is there a best practice for dealing with minor node-specific configuration differences? What I'd really want is not only a mapred-site.xml, but a mapred-node.xml as well that has node specific overrides. What I can't tell is if I make all the site level configuration changes in the name node mapred-site.xml and then the node specific mapred-site.xml files to node local changes if that does what I'm looking for.

Any insight would be appreciated.


Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 1 | next ›
Discussion Overview
groupcommon-user @
postedOct 30, '09 at 7:21p
activeOct 30, '09 at 7:21p

1 user in discussion

Andy Sautins: 1 post



site design / logo © 2022 Grokbase