FAQ
Hi All:

I have mapred.tasktracker.tasks.maximum set to 4 in our conf/hadoop-site.xml, yet I frequently see 5-6 instances of org.apache.hadoop.mapred.TaskTracker$Child running on the slave nodes. Is there another setting I need to tweak in order to dial back the number of children running? The effect of running this many children is that our boxes have extremely high load factors, and eventually mapred tasks start timing out and failing.

Note that the number of instances is for a single job. I see far more if I run multiple jobs simultaneously (something we do not typically do).

This is on Hadoop 0.15.0, upgrading is not an option at the moment.

Any help appreciate...
Thanks,
C G

Search Discussions

  • Amareshwari Sriramadasu at Jun 19, 2008 at 11:19 am

    C G wrote:
    Hi All:

    I have mapred.tasktracker.tasks.maximum set to 4 in our conf/hadoop-site.xml, yet I frequently see 5-6 instances of org.apache.hadoop.mapred.TaskTracker$Child running on the slave nodes. Is there another setting I need to tweak in order to dial back the number of children running? The effect of running this many children is that our boxes have extremely high load factors, and eventually mapred tasks start timing out and failing.
    If mapred.tasktracker.tasks.maximum is set to four, the tasktracker has
    4 map slots and 4 reduce slots, summing up to 8 slots. Then seeing 5-6
    instances of org.apache.hadoop.mapred.TaskTracker$Child is expected. If
    you want only 4 instances of it, mapred.tasktracker.tasks.maximum
    should be 2. thus making 2 map slots and 2 reduce slots.
    And as far as I know there is no other config variable for tweaking the
    number of children.
    Note that the number of instances is for a single job. I see far more if I run multiple jobs simultaneously (something we do not typically do).

    This is on Hadoop 0.15.0, upgrading is not an option at the moment.

    Any help appreciate...
    Thanks,
    C G

    Thanks
    Amareshwari

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJun 19, '08 at 11:01a
activeJun 19, '08 at 11:19a
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Amareshwari Sriramadasu: 1 post C G: 1 post

People

Translate

site design / logo © 2022 Grokbase