FAQ
Hi,

The optimization of one Hadoop job I'm running would benefit from knowing
the
maximum number of map slots in the Hadoop cluster.

This number can be obtained (if my understanding is correct) by:

* parsing the mapred-site.xml file to get
the mapred.tasktracker.map.tasks.maximum value (assuming it is set of
course)

* parsing the slaves file to get the maximum number of compute nodes in the
cluster

* multiplying the 2 values

My question is:
I would like to learn about *all* possible ways to get this information
through API calls (either the Hadoop Common API or the Hadoop MapReduce
API), i.e. obtaining it through a Job object, through a Configuration
object,...

Thanks in advance and have a great day,

Cyril

Search Discussions

  • Hemanth Yamijala at Sep 5, 2010 at 9:42 am
    Hi,
    The optimization of one Hadoop job I'm running would benefit from knowing
    the
    maximum number of map slots in the Hadoop cluster.

    This number can be obtained (if my understanding is correct) by:

    * parsing the mapred-site.xml file to get
    the mapred.tasktracker.map.tasks.maximum value (assuming it is set of
    course)

    * parsing the slaves file to get the maximum number of compute nodes in the
    cluster

    * multiplying the 2 values

    My question is:
    I would like to learn about *all* possible ways to get this information
    through API calls (either the Hadoop Common API or the Hadoop MapReduce
    API), i.e. obtaining it through a Job object, through a Configuration
    object,...
    The easiest way I can think of is using
    o.a.h.m.ClusterStatus.getMaxMapTasks(). You can get an instance to
    ClusterStatus using JobClient.getClusterStatus().

    Thanks
    hemanth

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedSep 3, '10 at 3:47p
activeSep 5, '10 at 9:42a
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Cyril Briquet: 1 post Hemanth Yamijala: 1 post

People

Translate

site design / logo © 2022 Grokbase