FAQ
My cluster nodes have 2 dual-core processors, so, in general, I want
to configure my nodes with a maximum of 3 task processes executed per
node at a time.

But, for some jobs, my tasks load large amounts of memory, and I
cannot fit 3 such tasks on a single node. For these jobs, I'd like to
enforce running a maximum of 1 task process per node at a time.

I've tried to enforce this by setting
mapred.tasktracker.map.tasks.maximum at runtime, but I see it has no
effect, because this is a configuration for the TaskTracker, which is
of course already running before my job starts.

Is there no way to configure a maximum # of map tasks per node on a
per-job basis?

Thanks!

Search Discussions

  • S D at Mar 14, 2009 at 2:05 am
    I ran into this problem as well and several people on this list provided a
    helpful response: once the tasktracker starts, the maximum number of tasks
    per node can not be changed. In my case, I've solved this challenge by
    stopping and starting mapred (stop-mapred.sh, start-mapred.sh) between jobs.
    There is a jira so this may be changed in the future: jira HADOOP-5170 (
    http://issues.apache.org/jira/browse/HADOOP-5170)

    John
    On Fri, Mar 13, 2009 at 9:47 PM, Stuart White wrote:

    My cluster nodes have 2 dual-core processors, so, in general, I want
    to configure my nodes with a maximum of 3 task processes executed per
    node at a time.

    But, for some jobs, my tasks load large amounts of memory, and I
    cannot fit 3 such tasks on a single node. For these jobs, I'd like to
    enforce running a maximum of 1 task process per node at a time.

    I've tried to enforce this by setting
    mapred.tasktracker.map.tasks.maximum at runtime, but I see it has no
    effect, because this is a configuration for the TaskTracker, which is
    of course already running before my job starts.

    Is there no way to configure a maximum # of map tasks per node on a
    per-job basis?

    Thanks!

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMar 14, '09 at 1:47a
activeMar 14, '09 at 2:05a
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

S D: 1 post Stuart White: 1 post

People

Translate

site design / logo © 2023 Grokbase