FAQ
Hello all

Is it possible for a Hadoop program to override
mapred.tasktracker.tasks.maximum at runtime? I've found that my job
overloads our nodes when running our default 8 tasks per node, but if
I decrease mapred.tasktracker.tasks.maximum in the config file and
restart the cluster it seems to go fine. However, when I tried to set
this parameter in my program it seemed to be ignored. I'd like to set
this at runtime rather than having to restart the cluster. Is this
only read when tasktrackers are started or can I override it?

Thanks!

-steve

Search Discussions

  • Jason Venner at Feb 7, 2008 at 5:01 pm
    This should be one of the features coming in 0.16 via HOD

    Steve Schlosser wrote:
    Hello all

    Is it possible for a Hadoop program to override
    mapred.tasktracker.tasks.maximum at runtime? I've found that my job
    overloads our nodes when running our default 8 tasks per node, but if
    I decrease mapred.tasktracker.tasks.maximum in the config file and
    restart the cluster it seems to go fine. However, when I tried to set
    this parameter in my program it seemed to be ignored. I'd like to set
    this at runtime rather than having to restart the cluster. Is this
    only read when tasktrackers are started or can I override it?

    Thanks!

    -steve
  • Arun C Murthy at Feb 7, 2008 at 10:51 pm

    On Feb 7, 2008, at 7:23 AM, Steve Schlosser wrote:

    Hello all

    Is it possible for a Hadoop program to override
    mapred.tasktracker.tasks.maximum at runtime? I've found that my job
    overloads our nodes when running our default 8 tasks per node, but if
    I decrease mapred.tasktracker.tasks.maximum in the config file and
    restart the cluster it seems to go fine. However, when I tried to set
    this parameter in my program it seemed to be ignored.
    No, mapred.tasktracker.tasks.maximum (mapred.tasktracker.{map|reduce}
    tasks.maximum in 0.16.0) is a property of the cluster and not a job-
    specific property. Hence you cannot override it.

    As Jason mentioned, you could use Haddop-On-Demand (a resource
    utilization framework coming in 0.16.0) to launch a new cluster per
    job and then configure each job (and hence cluster) differently...

    Arun
    I'd like to set
    this at runtime rather than having to restart the cluster. Is this
    only read when tasktrackers are started or can I override it?

    Thanks!

    -steve

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedFeb 7, '08 at 3:24p
activeFeb 7, '08 at 10:51p
posts3
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2023 Grokbase