FAQ
Hi everyone,

Is it possible to configure Hadoop's TaskTracker in a way that it
shutdown itself once being idle (no task to run) for a configurable
amount of time?  I guess it
is not possible in current implementation. This feature is very useful
in situations (like of ours) where a MapReduce jobs can be scheduled
by a batch processing
system.  In other words, the ability to schedule and run a single
MapReduce job on a computing cluster that is also handling
non-MapReduce job and vacating
resources as soon as possible is very desirable for our high
throughput community.

Please let me know If you think this is already possible in Hadoop.
Otherwise, we would be happy to work on contributing a patch to
be included in  Hadoop to server this purpose.

Thanks for your help.

--
Faisal
Condor Project

Search Discussions

  • Allen Wittenauer at Jul 1, 2009 at 8:15 pm
    On 7/1/09 12:51 PM, "Faisal Khan" wrote:
    In other words, the ability to schedule and run a single
    MapReduce job on a computing cluster that is also handling
    non-MapReduce job and vacating
    resources as soon as possible is very desirable for our high
    throughput community.
    Take a look at hod. This is essentially what it does, by letting PBS
    (torque) schedule nodes to run MR and/or HDFS frameworks temporarily.
  • Dhruba Borthakur at Jul 2, 2009 at 12:42 am
    You can also decommission taskTrackers when you know that you do not need
    it anymore.:

    http://issues.apache.org/jira/browse/HADOOP-5643

    thanks,
    dhruba

    On Wed, Jul 1, 2009 at 1:13 PM, Allen Wittenauer wrote:




    On 7/1/09 12:51 PM, "Faisal Khan" wrote:
    In other words, the ability to schedule and run a single
    MapReduce job on a computing cluster that is also handling
    non-MapReduce job and vacating
    resources as soon as possible is very desirable for our high
    throughput community.
    Take a look at hod. This is essentially what it does, by letting PBS
    (torque) schedule nodes to run MR and/or HDFS frameworks temporarily.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJul 1, '09 at 7:51p
activeJul 2, '09 at 12:42a
posts3
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase