FAQ
Hi all,

I have been using the Hadoop Fair Scheduler for some experiments on a
100 node cluster with 2 map slots per node (hence, a total of 200 map
slots).

In one of my experiments, all the map tasks finish within a heartbeat
interval of 3 seconds. I noticed that the maximum number of
concurrently
active map slots on my cluster never exceeds 100, and hence, the
cluster utilization during my experiments never exceeds 50% even when
large jobs with more than a 1000 maps are being executed.

A look at the Fair Scheduler code (in particular, the assignTasks
function) revealed the reason.
As per my understanding, with the implementation in Hadoop 0.20.0, a
TaskTracker is not assigned more than 1 map and 1 reduce task per
heart beat.

In my experiments, in every heart beat, each TT has 2 free map slots
but is assigned only 1 map task, and hence, the utilization never goes
beyond 50%.

Of course, this (degenerate) case does not arise when map tasks take
more than one 1 heart beat interval to finish. For example, I repeated
the experiments with maps tasks taking close to 15 s to finish and
noticed close to 100 % utilization when large jobs were executing.

Why does the Fair Scheduler not assign more than one map task to a TT
per heart beat? Is this done to spread the load uniformly across the
cluster?
I looked at assignTasks function in the default Hadoop scheduler
(JobQueueTaskScheduler.java), and it does assign more than 1 map task
per heart beat to a TT.

It will be easy to change the Fair Scheduler to assign more than 1 map
task to a TT per heart beat (I did that and achieved 100% utilization
even with small map tasks). But I am wondering, if doing so will
violate some fairness properties.

Thanks,
Abhishek

Search Discussions

  • Todd Lipcon at Apr 11, 2010 at 7:58 pm
    Hi Abhishek,

    This behavior is improved by MAPREDUCE-706 I believe (not certain that
    that's the JIRA, but I know it's fixed in trunk fairscheduler). These
    patches are included in CDH3 (currently in beta)
    http://archive.cloudera.com/cdh/3/

    In general, though, map tasks that are so short are not going to be very
    efficient - even with fast assignment there is some constant overhead per
    task.

    Thanks
    -Todd
    On Sun, Apr 11, 2010 at 11:42 AM, abhishek sharma wrote:

    Hi all,

    I have been using the Hadoop Fair Scheduler for some experiments on a
    100 node cluster with 2 map slots per node (hence, a total of 200 map
    slots).

    In one of my experiments, all the map tasks finish within a heartbeat
    interval of 3 seconds. I noticed that the maximum number of
    concurrently
    active map slots on my cluster never exceeds 100, and hence, the
    cluster utilization during my experiments never exceeds 50% even when
    large jobs with more than a 1000 maps are being executed.

    A look at the Fair Scheduler code (in particular, the assignTasks
    function) revealed the reason.
    As per my understanding, with the implementation in Hadoop 0.20.0, a
    TaskTracker is not assigned more than 1 map and 1 reduce task per
    heart beat.

    In my experiments, in every heart beat, each TT has 2 free map slots
    but is assigned only 1 map task, and hence, the utilization never goes
    beyond 50%.

    Of course, this (degenerate) case does not arise when map tasks take
    more than one 1 heart beat interval to finish. For example, I repeated
    the experiments with maps tasks taking close to 15 s to finish and
    noticed close to 100 % utilization when large jobs were executing.

    Why does the Fair Scheduler not assign more than one map task to a TT
    per heart beat? Is this done to spread the load uniformly across the
    cluster?
    I looked at assignTasks function in the default Hadoop scheduler
    (JobQueueTaskScheduler.java), and it does assign more than 1 map task
    per heart beat to a TT.

    It will be easy to change the Fair Scheduler to assign more than 1 map
    task to a TT per heart beat (I did that and achieved 100% utilization
    even with small map tasks). But I am wondering, if doing so will
    violate some fairness properties.

    Thanks,
    Abhishek


    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Ted Yu at Apr 11, 2010 at 9:51 pm
    Reading assignTasks() in 0.20.2 reveals that the number of map tasks
    assigned is not limited to 1 per heartbeat.

    Cheers
    On Sun, Apr 11, 2010 at 12:30 PM, Todd Lipcon wrote:

    Hi Abhishek,

    This behavior is improved by MAPREDUCE-706 I believe (not certain that
    that's the JIRA, but I know it's fixed in trunk fairscheduler). These
    patches are included in CDH3 (currently in beta)
    http://archive.cloudera.com/cdh/3/

    In general, though, map tasks that are so short are not going to be very
    efficient - even with fast assignment there is some constant overhead per
    task.

    Thanks
    -Todd
    On Sun, Apr 11, 2010 at 11:42 AM, abhishek sharma wrote:

    Hi all,

    I have been using the Hadoop Fair Scheduler for some experiments on a
    100 node cluster with 2 map slots per node (hence, a total of 200 map
    slots).

    In one of my experiments, all the map tasks finish within a heartbeat
    interval of 3 seconds. I noticed that the maximum number of
    concurrently
    active map slots on my cluster never exceeds 100, and hence, the
    cluster utilization during my experiments never exceeds 50% even when
    large jobs with more than a 1000 maps are being executed.

    A look at the Fair Scheduler code (in particular, the assignTasks
    function) revealed the reason.
    As per my understanding, with the implementation in Hadoop 0.20.0, a
    TaskTracker is not assigned more than 1 map and 1 reduce task per
    heart beat.

    In my experiments, in every heart beat, each TT has 2 free map slots
    but is assigned only 1 map task, and hence, the utilization never goes
    beyond 50%.

    Of course, this (degenerate) case does not arise when map tasks take
    more than one 1 heart beat interval to finish. For example, I repeated
    the experiments with maps tasks taking close to 15 s to finish and
    noticed close to 100 % utilization when large jobs were executing.

    Why does the Fair Scheduler not assign more than one map task to a TT
    per heart beat? Is this done to spread the load uniformly across the
    cluster?
    I looked at assignTasks function in the default Hadoop scheduler
    (JobQueueTaskScheduler.java), and it does assign more than 1 map task
    per heart beat to a TT.

    It will be easy to change the Fair Scheduler to assign more than 1 map
    task to a TT per heart beat (I did that and achieved 100% utilization
    even with small map tasks). But I am wondering, if doing so will
    violate some fairness properties.

    Thanks,
    Abhishek


    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Abhishek sharma at Apr 11, 2010 at 9:58 pm
    Hi Ted,

    I was referring to version 0.20.0. As Todd pointed out, the issue I
    pointed out was fixed in version 0.20.2.

    I only looked at the Cloudera version 0.20.2+228
    (http://archive.cloudera.com/cdh/3/) currently in beta.

    I guess Hadoop 0.20.2 also has the fix. I will take a look at that too.

    Thanks,
    Abhishek
    On Sun, Apr 11, 2010 at 2:51 PM, Ted Yu wrote:
    Reading assignTasks() in 0.20.2 reveals that the number of map tasks
    assigned is not limited to 1 per heartbeat.

    Cheers
    On Sun, Apr 11, 2010 at 12:30 PM, Todd Lipcon wrote:

    Hi Abhishek,

    This behavior is improved by MAPREDUCE-706 I believe (not certain that
    that's the JIRA, but I know it's fixed in trunk fairscheduler). These
    patches are included in CDH3 (currently in beta)
    http://archive.cloudera.com/cdh/3/

    In general, though, map tasks that are so short are not going to be very
    efficient - even with fast assignment there is some constant overhead per
    task.

    Thanks
    -Todd

    On Sun, Apr 11, 2010 at 11:42 AM, abhishek sharma <absharma@usc.edu>
    wrote:
    Hi all,

    I have been using the Hadoop Fair Scheduler for some experiments on a
    100 node cluster with 2 map slots per node (hence, a total of 200 map
    slots).

    In one of my experiments, all the map tasks finish within a heartbeat
    interval of 3 seconds. I noticed that the maximum number of
    concurrently
    active map slots on my cluster never exceeds 100, and hence, the
    cluster utilization during my experiments never exceeds 50% even when
    large jobs with more than a 1000 maps are being executed.

    A look at the Fair Scheduler code (in particular, the assignTasks
    function) revealed the reason.
    As per my understanding, with the implementation in Hadoop 0.20.0, a
    TaskTracker is not assigned more than 1 map and 1 reduce task per
    heart beat.

    In my experiments, in every heart beat, each TT has 2 free map slots
    but is assigned only 1 map task, and hence, the utilization never goes
    beyond 50%.

    Of course, this (degenerate) case does not arise when map tasks take
    more than one 1 heart beat interval to finish. For example, I repeated
    the experiments with maps tasks taking close to 15 s to finish and
    noticed close to 100 % utilization when large jobs were executing.

    Why does the Fair Scheduler not assign more than one map task to a TT
    per heart beat? Is this done to spread the load uniformly across the
    cluster?
    I looked at assignTasks function in the default Hadoop scheduler
    (JobQueueTaskScheduler.java), and it does assign more than 1 map task
    per heart beat to a TT.

    It will be easy to change the Fair Scheduler to assign more than 1 map
    task to a TT per heart beat (I did that and achieved 100% utilization
    even with small map tasks). But I am wondering, if doing so will
    violate some fairness properties.

    Thanks,
    Abhishek


    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Abhishek sharma at Apr 14, 2010 at 6:21 am
    Hi Ted,

    Were you referring to the Hadoop 0.20.2 distribution or the CDH version?

    I just looked at the FairScheduler assignTasks function in Hadoop
    dist. 0.20.2 and it is the same as version 0.20.0, and it will assign
    only 1 Map and 1 reduce task to a tasktracker per heartbeat (as far I
    can tell by reading the code and my experiments).

    Abhishek


    On Sun, Apr 11, 2010 at 2:51 PM, Ted Yu wrote:
    Reading assignTasks() in 0.20.2 reveals that the number of map tasks
    assigned is not limited to 1 per heartbeat.

    Cheers
    On Sun, Apr 11, 2010 at 12:30 PM, Todd Lipcon wrote:

    Hi Abhishek,

    This behavior is improved by MAPREDUCE-706 I believe (not certain that
    that's the JIRA, but I know it's fixed in trunk fairscheduler). These
    patches are included in CDH3 (currently in beta)
    http://archive.cloudera.com/cdh/3/

    In general, though, map tasks that are so short are not going to be very
    efficient - even with fast assignment there is some constant overhead per
    task.

    Thanks
    -Todd

    On Sun, Apr 11, 2010 at 11:42 AM, abhishek sharma <absharma@usc.edu>
    wrote:
    Hi all,

    I have been using the Hadoop Fair Scheduler for some experiments on a
    100 node cluster with 2 map slots per node (hence, a total of 200 map
    slots).

    In one of my experiments, all the map tasks finish within a heartbeat
    interval of 3 seconds. I noticed that the maximum number of
    concurrently
    active map slots on my cluster never exceeds 100, and hence, the
    cluster utilization during my experiments never exceeds 50% even when
    large jobs with more than a 1000 maps are being executed.

    A look at the Fair Scheduler code (in particular, the assignTasks
    function) revealed the reason.
    As per my understanding, with the implementation in Hadoop 0.20.0, a
    TaskTracker is not assigned more than 1 map and 1 reduce task per
    heart beat.

    In my experiments, in every heart beat, each TT has 2 free map slots
    but is assigned only 1 map task, and hence, the utilization never goes
    beyond 50%.

    Of course, this (degenerate) case does not arise when map tasks take
    more than one 1 heart beat interval to finish. For example, I repeated
    the experiments with maps tasks taking close to 15 s to finish and
    noticed close to 100 % utilization when large jobs were executing.

    Why does the Fair Scheduler not assign more than one map task to a TT
    per heart beat? Is this done to spread the load uniformly across the
    cluster?
    I looked at assignTasks function in the default Hadoop scheduler
    (JobQueueTaskScheduler.java), and it does assign more than 1 map task
    per heart beat to a TT.

    It will be easy to change the Fair Scheduler to assign more than 1 map
    task to a TT per heart beat (I did that and achieved 100% utilization
    even with small map tasks). But I am wondering, if doing so will
    violate some fairness properties.

    Thanks,
    Abhishek


    --
    Todd Lipcon
    Software Engineer, Cloudera

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedApr 11, '10 at 7:45p
activeApr 14, '10 at 6:21a
posts5
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase