FAQ
The capacity-scheduler should assign multiple tasks per heartbeat
-----------------------------------------------------------------

Key: HADOOP-5090
URL: https://issues.apache.org/jira/browse/HADOOP-5090
Project: Hadoop Core
Issue Type: Bug
Components: contrib/capacity-sched
Reporter: Arun C Murthy
Priority: Critical


HADOOP-3136 changed the default o.a.h.mapred.JobQueueTaskScheduler to assign multiple tasks per TaskTracker heartbeat, the capacity-scheduler should do the same.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Vinod K V (JIRA) at May 4, 2009 at 10:55 am
    [ https://issues.apache.org/jira/browse/HADOOP-5090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Vinod K V updated HADOOP-5090:
    ------------------------------

    Attachment: HADOOP-5090-20090504.txt

    Attaching patch. With this patch, CapacityTaskScheduler assigns multiple tasks in a single heartbeat. It assigns multiple maps just like the default scheduler - multiple local tasks and at-most one off-switch tasks; and multiple reduces. It also keeps track of the tasks decided to be assigned in a particular scheduling iteration so as that high memory jobs are blocked for scheduling and user-limits are respected while giving away multiple tasks.

    The patch also has test-cases. Benchmarking is in progress.
    The capacity-scheduler should assign multiple tasks per heartbeat
    -----------------------------------------------------------------

    Key: HADOOP-5090
    URL: https://issues.apache.org/jira/browse/HADOOP-5090
    Project: Hadoop Core
    Issue Type: Bug
    Components: contrib/capacity-sched
    Reporter: Arun C Murthy
    Priority: Critical
    Attachments: HADOOP-5090-20090504.txt


    HADOOP-3136 changed the default o.a.h.mapred.JobQueueTaskScheduler to assign multiple tasks per TaskTracker heartbeat, the capacity-scheduler should do the same.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Vinod K V (JIRA) at May 4, 2009 at 10:57 am
    [ https://issues.apache.org/jira/browse/HADOOP-5090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Vinod K V updated HADOOP-5090:
    ------------------------------

    Status: Patch Available (was: Open)
    The capacity-scheduler should assign multiple tasks per heartbeat
    -----------------------------------------------------------------

    Key: HADOOP-5090
    URL: https://issues.apache.org/jira/browse/HADOOP-5090
    Project: Hadoop Core
    Issue Type: Bug
    Components: contrib/capacity-sched
    Reporter: Arun C Murthy
    Priority: Critical
    Attachments: HADOOP-5090-20090504.txt


    HADOOP-3136 changed the default o.a.h.mapred.JobQueueTaskScheduler to assign multiple tasks per TaskTracker heartbeat, the capacity-scheduler should do the same.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Vinod K V (JIRA) at May 4, 2009 at 10:57 am
    [ https://issues.apache.org/jira/browse/HADOOP-5090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Vinod K V reassigned HADOOP-5090:
    ---------------------------------

    Assignee: Vinod K V
    The capacity-scheduler should assign multiple tasks per heartbeat
    -----------------------------------------------------------------

    Key: HADOOP-5090
    URL: https://issues.apache.org/jira/browse/HADOOP-5090
    Project: Hadoop Core
    Issue Type: Bug
    Components: contrib/capacity-sched
    Reporter: Arun C Murthy
    Assignee: Vinod K V
    Priority: Critical
    Attachments: HADOOP-5090-20090504.txt


    HADOOP-3136 changed the default o.a.h.mapred.JobQueueTaskScheduler to assign multiple tasks per TaskTracker heartbeat, the capacity-scheduler should do the same.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at May 5, 2009 at 4:06 am
    [ https://issues.apache.org/jira/browse/HADOOP-5090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705888#action_12705888 ]

    Hadoop QA commented on HADOOP-5090:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12407144/HADOOP-5090-20090504.txt
    against trunk revision 771505.

    +1 @author. The patch does not contain any @author tags.

    +1 tests included. The patch appears to include 3 new or modified tests.

    +1 javadoc. The javadoc tool did not generate any warning messages.

    +1 javac. The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs. The patch does not introduce any new Findbugs warnings.

    +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

    +1 release audit. The applied patch does not increase the total number of release audit warnings.

    +1 core tests. The patch passed core unit tests.

    -1 contrib tests. The patch failed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/288/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/288/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/288/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/288/console

    This message is automatically generated.
    The capacity-scheduler should assign multiple tasks per heartbeat
    -----------------------------------------------------------------

    Key: HADOOP-5090
    URL: https://issues.apache.org/jira/browse/HADOOP-5090
    Project: Hadoop Core
    Issue Type: Bug
    Components: contrib/capacity-sched
    Reporter: Arun C Murthy
    Assignee: Vinod K V
    Priority: Critical
    Attachments: HADOOP-5090-20090504.txt


    HADOOP-3136 changed the default o.a.h.mapred.JobQueueTaskScheduler to assign multiple tasks per TaskTracker heartbeat, the capacity-scheduler should do the same.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Vinod K V (JIRA) at May 6, 2009 at 4:50 am
    [ https://issues.apache.org/jira/browse/HADOOP-5090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Vinod K V updated HADOOP-5090:
    ------------------------------

    Status: Open (was: Patch Available)
    The capacity-scheduler should assign multiple tasks per heartbeat
    -----------------------------------------------------------------

    Key: HADOOP-5090
    URL: https://issues.apache.org/jira/browse/HADOOP-5090
    Project: Hadoop Core
    Issue Type: Bug
    Components: contrib/capacity-sched
    Reporter: Arun C Murthy
    Assignee: Vinod K V
    Priority: Critical
    Attachments: HADOOP-5090-20090504.txt


    HADOOP-3136 changed the default o.a.h.mapred.JobQueueTaskScheduler to assign multiple tasks per TaskTracker heartbeat, the capacity-scheduler should do the same.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Vinod K V (JIRA) at May 6, 2009 at 4:52 am
    [ https://issues.apache.org/jira/browse/HADOOP-5090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Vinod K V updated HADOOP-5090:
    ------------------------------

    Status: Patch Available (was: Open)
    The capacity-scheduler should assign multiple tasks per heartbeat
    -----------------------------------------------------------------

    Key: HADOOP-5090
    URL: https://issues.apache.org/jira/browse/HADOOP-5090
    Project: Hadoop Core
    Issue Type: Bug
    Components: contrib/capacity-sched
    Reporter: Arun C Murthy
    Assignee: Vinod K V
    Priority: Critical
    Attachments: HADOOP-5090-20090504.txt, HADOOP-5090-20090506.txt


    HADOOP-3136 changed the default o.a.h.mapred.JobQueueTaskScheduler to assign multiple tasks per TaskTracker heartbeat, the capacity-scheduler should do the same.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Vinod K V (JIRA) at May 6, 2009 at 4:52 am
    [ https://issues.apache.org/jira/browse/HADOOP-5090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Vinod K V updated HADOOP-5090:
    ------------------------------

    Attachment: HADOOP-5090-20090506.txt

    Updated patch.
    The capacity-scheduler should assign multiple tasks per heartbeat
    -----------------------------------------------------------------

    Key: HADOOP-5090
    URL: https://issues.apache.org/jira/browse/HADOOP-5090
    Project: Hadoop Core
    Issue Type: Bug
    Components: contrib/capacity-sched
    Reporter: Arun C Murthy
    Assignee: Vinod K V
    Priority: Critical
    Attachments: HADOOP-5090-20090504.txt, HADOOP-5090-20090506.txt


    HADOOP-3136 changed the default o.a.h.mapred.JobQueueTaskScheduler to assign multiple tasks per TaskTracker heartbeat, the capacity-scheduler should do the same.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at May 8, 2009 at 8:53 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707523#action_12707523 ]

    Hadoop QA commented on HADOOP-5090:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12407314/HADOOP-5090-20090506.txt
    against trunk revision 772960.

    +1 @author. The patch does not contain any @author tags.

    +1 tests included. The patch appears to include 3 new or modified tests.

    -1 patch. The patch command could not apply the patch.

    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/304/console

    This message is automatically generated.
    The capacity-scheduler should assign multiple tasks per heartbeat
    -----------------------------------------------------------------

    Key: HADOOP-5090
    URL: https://issues.apache.org/jira/browse/HADOOP-5090
    Project: Hadoop Core
    Issue Type: Bug
    Components: contrib/capacity-sched
    Reporter: Arun C Murthy
    Assignee: Vinod K V
    Priority: Critical
    Attachments: HADOOP-5090-20090504.txt, HADOOP-5090-20090506.txt


    HADOOP-3136 changed the default o.a.h.mapred.JobQueueTaskScheduler to assign multiple tasks per TaskTracker heartbeat, the capacity-scheduler should do the same.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Vinod K V (JIRA) at Jun 4, 2009 at 5:13 am
    [ https://issues.apache.org/jira/browse/HADOOP-5090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Vinod K V updated HADOOP-5090:
    ------------------------------

    Status: Open (was: Patch Available)
    The capacity-scheduler should assign multiple tasks per heartbeat
    -----------------------------------------------------------------

    Key: HADOOP-5090
    URL: https://issues.apache.org/jira/browse/HADOOP-5090
    Project: Hadoop Core
    Issue Type: Bug
    Components: contrib/capacity-sched
    Reporter: Arun C Murthy
    Assignee: Vinod K V
    Priority: Critical
    Attachments: HADOOP-5090-20090504.txt, HADOOP-5090-20090506.txt


    HADOOP-3136 changed the default o.a.h.mapred.JobQueueTaskScheduler to assign multiple tasks per TaskTracker heartbeat, the capacity-scheduler should do the same.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Vinod K V (JIRA) at Jun 4, 2009 at 5:13 am
    [ https://issues.apache.org/jira/browse/HADOOP-5090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Vinod K V updated HADOOP-5090:
    ------------------------------

    Attachment: HADOOP-5090-20090604.txt

    Updated patch. This has to be applied over the latest patch for HADOOP-5884.
    The capacity-scheduler should assign multiple tasks per heartbeat
    -----------------------------------------------------------------

    Key: HADOOP-5090
    URL: https://issues.apache.org/jira/browse/HADOOP-5090
    Project: Hadoop Core
    Issue Type: Bug
    Components: contrib/capacity-sched
    Reporter: Arun C Murthy
    Assignee: Vinod K V
    Priority: Critical
    Attachments: HADOOP-5090-20090504.txt, HADOOP-5090-20090506.txt, HADOOP-5090-20090604.txt


    HADOOP-3136 changed the default o.a.h.mapred.JobQueueTaskScheduler to assign multiple tasks per TaskTracker heartbeat, the capacity-scheduler should do the same.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Arun C Murthy (JIRA) at Jun 5, 2009 at 7:06 am
    [ https://issues.apache.org/jira/browse/HADOOP-5090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716526#action_12716526 ]

    Arun C Murthy commented on HADOOP-5090:
    ---------------------------------------

    I'd strongly urge *against* assigning multiple reduces per task. When I did it HADOOP-3136 it caused _bad_ imbalances with reduces... for e.g. consider 2 jobs - one with 'small' reduces and other with 'heavy' reduces. If we assign multiple reduces then a portion of the cluster (tasktrackers) will run the 'small' reduces and the others will run 'heavy' reduces leading to bad imbalances in load on the machine. Given that we decided to assign only 1 reduce per heartbeat wiht HADOOP-3136 to achieve better load balance.
    The capacity-scheduler should assign multiple tasks per heartbeat
    -----------------------------------------------------------------

    Key: HADOOP-5090
    URL: https://issues.apache.org/jira/browse/HADOOP-5090
    Project: Hadoop Core
    Issue Type: Bug
    Components: contrib/capacity-sched
    Reporter: Arun C Murthy
    Assignee: Vinod K V
    Priority: Critical
    Attachments: HADOOP-5090-20090504.txt, HADOOP-5090-20090506.txt, HADOOP-5090-20090604.txt


    HADOOP-3136 changed the default o.a.h.mapred.JobQueueTaskScheduler to assign multiple tasks per TaskTracker heartbeat, the capacity-scheduler should do the same.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Arun C Murthy (JIRA) at Jun 5, 2009 at 6:59 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716723#action_12716723 ]

    Arun C Murthy commented on HADOOP-5090:
    ---------------------------------------

    Also, I *really* don't think having a config to control assignment of one or many tasks is a good idea. We should just stick with multiple assignments.
    The capacity-scheduler should assign multiple tasks per heartbeat
    -----------------------------------------------------------------

    Key: HADOOP-5090
    URL: https://issues.apache.org/jira/browse/HADOOP-5090
    Project: Hadoop Core
    Issue Type: Bug
    Components: contrib/capacity-sched
    Reporter: Arun C Murthy
    Assignee: Vinod K V
    Priority: Critical
    Attachments: HADOOP-5090-20090504.txt, HADOOP-5090-20090506.txt, HADOOP-5090-20090604.txt


    HADOOP-3136 changed the default o.a.h.mapred.JobQueueTaskScheduler to assign multiple tasks per TaskTracker heartbeat, the capacity-scheduler should do the same.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedJan 21, '09 at 6:54a
activeJun 5, '09 at 6:59p
posts13
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Arun C Murthy (JIRA): 13 posts

People

Translate

site design / logo © 2022 Grokbase