FAQ
With fair scheduler, long running jobs can easily occurpy a lot of task slots
-----------------------------------------------------------------------------

Key: HADOOP-5701
URL: https://issues.apache.org/jira/browse/HADOOP-5701
Project: Hadoop Core
Issue Type: Bug
Components: contrib/fair-share
Reporter: Tsz Wo (Nicholas), SZE


Current fair scheduler implementation favor long running jobs since once a task slot is assigned to a job, the fair scheduler is not able to reclaim it.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Tsz Wo (Nicholas), SZE (JIRA) at Apr 17, 2009 at 6:54 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700284#action_12700284 ]

    Tsz Wo (Nicholas), SZE commented on HADOOP-5701:
    ------------------------------------------------

    Consider that a cluster have 2000 map slots and jobs submitted in the following sequence:
    1:00pm|JobA|1500 maps, each map runs 24 hours|
    1:30pm|JobB|1000 maps, each map runs 2 hours|
    1:40pm|JobC|3000 maps, each map runs 10 minutes|
    Then, all 1500 maps in JobA got scheduled and only 500 map slots remained in the cluster at 1pm. 30 minutes later, JobB came and only 500 maps slots got scheduled. At 1:40pm, JobC came but no maps got scheduled until some maps in JobB finished 2 hours later.

    In this cases, JobA always has 75% of the capacity, JobB and JobC never able to obtain 1/N of the capacity. If JobA has 2000 maps, other jobs have to wait for maps in JobA to finish and have no progress in 24 hours.
    With fair scheduler, long running jobs can easily occurpy a lot of task slots
    -----------------------------------------------------------------------------

    Key: HADOOP-5701
    URL: https://issues.apache.org/jira/browse/HADOOP-5701
    Project: Hadoop Core
    Issue Type: Bug
    Components: contrib/fair-share
    Reporter: Tsz Wo (Nicholas), SZE

    Current fair scheduler implementation favor long running jobs since once a task slot is assigned to a job, the fair scheduler is not able to reclaim it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hong Tang (JIRA) at Apr 23, 2009 at 8:11 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12702084#action_12702084 ]

    Hong Tang commented on HADOOP-5701:
    -----------------------------------

    In other words, fair share scheduler cannot really allocate fair shares without preemption.
    With fair scheduler, long running jobs can easily occurpy a lot of task slots
    -----------------------------------------------------------------------------

    Key: HADOOP-5701
    URL: https://issues.apache.org/jira/browse/HADOOP-5701
    Project: Hadoop Core
    Issue Type: Bug
    Components: contrib/fair-share
    Reporter: Tsz Wo (Nicholas), SZE

    Current fair scheduler implementation favor long running jobs since once a task slot is assigned to a job, the fair scheduler is not able to reclaim it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Matei Zaharia (JIRA) at Jun 11, 2009 at 11:36 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718666#action_12718666 ]

    Matei Zaharia commented on HADOOP-5701:
    ---------------------------------------

    HADOOP-4665 will prevent this from happening. Do you want any additional features on top of that?
    With fair scheduler, long running jobs can easily occurpy a lot of task slots
    -----------------------------------------------------------------------------

    Key: HADOOP-5701
    URL: https://issues.apache.org/jira/browse/HADOOP-5701
    Project: Hadoop Core
    Issue Type: Bug
    Components: contrib/fair-share
    Reporter: Tsz Wo (Nicholas), SZE

    Current fair scheduler implementation favor long running jobs since once a task slot is assigned to a job, the fair scheduler is not able to reclaim it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Jun 12, 2009 at 12:09 am
    [ https://issues.apache.org/jira/browse/HADOOP-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Tsz Wo (Nicholas), SZE resolved HADOOP-5701.
    --------------------------------------------

    Resolution: Duplicate

    Matei, thank you for fixing the problem. We should close this as duplicated.
    With fair scheduler, long running jobs can easily occurpy a lot of task slots
    -----------------------------------------------------------------------------

    Key: HADOOP-5701
    URL: https://issues.apache.org/jira/browse/HADOOP-5701
    Project: Hadoop Core
    Issue Type: Bug
    Components: contrib/fair-share
    Reporter: Tsz Wo (Nicholas), SZE

    Current fair scheduler implementation favor long running jobs since once a task slot is assigned to a job, the fair scheduler is not able to reclaim it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Matei Zaharia (JIRA) at Jun 12, 2009 at 6:48 am
    [ https://issues.apache.org/jira/browse/HADOOP-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718740#action_12718740 ]

    Matei Zaharia commented on HADOOP-5701:
    ---------------------------------------

    HADOOP-4665 is not committed yet, but when it is, it should solve the problem.
    With fair scheduler, long running jobs can easily occurpy a lot of task slots
    -----------------------------------------------------------------------------

    Key: HADOOP-5701
    URL: https://issues.apache.org/jira/browse/HADOOP-5701
    Project: Hadoop Core
    Issue Type: Bug
    Components: contrib/fair-share
    Reporter: Tsz Wo (Nicholas), SZE

    Current fair scheduler implementation favor long running jobs since once a task slot is assigned to a job, the fair scheduler is not able to reclaim it.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedApr 17, '09 at 6:50p
activeJun 12, '09 at 6:48a
posts6
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Matei Zaharia (JIRA): 6 posts

People

Translate

site design / logo © 2022 Grokbase