FAQ
Remove pre-emption from the capacity scheduler code base
--------------------------------------------------------

Key: HADOOP-5726
URL: https://issues.apache.org/jira/browse/HADOOP-5726
Project: Hadoop Core
Issue Type: Improvement
Components: contrib/capacity-sched
Reporter: Hemanth Yamijala
Assignee: rahul k singh


In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • rahul k singh (JIRA) at Apr 23, 2009 at 10:19 am
    [ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    rahul k singh updated HADOOP-5726:
    ----------------------------------

    Attachment: cs-without-preemption-23-1-2009.patch

    Attaching the patch .

    Remove pre-emption from the capacity scheduler code base
    --------------------------------------------------------

    Key: HADOOP-5726
    URL: https://issues.apache.org/jira/browse/HADOOP-5726
    Project: Hadoop Core
    Issue Type: Improvement
    Components: contrib/capacity-sched
    Reporter: Hemanth Yamijala
    Assignee: rahul k singh
    Attachments: cs-without-preemption-23-1-2009.patch


    In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Arun C Murthy (JIRA) at Apr 24, 2009 at 10:00 am
    [ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12702304#action_12702304 ]

    Arun C Murthy commented on HADOOP-5726:
    ---------------------------------------

    +1 for the direction.

    I'd propose we take this opportunity to further improve the capacity-scheduler by simplifying it's design, refactoring inner-classes into standalone classes, using the JobChangeEvents more extensively rather than re-sort queues on every heartbeat etc. Maybe a new jira?

    Thoughts?
    Remove pre-emption from the capacity scheduler code base
    --------------------------------------------------------

    Key: HADOOP-5726
    URL: https://issues.apache.org/jira/browse/HADOOP-5726
    Project: Hadoop Core
    Issue Type: Improvement
    Components: contrib/capacity-sched
    Reporter: Hemanth Yamijala
    Assignee: rahul k singh
    Attachments: cs-without-preemption-23-1-2009.patch


    In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • rahul k singh (JIRA) at Apr 30, 2009 at 3:33 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704667#action_12704667 ]

    rahul k singh commented on HADOOP-5726:
    ---------------------------------------

    [exec]
    [exec]
    [exec]
    [exec]
    [exec] +1 overall.
    [exec]
    [exec] +1 @author. The patch does not contain any @author tags.
    [exec]
    [exec] +1 tests included. The patch appears to include 7 new or modified tests.
    [exec]
    [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
    [exec]
    [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
    [exec]
    [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
    [exec]
    [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
    [exec]
    [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
    [exec]
    [exec]
    [exec]
    [exec]
    [exec] ======================================================================
    [exec] ======================================================================
    [exec] Finished build.
    [exec] ======================================================================
    [exec] ======================================================================
    Remove pre-emption from the capacity scheduler code base
    --------------------------------------------------------

    Key: HADOOP-5726
    URL: https://issues.apache.org/jira/browse/HADOOP-5726
    Project: Hadoop Core
    Issue Type: Improvement
    Components: contrib/capacity-sched
    Reporter: Hemanth Yamijala
    Assignee: rahul k singh
    Attachments: cs-without-preemption-23-1-2009.patch


    In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • rahul k singh (JIRA) at Apr 30, 2009 at 3:34 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    rahul k singh updated HADOOP-5726:
    ----------------------------------

    Status: Patch Available (was: Open)
    Remove pre-emption from the capacity scheduler code base
    --------------------------------------------------------

    Key: HADOOP-5726
    URL: https://issues.apache.org/jira/browse/HADOOP-5726
    Project: Hadoop Core
    Issue Type: Improvement
    Components: contrib/capacity-sched
    Reporter: Hemanth Yamijala
    Assignee: rahul k singh
    Attachments: cs-without-preemption-23-1-2009.patch


    In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hemanth Yamijala (JIRA) at May 4, 2009 at 1:34 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705591#action_12705591 ]

    Hemanth Yamijala commented on HADOOP-5726:
    ------------------------------------------

    bq. I'd propose we take this opportunity to further improve the capacity-scheduler by simplifying it's design, refactoring inner-classes into standalone classes, using the JobChangeEvents more extensively rather than re-sort queues on every heartbeat etc. Maybe a new jira?

    Arun, I agree. Particularly with the refactoring of inner classes into standalone classes. Also, I think we should look at combining all queue state into a single class and promote it to a first class object. Today, this state is split between JobQueueManager and QueueSchedulingInfo.

    Can you please open a new jira with your ideas and we can discuss there ?
    Remove pre-emption from the capacity scheduler code base
    --------------------------------------------------------

    Key: HADOOP-5726
    URL: https://issues.apache.org/jira/browse/HADOOP-5726
    Project: Hadoop Core
    Issue Type: Improvement
    Components: contrib/capacity-sched
    Reporter: Hemanth Yamijala
    Assignee: rahul k singh
    Attachments: cs-without-preemption-23-1-2009.patch


    In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hemanth Yamijala (JIRA) at May 4, 2009 at 1:36 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705592#action_12705592 ]

    Hemanth Yamijala commented on HADOOP-5726:
    ------------------------------------------

    At this point, it looks like there's no objection to the removal of this code. I plan to commit this soon.

    On a related note, though this is a major change, it appears that committing this to the 0.20 branch will significantly simplify further development. Is there an objection to this ?
    Remove pre-emption from the capacity scheduler code base
    --------------------------------------------------------

    Key: HADOOP-5726
    URL: https://issues.apache.org/jira/browse/HADOOP-5726
    Project: Hadoop Core
    Issue Type: Improvement
    Components: contrib/capacity-sched
    Reporter: Hemanth Yamijala
    Assignee: rahul k singh
    Attachments: cs-without-preemption-23-1-2009.patch


    In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hemanth Yamijala (JIRA) at May 5, 2009 at 7:41 am
    [ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Hemanth Yamijala updated HADOOP-5726:
    -------------------------------------

    Resolution: Fixed
    Fix Version/s: 0.20.1
    Release Note: Removed pre-emption from capacity scheduler. The impact of this change is that capacities for queues can no longer be guaranteed within a given span of time. Also changed configuration variables to remove pre-emption related variables and better reflect the absence of guarantees.
    Hadoop Flags: [Incompatible change, Reviewed]
    Status: Resolved (was: Patch Available)

    I just committed this to trunk and branch 0.20. Thanks, Rahul !
    Remove pre-emption from the capacity scheduler code base
    --------------------------------------------------------

    Key: HADOOP-5726
    URL: https://issues.apache.org/jira/browse/HADOOP-5726
    Project: Hadoop Core
    Issue Type: Improvement
    Components: contrib/capacity-sched
    Reporter: Hemanth Yamijala
    Assignee: rahul k singh
    Fix For: 0.20.1

    Attachments: cs-without-preemption-23-1-2009.patch


    In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hudson (JIRA) at May 5, 2009 at 7:09 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12706152#action_12706152 ]

    Hudson commented on HADOOP-5726:
    --------------------------------

    Integrated in Hadoop-trunk #827 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/827/])
    . Remove pre-emption from capacity scheduler code base. Contributed by Rahul Kumar Singh.

    Remove pre-emption from the capacity scheduler code base
    --------------------------------------------------------

    Key: HADOOP-5726
    URL: https://issues.apache.org/jira/browse/HADOOP-5726
    Project: Hadoop Core
    Issue Type: Improvement
    Components: contrib/capacity-sched
    Reporter: Hemanth Yamijala
    Assignee: rahul k singh
    Fix For: 0.20.1

    Attachments: cs-without-preemption-23-1-2009.patch


    In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedApr 23, '09 at 7:52a
activeMay 5, '09 at 7:09p
posts9
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Hudson (JIRA): 9 posts

People

Translate

site design / logo © 2022 Grokbase