[Hadoop-common-dev] [jira] Created: (HADOOP-5726) Remove pre-emption from the capacity scheduler code base

[ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

rahul k singh updated HADOOP-5726:
----------------------------------

Attachment: cs-without-preemption-23-1-2009.patch

Attaching the patch .

[ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12702304#action_12702304 ]

Arun C Murthy commented on HADOOP-5726:
---------------------------------------

+1 for the direction.

I'd propose we take this opportunity to further improve the capacity-scheduler by simplifying it's design, refactoring inner-classes into standalone classes, using the JobChangeEvents more extensively rather than re-sort queues on every heartbeat etc. Maybe a new jira?

Thoughts?

[ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704667#action_12704667 ]

rahul k singh commented on HADOOP-5726:
---------------------------------------

[exec]
[exec]
[exec]
[exec]
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 7 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
[exec]
[exec]
[exec]
[exec] ======================================================================
[exec] ======================================================================
[exec] Finished build.
[exec] ======================================================================
[exec] ======================================================================

[ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

rahul k singh updated HADOOP-5726:
----------------------------------

Status: Patch Available (was: Open)

[ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705591#action_12705591 ]

Hemanth Yamijala commented on HADOOP-5726:
------------------------------------------

bq. I'd propose we take this opportunity to further improve the capacity-scheduler by simplifying it's design, refactoring inner-classes into standalone classes, using the JobChangeEvents more extensively rather than re-sort queues on every heartbeat etc. Maybe a new jira?

Arun, I agree. Particularly with the refactoring of inner classes into standalone classes. Also, I think we should look at combining all queue state into a single class and promote it to a first class object. Today, this state is split between JobQueueManager and QueueSchedulingInfo.

Can you please open a new jira with your ideas and we can discuss there ?

[ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705592#action_12705592 ]

Hemanth Yamijala commented on HADOOP-5726:
------------------------------------------

At this point, it looks like there's no objection to the removal of this code. I plan to commit this soon.

On a related note, though this is a major change, it appears that committing this to the 0.20 branch will significantly simplify further development. Is there an objection to this ?

[ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-5726:
-------------------------------------

Resolution: Fixed
Fix Version/s: 0.20.1
Release Note: Removed pre-emption from capacity scheduler. The impact of this change is that capacities for queues can no longer be guaranteed within a given span of time. Also changed configuration variables to remove pre-emption related variables and better reflect the absence of guarantees.
Hadoop Flags: [Incompatible change, Reviewed]
Status: Resolved (was: Patch Available)

I just committed this to trunk and branch 0.20. Thanks, Rahul !

[ https://issues.apache.org/jira/browse/HADOOP-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12706152#action_12706152 ]

Hudson commented on HADOOP-5726:
--------------------------------

Integrated in Hadoop-trunk #827 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/827/])
. Remove pre-emption from capacity scheduler code base. Contributed by Rahul Kumar Singh.