FAQ
Code to create the UI display string for queues in the Capacity Scheduler needs to be synchronized, and needs to better update its information
----------------------------------------------------------------------------------------------------------------------------------------------

Key: HADOOP-4984
URL: https://issues.apache.org/jira/browse/HADOOP-4984
Project: Hadoop Core
Issue Type: Bug
Components: contrib/capacity-sched
Reporter: Vivek Ratan


There are a couple of problems with _SchedulingInfo.toString()_, the code which creates the UI display string for a queue:
* it needs synchronized access to the _QueueSchedulingInfo_ object, as this same object can be updated by the reclaim-capacity thread, and during a heartbeat.
* the code directly updates its count of running map/reduce tasks. this should be done in a better way, perhaps by calling updateQSIObjects(), rather than walking through the data structures directly. It's also not clear that we want to pay the performance penalty of updating the structures. it maybe OK to provide slightly stale info (the 'staleness' is tiny, in a steady-state and large system, where heartbeats are coming in frequently).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Vivek Ratan (JIRA) at Jan 6, 2009 at 5:50 am
    [ https://issues.apache.org/jira/browse/HADOOP-4984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661050#action_12661050 ]

    Vivek Ratan commented on HADOOP-4984:
    -------------------------------------

    The refactoring work in HADOOP-4980 fixes this problem. While I realize that we don't usually want to clump together more than one fix in a patch, the work in HADOOP-4980 went a long long way in simplifying the fix for this patch, so I didn't create a separate patch here. By making the _SchedulingInfo_ class be functionally 'outside' the scheduler class, and thus unaware of the latter's data structures, and by moving the generation of the display strings to the concerned _QueueSchedulingInfo_ and _TaskSchedulingInfo_ objects, the synchronization problem is easily addressed. we also don't update the QSI objects, preferring to show potentially slightly stale information, but without a performance penalty.
    Code to create the UI display string for queues in the Capacity Scheduler needs to be synchronized, and needs to better update its information
    ----------------------------------------------------------------------------------------------------------------------------------------------

    Key: HADOOP-4984
    URL: https://issues.apache.org/jira/browse/HADOOP-4984
    Project: Hadoop Core
    Issue Type: Bug
    Components: contrib/capacity-sched
    Reporter: Vivek Ratan

    There are a couple of problems with _SchedulingInfo.toString()_, the code which creates the UI display string for a queue:
    * it needs synchronized access to the _QueueSchedulingInfo_ object, as this same object can be updated by the reclaim-capacity thread, and during a heartbeat.
    * the code directly updates its count of running map/reduce tasks. this should be done in a better way, perhaps by calling updateQSIObjects(), rather than walking through the data structures directly. It's also not clear that we want to pay the performance penalty of updating the structures. it maybe OK to provide slightly stale info (the 'staleness' is tiny, in a steady-state and large system, where heartbeats are coming in frequently).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedJan 6, '09 at 5:44a
activeJan 6, '09 at 5:50a
posts2
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Vivek Ratan (JIRA): 2 posts

People

Translate

site design / logo © 2022 Grokbase