FAQ
Despite plenty of available mapper slots (as inferred by "Map Task Capacity" minus "Maps" on the jobtracker) our jobs often linger with all or many of the map tasks stuck in a pending status for five to ten minutes. Eventually they start, but why the seemingly unnecessary delay? There also might be a correlation between starting an unrelated job to "unstick" the pending tasks of the previous job...although this correlation is difficult to quantify; it may be a coincidence.

Does anyone have a theory as to what is going on here?

Thanks.

________________________________________________________________________________
Keith Wiley kwiley@keithwiley.com keithwiley.com music.keithwiley.com

"Luminous beings are we, not this crude matter."
-- Yoda
________________________________________________________________________________

Search Discussions

  • Harsh J at Mar 15, 2011 at 4:43 pm
    [ Moving this to mapreduce-user@ - since it isn't a common-development
    related question, but a mapreduce specific one || bcc-ing common-dev@
    ]
    On Tue, Mar 15, 2011 at 1:25 PM, Keith Wiley wrote:
    Despite plenty of available mapper slots (as inferred by "Map Task Capacity" minus "Maps" on the jobtracker) our jobs often linger with all or many of the map tasks stuck in a pending status for five to ten minutes.  Eventually they start, but why the seemingly unnecessary delay?  There also might be a correlation between starting an unrelated job to "unstick" the pending tasks of the previous job...although this correlation is difficult to quantify; it may be a coincidence.
    Could we additionally know what scheduler is in effect here, and does
    the log of JobTracker contain anything you think may be remotely
    related to this issue?

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMar 15, '11 at 1:33p
activeMar 15, '11 at 4:43p
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Harsh J: 1 post Keith Wiley: 1 post

People

Translate

site design / logo © 2022 Grokbase