FAQ
Guys,

Is it possible for a fair scheduler pool config to have a high upper bound
capacity for a couple of pools but those pools not be able to fully reach
their specified capacity due to the some unseen mechanic?

I have been POCing a use case on a moderate cluster and I have 2 classes of
pools 1) ad-hoc access 2) controlled ETLs (all of which are routed based on
${user.name}).

Pool <class ad-hoc>
Mappers: 2min/60max
Reducers: 2min/30max
Jobs: 10max
Weight: 1.0

Pool <class ETL>
Mappers: 150min/300max
Reducers: 150min/300max
Jobs: 300max
Weight: 4.0

We are seeing an issue where the the class ETL capacity never gets above
around 30-40 concurrent jobs even though each job load is over 5-10 mins
(so no chance they are finishing so quickly that we can't fill up the pool).

Regards,
Jerry

--

Search Discussions

  • Todd Lipcon at Nov 15, 2012 at 5:57 pm
    Hi Jerry,

    If you're able to reproduce this, can you grab
    http://jobtracker:50030/scheduler?advanced and paste it here? The fair
    scheduler pool metrics from /metrics or /jmx would also be helpful. They'll
    show the calculated demand, fair share, etc.

    -Todd
    On Wed, Nov 14, 2012 at 4:30 PM, Jerry Pascalii wrote:

    Guys,

    Is it possible for a fair scheduler pool config to have a high upper bound
    capacity for a couple of pools but those pools not be able to fully reach
    their specified capacity due to the some unseen mechanic?

    I have been POCing a use case on a moderate cluster and I have 2 classes
    of pools 1) ad-hoc access 2) controlled ETLs (all of which are routed based
    on ${user.name}).

    Pool <class ad-hoc>
    Mappers: 2min/60max
    Reducers: 2min/30max
    Jobs: 10max
    Weight: 1.0

    Pool <class ETL>
    Mappers: 150min/300max
    Reducers: 150min/300max
    Jobs: 300max
    Weight: 4.0

    We are seeing an issue where the the class ETL capacity never gets above
    around 30-40 concurrent jobs even though each job load is over 5-10 mins
    (so no chance they are finishing so quickly that we can't fill up the pool).

    Regards,
    Jerry

    --




    --
    Todd Lipcon
    Software Engineer, Cloudera

    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcdh-user @
categorieshadoop
postedNov 15, '12 at 12:30a
activeNov 15, '12 at 5:57p
posts2
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Todd Lipcon: 1 post Jerry Pascalii: 1 post

People

Translate

site design / logo © 2022 Grokbase