FAQ
All - I really need help/pointers to setup Fair Scheduler in CDH4. Any help
would be greatly appreciated. Thanks Willy

--

Search Discussions

  • Harsh J at Aug 10, 2012 at 11:08 am
    Hi Willy,

    The instructions at
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html should
    still apply to MR1. Is this what you're looking for?
    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang wrote:
    All - I really need help/pointers to setup Fair Scheduler in CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J

    --
  • Willy Chang at Aug 13, 2012 at 7:39 pm
    Thanks Harsh! We're using MR2. Can u send me the link for that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html should
    still apply to MR1. Is this what you're looking for?
    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang wrote:
    All - I really need help/pointers to setup Fair Scheduler in CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --
  • Willy Chang at Aug 13, 2012 at 10:12 pm
    Also, where do i download the fair scheduler jar file for mr2 & CDH4.
    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang wrote:

    Thanks Harsh! We're using MR2. Can u send me the link for that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    All - I really need help/pointers to setup Fair Scheduler in CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --
  • Santhosh Srinivasan at Aug 14, 2012 at 1:05 am
    Hi Willy,

    Fair scheduler was added to MR2 last month (see -
    https://issues.apache.org/jira/browse/MAPREDUCE-3451). It will be available
    in CDH-4.1

    Thanks,
    Santhosh
    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang wrote:

    Also, where do i download the fair scheduler jar file for mr2 & CDH4.

    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang wrote:

    Thanks Harsh! We're using MR2. Can u send me the link for that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at
    http://archive.cloudera.com/**cdh4/cdh/4/mr1/fair_scheduler.**html<http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html>should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    All - I really need help/pointers to setup Fair Scheduler in CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --
  • Willy Chang at Aug 14, 2012 at 4:54 pm
    But can someone please post the link where to get the fair scheduler jar
    for MR2 CDH4? Thanks!
    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh Srinivasan wrote:

    Hi Willy,

    Fair scheduler was added to MR2 last month (see -
    https://issues.apache.org/jira/browse/MAPREDUCE-3451). It will be
    available in CDH-4.1

    Thanks,
    Santhosh

    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang <willy.c...@gmail.com<javascript:>
    wrote:
    Also, where do i download the fair scheduler jar file for mr2 & CDH4.

    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang wrote:

    Thanks Harsh! We're using MR2. Can u send me the link for that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at
    http://archive.cloudera.com/**cdh4/cdh/4/mr1/fair_scheduler.**html<http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html>should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    All - I really need help/pointers to setup Fair Scheduler in CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --
  • Santhosh Srinivasan at Aug 15, 2012 at 5:51 pm
    Hi Willy,

    At this point in time, we have not published any artifacts for Fair
    Scheduler in MR2 for public consumption. Before releasing features,
    Cloudera gives a lot of importance to testing out the various components to
    safeguard customers from using untested code. Further, given that CDH 4.1
    will be released soon, it might be best to wait for CDH4.1.

    Thanks,
    Santhosh
    On Tue, Aug 14, 2012 at 9:54 AM, Willy Chang wrote:

    But can someone please post the link where to get the fair scheduler jar
    for MR2 CDH4? Thanks!

    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh Srinivasan wrote:

    Hi Willy,

    Fair scheduler was added to MR2 last month (see -
    https://issues.apache.org/**jira/browse/MAPREDUCE-3451<https://issues.apache.org/jira/browse/MAPREDUCE-3451>).
    It will be available in CDH-4.1

    Thanks,
    Santhosh
    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang wrote:

    Also, where do i download the fair scheduler jar file for mr2 & CDH4.

    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang wrote:

    Thanks Harsh! We're using MR2. Can u send me the link for that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at
    http://archive.cloudera.com/**cd**h4/cdh/4/mr1/fair_scheduler.**ht**ml<http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html>should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    All - I really need help/pointers to setup Fair Scheduler in CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --

    --
  • Daniel Oxenhandler at Oct 23, 2012 at 8:36 pm
    Santhosh,

    Is there any update on this? we are running CDH 4.1 and would like to use
    fair scheduler. Is this available somewhere now? I checked the release
    notes for CDH4.1.1 and there is no mention of fair scheduler.

    Thanks!
    -Daniel
    On Wednesday, August 15, 2012 10:51:25 AM UTC-7, Santhosh Srinivasan wrote:

    Hi Willy,

    At this point in time, we have not published any artifacts for Fair
    Scheduler in MR2 for public consumption. Before releasing features,
    Cloudera gives a lot of importance to testing out the various components to
    safeguard customers from using untested code. Further, given that CDH 4.1
    will be released soon, it might be best to wait for CDH4.1.

    Thanks,
    Santhosh

    On Tue, Aug 14, 2012 at 9:54 AM, Willy Chang <willy.c...@gmail.com<javascript:>
    wrote:
    But can someone please post the link where to get the fair scheduler jar
    for MR2 CDH4? Thanks!

    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh Srinivasan wrote:

    Hi Willy,

    Fair scheduler was added to MR2 last month (see -
    https://issues.apache.org/**jira/browse/MAPREDUCE-3451<https://issues.apache.org/jira/browse/MAPREDUCE-3451>).
    It will be available in CDH-4.1

    Thanks,
    Santhosh
    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang wrote:

    Also, where do i download the fair scheduler jar file for mr2 & CDH4.

    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang wrote:

    Thanks Harsh! We're using MR2. Can u send me the link for that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at
    http://archive.cloudera.com/**cd**h4/cdh/4/mr1/fair_scheduler.**ht**
    ml <http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html>should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    All - I really need help/pointers to setup Fair Scheduler in CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --

    --
  • Willy Chang at Oct 24, 2012 at 6:54 pm
    Santhosh, Can you please send any artifacts for Fair Scheduler in MR2 at
    willy.chang.ic@gmail.com? We need this urgently. Thanks!
    On Tuesday, October 23, 2012 1:36:54 PM UTC-7, Daniel Oxenhandler wrote:

    Santhosh,

    Is there any update on this? we are running CDH 4.1 and would like to use
    fair scheduler. Is this available somewhere now? I checked the release
    notes for CDH4.1.1 and there is no mention of fair scheduler.

    Thanks!
    -Daniel
    On Wednesday, August 15, 2012 10:51:25 AM UTC-7, Santhosh Srinivasan wrote:

    Hi Willy,

    At this point in time, we have not published any artifacts for Fair
    Scheduler in MR2 for public consumption. Before releasing features,
    Cloudera gives a lot of importance to testing out the various components to
    safeguard customers from using untested code. Further, given that CDH
    4.1 will be released soon, it might be best to wait for CDH4.1.

    Thanks,
    Santhosh
    On Tue, Aug 14, 2012 at 9:54 AM, Willy Chang wrote:

    But can someone please post the link where to get the fair scheduler jar
    for MR2 CDH4? Thanks!

    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh Srinivasan wrote:

    Hi Willy,

    Fair scheduler was added to MR2 last month (see -
    https://issues.apache.org/**jira/browse/MAPREDUCE-3451<https://issues.apache.org/jira/browse/MAPREDUCE-3451>).
    It will be available in CDH-4.1

    Thanks,
    Santhosh
    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang wrote:

    Also, where do i download the fair scheduler jar file for mr2 & CDH4.

    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang wrote:

    Thanks Harsh! We're using MR2. Can u send me the link for that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at
    http://archive.cloudera.com/**cd**h4/cdh/4/mr1/fair_scheduler.**ht**
    ml <http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html>should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    All - I really need help/pointers to setup Fair Scheduler in CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --

    --
  • Santhosh Srinivasan at Oct 25, 2012 at 5:34 pm
    Hi Willy,

    The fair share scheduler functionality has been ported in part in CDH4.1.
    Its missing the Web UI (with metrics), making it difficult to use. We are
    porting the Web UI as part of CDH4.2.

    If you can do without Web UI for now, you can give it a shot in CDH4.1

    Thanks,
    Santhosh
    On Wed, Oct 24, 2012 at 11:54 AM, Willy Chang wrote:

    Santhosh, Can you please send any artifacts for Fair Scheduler in MR2 at
    willy.chang.ic@gmail.com? We need this urgently. Thanks!

    On Tuesday, October 23, 2012 1:36:54 PM UTC-7, Daniel Oxenhandler wrote:

    Santhosh,

    Is there any update on this? we are running CDH 4.1 and would like to use
    fair scheduler. Is this available somewhere now? I checked the release
    notes for CDH4.1.1 and there is no mention of fair scheduler.

    Thanks!
    -Daniel

    On Wednesday, August 15, 2012 10:51:25 AM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    At this point in time, we have not published any artifacts for Fair
    Scheduler in MR2 for public consumption. Before releasing features,
    Cloudera gives a lot of importance to testing out the various components to
    safeguard customers from using untested code. Further, given that CDH
    4.1 will be released soon, it might be best to wait for CDH4.1.

    Thanks,
    Santhosh
    On Tue, Aug 14, 2012 at 9:54 AM, Willy Chang wrote:

    But can someone please post the link where to get the fair scheduler
    jar for MR2 CDH4? Thanks!

    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh Srinivasan wrote:

    Hi Willy,

    Fair scheduler was added to MR2 last month (see -
    https://issues.apache.org/**ji**ra/browse/MAPREDUCE-3451<https://issues.apache.org/jira/browse/MAPREDUCE-3451>).
    It will be available in CDH-4.1

    Thanks,
    Santhosh
    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang wrote:

    Also, where do i download the fair scheduler jar file for mr2 & CDH4.

    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang wrote:

    Thanks Harsh! We're using MR2. Can u send me the link for that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at
    http://archive.cloudera.com/**cd****h4/cdh/4/mr1/fair_scheduler.**
    ht****ml<http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html>should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    All - I really need help/pointers to setup Fair Scheduler in
    CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --

    --

    --
  • Willy Chang at Oct 25, 2012 at 5:49 pm
    Santhosh, Would be greatly appreciated if u could provide the instructions
    to setup the fair scheduler in CDH4.1. Thanks!
    On Thursday, October 25, 2012 10:34:45 AM UTC-7, Santhosh Srinivasan wrote:

    Hi Willy,

    The fair share scheduler functionality has been ported in part in CDH4.1.
    Its missing the Web UI (with metrics), making it difficult to use. We are
    porting the Web UI as part of CDH4.2.

    If you can do without Web UI for now, you can give it a shot in CDH4.1

    Thanks,
    Santhosh

    On Wed, Oct 24, 2012 at 11:54 AM, Willy Chang <willy.c...@gmail.com<javascript:>
    wrote:
    Santhosh, Can you please send any artifacts for Fair Scheduler in MR2 at
    willy.c...@gmail.com <javascript:>? We need this urgently. Thanks!

    On Tuesday, October 23, 2012 1:36:54 PM UTC-7, Daniel Oxenhandler wrote:

    Santhosh,

    Is there any update on this? we are running CDH 4.1 and would like to
    use fair scheduler. Is this available somewhere now? I checked the release
    notes for CDH4.1.1 and there is no mention of fair scheduler.

    Thanks!
    -Daniel

    On Wednesday, August 15, 2012 10:51:25 AM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    At this point in time, we have not published any artifacts for Fair
    Scheduler in MR2 for public consumption. Before releasing features,
    Cloudera gives a lot of importance to testing out the various components to
    safeguard customers from using untested code. Further, given that CDH
    4.1 will be released soon, it might be best to wait for CDH4.1.

    Thanks,
    Santhosh
    On Tue, Aug 14, 2012 at 9:54 AM, Willy Chang wrote:

    But can someone please post the link where to get the fair scheduler
    jar for MR2 CDH4? Thanks!

    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh Srinivasan wrote:

    Hi Willy,

    Fair scheduler was added to MR2 last month (see -
    https://issues.apache.org/**ji**ra/browse/MAPREDUCE-3451<https://issues.apache.org/jira/browse/MAPREDUCE-3451>).
    It will be available in CDH-4.1

    Thanks,
    Santhosh
    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang wrote:

    Also, where do i download the fair scheduler jar file for mr2 & CDH4.

    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang wrote:

    Thanks Harsh! We're using MR2. Can u send me the link for that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at
    http://archive.cloudera.com/**cd****h4/cdh/4/mr1/fair_scheduler.**
    ht****ml<http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html>should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    All - I really need help/pointers to setup Fair Scheduler in
    CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --

    --

    --
  • Sandy at Oct 26, 2012 at 12:47 am
    Hey Willy,

    Put this into your yarn-site.xml:
       <property>
         <name>yarn.resourcemanager.scheduler.class</name>

    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
       </property>

    Then, make sure that you have a fair-scheduler.xml in your YARN
    configuration directory.
    Its contents use the same format as they did in MR1, so you can look
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html for what to
    put in there. For a barebones one, you can try
    ---
    <?xml version="1.0"?>
    <allocations>
    </allocations>
    ---

    When you start the resourcemanager, the fair scheduler should be running.

    Hope that helps!
    -Sandy
    On Thursday, October 25, 2012 10:49:49 AM UTC-7, Willy Chang wrote:

    Santhosh, Would be greatly appreciated if u could provide the instructions
    to setup the fair scheduler in CDH4.1. Thanks!
    On Thursday, October 25, 2012 10:34:45 AM UTC-7, Santhosh Srinivasan wrote:

    Hi Willy,

    The fair share scheduler functionality has been ported in part in CDH4.1.
    Its missing the Web UI (with metrics), making it difficult to use. We are
    porting the Web UI as part of CDH4.2.

    If you can do without Web UI for now, you can give it a shot in CDH4.1

    Thanks,
    Santhosh
    On Wed, Oct 24, 2012 at 11:54 AM, Willy Chang wrote:

    Santhosh, Can you please send any artifacts for Fair Scheduler in MR2
    at willy.c...@gmail.com? We need this urgently. Thanks!

    On Tuesday, October 23, 2012 1:36:54 PM UTC-7, Daniel Oxenhandler wrote:

    Santhosh,

    Is there any update on this? we are running CDH 4.1 and would like to
    use fair scheduler. Is this available somewhere now? I checked the release
    notes for CDH4.1.1 and there is no mention of fair scheduler.

    Thanks!
    -Daniel

    On Wednesday, August 15, 2012 10:51:25 AM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    At this point in time, we have not published any artifacts for Fair
    Scheduler in MR2 for public consumption. Before releasing features,
    Cloudera gives a lot of importance to testing out the various components to
    safeguard customers from using untested code. Further, given that CDH
    4.1 will be released soon, it might be best to wait for CDH4.1.

    Thanks,
    Santhosh
    On Tue, Aug 14, 2012 at 9:54 AM, Willy Chang wrote:

    But can someone please post the link where to get the fair scheduler
    jar for MR2 CDH4? Thanks!


    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    Fair scheduler was added to MR2 last month (see -
    https://issues.apache.org/**ji**ra/browse/MAPREDUCE-3451<https://issues.apache.org/jira/browse/MAPREDUCE-3451>).
    It will be available in CDH-4.1

    Thanks,
    Santhosh
    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang wrote:

    Also, where do i download the fair scheduler jar file for mr2 &
    CDH4.

    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang wrote:

    Thanks Harsh! We're using MR2. Can u send me the link for that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at
    http://archive.cloudera.com/**cd****h4/cdh/4/mr1/fair_scheduler.*
    *ht****ml<http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html>should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang <
    willy.c...@gmail.com> wrote:
    All - I really need help/pointers to setup Fair Scheduler in
    CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --

    --

    --
  • Willy Chang at Oct 26, 2012 at 1:09 am
    Thanks Sandra. Can you also provide a command to confirm it is
    working? As expected, the {hostname}:8088/cluster/scheduler no longer
    works after adding the configs below.
    On Thu, Oct 25, 2012 at 5:47 PM, sandy wrote:
    Hey Willy,

    Put this into your yarn-site.xml:
    <property>
    <name>yarn.resourcemanager.scheduler.class</name>

    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    </property>

    Then, make sure that you have a fair-scheduler.xml in your YARN
    configuration directory.
    Its contents use the same format as they did in MR1, so you can look
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html for what to
    put in there. For a barebones one, you can try
    ---
    <?xml version="1.0"?>
    <allocations>
    </allocations>
    ---

    When you start the resourcemanager, the fair scheduler should be running.

    Hope that helps!
    -Sandy
    On Thursday, October 25, 2012 10:49:49 AM UTC-7, Willy Chang wrote:

    Santhosh, Would be greatly appreciated if u could provide the instructions
    to setup the fair scheduler in CDH4.1. Thanks!

    On Thursday, October 25, 2012 10:34:45 AM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    The fair share scheduler functionality has been ported in part in CDH4.1.
    Its missing the Web UI (with metrics), making it difficult to use. We are
    porting the Web UI as part of CDH4.2.

    If you can do without Web UI for now, you can give it a shot in CDH4.1

    Thanks,
    Santhosh

    On Wed, Oct 24, 2012 at 11:54 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    Santhosh, Can you please send any artifacts for Fair Scheduler in MR2 at
    willy.c...@gmail.com? We need this urgently. Thanks!

    On Tuesday, October 23, 2012 1:36:54 PM UTC-7, Daniel Oxenhandler wrote:

    Santhosh,

    Is there any update on this? we are running CDH 4.1 and would like to
    use fair scheduler. Is this available somewhere now? I checked the release
    notes for CDH4.1.1 and there is no mention of fair scheduler.

    Thanks!
    -Daniel

    On Wednesday, August 15, 2012 10:51:25 AM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    At this point in time, we have not published any artifacts for Fair
    Scheduler in MR2 for public consumption. Before releasing features, Cloudera
    gives a lot of importance to testing out the various components to safeguard
    customers from using untested code. Further, given that CDH 4.1 will be
    released soon, it might be best to wait for CDH4.1.

    Thanks,
    Santhosh

    On Tue, Aug 14, 2012 at 9:54 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    But can someone please post the link where to get the fair scheduler
    jar for MR2 CDH4? Thanks!


    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    Fair scheduler was added to MR2 last month (see -
    https://issues.apache.org/jira/browse/MAPREDUCE-3451). It will be available
    in CDH-4.1

    Thanks,
    Santhosh

    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang <willy.c...@gmail.com>
    wrote:
    Also, where do i download the fair scheduler jar file for mr2 &
    CDH4.

    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang wrote:

    Thanks Harsh! We're using MR2. Can u send me the link for that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html
    should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang
    wrote:
    All - I really need help/pointers to setup Fair Scheduler in
    CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --

    --

    --

    --
  • Sandy Ryza at Oct 26, 2012 at 1:54 am
    If that page no longer shows up and you're able to successfully run jobs on
    the cluster, then it should be working.

    -Sandy
    On Thu, Oct 25, 2012 at 6:09 PM, Willy Chang wrote:

    Thanks Sandra. Can you also provide a command to confirm it is
    working? As expected, the {hostname}:8088/cluster/scheduler no longer
    works after adding the configs below.
    On Thu, Oct 25, 2012 at 5:47 PM, sandy wrote:
    Hey Willy,

    Put this into your yarn-site.xml:
    <property>
    <name>yarn.resourcemanager.scheduler.class</name>

    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    </property>

    Then, make sure that you have a fair-scheduler.xml in your YARN
    configuration directory.
    Its contents use the same format as they did in MR1, so you can look
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html for what to
    put in there. For a barebones one, you can try
    ---
    <?xml version="1.0"?>
    <allocations>
    </allocations>
    ---

    When you start the resourcemanager, the fair scheduler should be running.

    Hope that helps!
    -Sandy
    On Thursday, October 25, 2012 10:49:49 AM UTC-7, Willy Chang wrote:

    Santhosh, Would be greatly appreciated if u could provide the
    instructions
    to setup the fair scheduler in CDH4.1. Thanks!

    On Thursday, October 25, 2012 10:34:45 AM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    The fair share scheduler functionality has been ported in part in
    CDH4.1.
    Its missing the Web UI (with metrics), making it difficult to use. We
    are
    porting the Web UI as part of CDH4.2.

    If you can do without Web UI for now, you can give it a shot in CDH4.1

    Thanks,
    Santhosh

    On Wed, Oct 24, 2012 at 11:54 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    Santhosh, Can you please send any artifacts for Fair Scheduler in MR2
    at
    willy.c...@gmail.com? We need this urgently. Thanks!


    On Tuesday, October 23, 2012 1:36:54 PM UTC-7, Daniel Oxenhandler
    wrote:
    Santhosh,

    Is there any update on this? we are running CDH 4.1 and would like to
    use fair scheduler. Is this available somewhere now? I checked the
    release
    notes for CDH4.1.1 and there is no mention of fair scheduler.

    Thanks!
    -Daniel

    On Wednesday, August 15, 2012 10:51:25 AM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    At this point in time, we have not published any artifacts for Fair
    Scheduler in MR2 for public consumption. Before releasing features,
    Cloudera
    gives a lot of importance to testing out the various components to
    safeguard
    customers from using untested code. Further, given that CDH 4.1
    will be
    released soon, it might be best to wait for CDH4.1.

    Thanks,
    Santhosh

    On Tue, Aug 14, 2012 at 9:54 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    But can someone please post the link where to get the fair
    scheduler
    jar for MR2 CDH4? Thanks!


    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    Fair scheduler was added to MR2 last month (see -
    https://issues.apache.org/jira/browse/MAPREDUCE-3451). It will
    be available
    in CDH-4.1

    Thanks,
    Santhosh

    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang <
    willy.c...@gmail.com>
    wrote:
    Also, where do i download the fair scheduler jar file for mr2 &
    CDH4.

    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang wrote:

    Thanks Harsh! We're using MR2. Can u send me the link for that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html
    should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang
    wrote:
    All - I really need help/pointers to setup Fair Scheduler in
    CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --

    --

    --

    --


    --
  • Willy Chang at Oct 26, 2012 at 7:39 pm
    Also, -jobConfOptions mapred.fairscheduler.pool="reserved-medium" doesn't
    work.
    On Thursday, October 25, 2012 6:54:15 PM UTC-7, sandy wrote:

    If that page no longer shows up and you're able to successfully run jobs
    on the cluster, then it should be working.

    -Sandy

    On Thu, Oct 25, 2012 at 6:09 PM, Willy Chang <willy.c...@gmail.com<javascript:>
    wrote:
    Thanks Sandra. Can you also provide a command to confirm it is
    working? As expected, the {hostname}:8088/cluster/scheduler no longer
    works after adding the configs below.

    On Thu, Oct 25, 2012 at 5:47 PM, sandy <sandy...@cloudera.com<javascript:>>
    wrote:
    Hey Willy,

    Put this into your yarn-site.xml:
    <property>
    <name>yarn.resourcemanager.scheduler.class</name>

    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    </property>

    Then, make sure that you have a fair-scheduler.xml in your YARN
    configuration directory.
    Its contents use the same format as they did in MR1, so you can look
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html for what to
    put in there. For a barebones one, you can try
    ---
    <?xml version="1.0"?>
    <allocations>
    </allocations>
    ---

    When you start the resourcemanager, the fair scheduler should be running.
    Hope that helps!
    -Sandy
    On Thursday, October 25, 2012 10:49:49 AM UTC-7, Willy Chang wrote:

    Santhosh, Would be greatly appreciated if u could provide the
    instructions
    to setup the fair scheduler in CDH4.1. Thanks!

    On Thursday, October 25, 2012 10:34:45 AM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    The fair share scheduler functionality has been ported in part in
    CDH4.1.
    Its missing the Web UI (with metrics), making it difficult to use. We
    are
    porting the Web UI as part of CDH4.2.

    If you can do without Web UI for now, you can give it a shot in CDH4.1

    Thanks,
    Santhosh

    On Wed, Oct 24, 2012 at 11:54 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    Santhosh, Can you please send any artifacts for Fair Scheduler in
    MR2 at
    willy.c...@gmail.com? We need this urgently. Thanks!


    On Tuesday, October 23, 2012 1:36:54 PM UTC-7, Daniel Oxenhandler
    wrote:
    Santhosh,

    Is there any update on this? we are running CDH 4.1 and would like
    to
    use fair scheduler. Is this available somewhere now? I checked the
    release
    notes for CDH4.1.1 and there is no mention of fair scheduler.

    Thanks!
    -Daniel

    On Wednesday, August 15, 2012 10:51:25 AM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    At this point in time, we have not published any artifacts for Fair
    Scheduler in MR2 for public consumption. Before releasing
    features, Cloudera
    gives a lot of importance to testing out the various components to
    safeguard
    customers from using untested code. Further, given that CDH 4.1
    will be
    released soon, it might be best to wait for CDH4.1.

    Thanks,
    Santhosh

    On Tue, Aug 14, 2012 at 9:54 AM, Willy Chang <willy.c...@gmail.com
    wrote:
    But can someone please post the link where to get the fair
    scheduler
    jar for MR2 CDH4? Thanks!


    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    Fair scheduler was added to MR2 last month (see -
    https://issues.apache.org/jira/browse/MAPREDUCE-3451). It will
    be available
    in CDH-4.1

    Thanks,
    Santhosh

    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang <
    willy.c...@gmail.com>
    wrote:
    Also, where do i download the fair scheduler jar file for mr2 &
    CDH4.

    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang wrote:

    Thanks Harsh! We're using MR2. Can u send me the link for that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html
    should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang
    wrote:
    All - I really need help/pointers to setup Fair Scheduler in
    CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --

    --

    --

    --


    --
  • Sandy Ryza at Oct 26, 2012 at 10:28 pm
    Application-level priority is not currently supported in YARN. You can
    track its progress at https://issues.apache.org/jira/browse/YARN-111.

    I believe mapred.fairscheduler.pool is no longer used. Use
    mapreduce.job.queuename instead.

    -Sandy
    On Fri, Oct 26, 2012 at 12:33 PM, Willy Chang wrote:

    Also, -jobConfOptions mapred.fairscheduler.pool="reserved-medium" doesn't
    work.

    On Thursday, October 25, 2012 6:54:15 PM UTC-7, sandy wrote:

    If that page no longer shows up and you're able to successfully run jobs
    on the cluster, then it should be working.

    -Sandy
    On Thu, Oct 25, 2012 at 6:09 PM, Willy Chang wrote:

    Thanks Sandra. Can you also provide a command to confirm it is
    working? As expected, the {hostname}:8088/cluster/**scheduler no longer
    works after adding the configs below.
    On Thu, Oct 25, 2012 at 5:47 PM, sandy wrote:
    Hey Willy,

    Put this into your yarn-site.xml:
    <property>
    <name>yarn.resourcemanager.**scheduler.class</name>

    <value>org.apache.hadoop.yarn.**server.resourcemanager.**
    scheduler.fair.FairScheduler</**value>
    </property>

    Then, make sure that you have a fair-scheduler.xml in your YARN
    configuration directory.
    Its contents use the same format as they did in MR1, so you can look
    http://archive.cloudera.com/**cdh4/cdh/4/mr1/fair_scheduler.**html<http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html>for what to
    put in there. For a barebones one, you can try
    ---
    <?xml version="1.0"?>
    <allocations>
    </allocations>
    ---

    When you start the resourcemanager, the fair scheduler should be running.
    Hope that helps!
    -Sandy
    On Thursday, October 25, 2012 10:49:49 AM UTC-7, Willy Chang wrote:

    Santhosh, Would be greatly appreciated if u could provide the
    instructions
    to setup the fair scheduler in CDH4.1. Thanks!

    On Thursday, October 25, 2012 10:34:45 AM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    The fair share scheduler functionality has been ported in part in
    CDH4.1.
    Its missing the Web UI (with metrics), making it difficult to use.
    We are
    porting the Web UI as part of CDH4.2.

    If you can do without Web UI for now, you can give it a shot in
    CDH4.1
    Thanks,
    Santhosh

    On Wed, Oct 24, 2012 at 11:54 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    Santhosh, Can you please send any artifacts for Fair Scheduler in
    MR2 at
    willy.c...@gmail.com? We need this urgently. Thanks!


    On Tuesday, October 23, 2012 1:36:54 PM UTC-7, Daniel Oxenhandler
    wrote:
    Santhosh,

    Is there any update on this? we are running CDH 4.1 and would like
    to
    use fair scheduler. Is this available somewhere now? I checked the
    release
    notes for CDH4.1.1 and there is no mention of fair scheduler.

    Thanks!
    -Daniel

    On Wednesday, August 15, 2012 10:51:25 AM UTC-7, Santhosh
    Srinivasan
    wrote:
    Hi Willy,

    At this point in time, we have not published any artifacts for
    Fair
    Scheduler in MR2 for public consumption. Before releasing
    features, Cloudera
    gives a lot of importance to testing out the various components
    to safeguard
    customers from using untested code. Further, given that CDH 4.1
    will be
    released soon, it might be best to wait for CDH4.1.

    Thanks,
    Santhosh

    On Tue, Aug 14, 2012 at 9:54 AM, Willy Chang <
    willy.c...@gmail.com>
    wrote:
    But can someone please post the link where to get the fair
    scheduler
    jar for MR2 CDH4? Thanks!


    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh Srinivasan
    wrote:
    It will be available
    in CDH-4.1

    Thanks,
    Santhosh

    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang <
    willy.c...@gmail.com>
    wrote:
    Also, where do i download the fair scheduler jar file for mr2 &
    CDH4.


    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang
    wrote:
    Thanks Harsh! We're using MR2. Can u send me the link for
    that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at
    http://archive.cloudera.com/**cdh4/cdh/4/mr1/fair_scheduler.
    **html <http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html>
    should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang
    wrote:
    All - I really need help/pointers to setup Fair Scheduler
    in
    CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --

    --

    --

    --


    --

    --
  • Willy Chang at Oct 26, 2012 at 10:01 pm
    Thank You Sandy! You seems to be good answering the questions in a
    breeze. On a different note, is there a way to gracefully shutdown
    yarn services?
    On Fri, Oct 26, 2012 at 2:24 PM, Sandy Ryza wrote:
    Application-level priority is not currently supported in YARN. You can
    track its progress at https://issues.apache.org/jira/browse/YARN-111.

    I believe mapred.fairscheduler.pool is no longer used. Use
    mapreduce.job.queuename instead.

    -Sandy

    On Fri, Oct 26, 2012 at 12:33 PM, Willy Chang wrote:

    Also, -jobConfOptions mapred.fairscheduler.pool="reserved-medium" doesn't
    work.

    On Thursday, October 25, 2012 6:54:15 PM UTC-7, sandy wrote:

    If that page no longer shows up and you're able to successfully run jobs
    on the cluster, then it should be working.

    -Sandy

    On Thu, Oct 25, 2012 at 6:09 PM, Willy Chang <willy.c...@gmail.com>
    wrote:
    Thanks Sandra. Can you also provide a command to confirm it is
    working? As expected, the {hostname}:8088/cluster/scheduler no longer
    works after adding the configs below.
    On Thu, Oct 25, 2012 at 5:47 PM, sandy wrote:
    Hey Willy,

    Put this into your yarn-site.xml:
    <property>
    <name>yarn.resourcemanager.scheduler.class</name>


    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    </property>

    Then, make sure that you have a fair-scheduler.xml in your YARN
    configuration directory.
    Its contents use the same format as they did in MR1, so you can look
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html for
    what to
    put in there. For a barebones one, you can try
    ---
    <?xml version="1.0"?>
    <allocations>
    </allocations>
    ---

    When you start the resourcemanager, the fair scheduler should be
    running.

    Hope that helps!
    -Sandy
    On Thursday, October 25, 2012 10:49:49 AM UTC-7, Willy Chang wrote:

    Santhosh, Would be greatly appreciated if u could provide the
    instructions
    to setup the fair scheduler in CDH4.1. Thanks!

    On Thursday, October 25, 2012 10:34:45 AM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    The fair share scheduler functionality has been ported in part in
    CDH4.1.
    Its missing the Web UI (with metrics), making it difficult to use.
    We are
    porting the Web UI as part of CDH4.2.

    If you can do without Web UI for now, you can give it a shot in
    CDH4.1

    Thanks,
    Santhosh

    On Wed, Oct 24, 2012 at 11:54 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    Santhosh, Can you please send any artifacts for Fair Scheduler in
    MR2 at
    willy.c...@gmail.com? We need this urgently. Thanks!


    On Tuesday, October 23, 2012 1:36:54 PM UTC-7, Daniel Oxenhandler
    wrote:
    Santhosh,

    Is there any update on this? we are running CDH 4.1 and would like
    to
    use fair scheduler. Is this available somewhere now? I checked the
    release
    notes for CDH4.1.1 and there is no mention of fair scheduler.

    Thanks!
    -Daniel

    On Wednesday, August 15, 2012 10:51:25 AM UTC-7, Santhosh
    Srinivasan
    wrote:
    Hi Willy,

    At this point in time, we have not published any artifacts for
    Fair
    Scheduler in MR2 for public consumption. Before releasing
    features, Cloudera
    gives a lot of importance to testing out the various components
    to safeguard
    customers from using untested code. Further, given that CDH 4.1
    will be
    released soon, it might be best to wait for CDH4.1.

    Thanks,
    Santhosh

    On Tue, Aug 14, 2012 at 9:54 AM, Willy Chang
    <willy.c...@gmail.com>
    wrote:
    But can someone please post the link where to get the fair
    scheduler
    jar for MR2 CDH4? Thanks!


    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    Fair scheduler was added to MR2 last month (see -
    https://issues.apache.org/jira/browse/MAPREDUCE-3451). It will
    be available
    in CDH-4.1

    Thanks,
    Santhosh

    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang
    <willy.c...@gmail.com>
    wrote:
    Also, where do i download the fair scheduler jar file for mr2
    &
    CDH4.


    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang
    wrote:
    Thanks Harsh! We're using MR2. Can u send me the link for
    that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at

    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html
    should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang
    wrote:
    All - I really need help/pointers to setup Fair Scheduler
    in
    CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --

    --

    --

    --

    --


    --

    --
  • Willy Chang at Oct 26, 2012 at 7:39 pm
    Sandy,

    Tried changing the job priority from NORMAL to VERY_LOW for testing
    purpose. But it is pegged at NORMAL though it says the priority has been
    changed.

      hadoop job -set-priority job_1351271289935_0008 VERY_LOW
    DEPRECATED: Use of this script to execute mapred command is deprecated.
    Instead use the mapred command for it.

    12/10/26 19:02:48 INFO mapred.ClientServiceDelegate: Application state is
    completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
    server
    Changed job priority.

    hadoop job -list
    Total jobs:1
                       JobId State StartTime UserName Queue
      Priority UsedContainers RsvdContainers UsedMem RsvdMem NeededMem AM info
      job_1351271289935_0012 RUNNING 1351278054002 hadoop default
      NORMAL

    On Thursday, October 25, 2012 6:54:15 PM UTC-7, sandy wrote:

    If that page no longer shows up and you're able to successfully run jobs
    on the cluster, then it should be working.

    -Sandy

    On Thu, Oct 25, 2012 at 6:09 PM, Willy Chang <willy.c...@gmail.com<javascript:>
    wrote:
    Thanks Sandra. Can you also provide a command to confirm it is
    working? As expected, the {hostname}:8088/cluster/scheduler no longer
    works after adding the configs below.

    On Thu, Oct 25, 2012 at 5:47 PM, sandy <sandy...@cloudera.com<javascript:>>
    wrote:
    Hey Willy,

    Put this into your yarn-site.xml:
    <property>
    <name>yarn.resourcemanager.scheduler.class</name>

    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    </property>

    Then, make sure that you have a fair-scheduler.xml in your YARN
    configuration directory.
    Its contents use the same format as they did in MR1, so you can look
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html for what to
    put in there. For a barebones one, you can try
    ---
    <?xml version="1.0"?>
    <allocations>
    </allocations>
    ---

    When you start the resourcemanager, the fair scheduler should be running.
    Hope that helps!
    -Sandy
    On Thursday, October 25, 2012 10:49:49 AM UTC-7, Willy Chang wrote:

    Santhosh, Would be greatly appreciated if u could provide the
    instructions
    to setup the fair scheduler in CDH4.1. Thanks!

    On Thursday, October 25, 2012 10:34:45 AM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    The fair share scheduler functionality has been ported in part in
    CDH4.1.
    Its missing the Web UI (with metrics), making it difficult to use. We
    are
    porting the Web UI as part of CDH4.2.

    If you can do without Web UI for now, you can give it a shot in CDH4.1

    Thanks,
    Santhosh

    On Wed, Oct 24, 2012 at 11:54 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    Santhosh, Can you please send any artifacts for Fair Scheduler in
    MR2 at
    willy.c...@gmail.com? We need this urgently. Thanks!


    On Tuesday, October 23, 2012 1:36:54 PM UTC-7, Daniel Oxenhandler
    wrote:
    Santhosh,

    Is there any update on this? we are running CDH 4.1 and would like
    to
    use fair scheduler. Is this available somewhere now? I checked the
    release
    notes for CDH4.1.1 and there is no mention of fair scheduler.

    Thanks!
    -Daniel

    On Wednesday, August 15, 2012 10:51:25 AM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    At this point in time, we have not published any artifacts for Fair
    Scheduler in MR2 for public consumption. Before releasing
    features, Cloudera
    gives a lot of importance to testing out the various components to
    safeguard
    customers from using untested code. Further, given that CDH 4.1
    will be
    released soon, it might be best to wait for CDH4.1.

    Thanks,
    Santhosh

    On Tue, Aug 14, 2012 at 9:54 AM, Willy Chang <willy.c...@gmail.com
    wrote:
    But can someone please post the link where to get the fair
    scheduler
    jar for MR2 CDH4? Thanks!


    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    Fair scheduler was added to MR2 last month (see -
    https://issues.apache.org/jira/browse/MAPREDUCE-3451). It will
    be available
    in CDH-4.1

    Thanks,
    Santhosh

    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang <
    willy.c...@gmail.com>
    wrote:
    Also, where do i download the fair scheduler jar file for mr2 &
    CDH4.

    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang wrote:

    Thanks Harsh! We're using MR2. Can u send me the link for that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html
    should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang
    wrote:
    All - I really need help/pointers to setup Fair Scheduler in
    CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --

    --

    --

    --


    --
  • Willy Chang at Oct 29, 2012 at 6:54 pm
    Sandy, Though it is working, and your recommendation helps but it stills
    doesn't allow us to dynamically change to a different pool while the job is
    running. Or there is a way to do this?
    On Thursday, October 25, 2012 6:54:15 PM UTC-7, sandy wrote:

    If that page no longer shows up and you're able to successfully run jobs
    on the cluster, then it should be working.

    -Sandy

    On Thu, Oct 25, 2012 at 6:09 PM, Willy Chang <willy.c...@gmail.com<javascript:>
    wrote:
    Thanks Sandra. Can you also provide a command to confirm it is
    working? As expected, the {hostname}:8088/cluster/scheduler no longer
    works after adding the configs below.

    On Thu, Oct 25, 2012 at 5:47 PM, sandy <sandy...@cloudera.com<javascript:>>
    wrote:
    Hey Willy,

    Put this into your yarn-site.xml:
    <property>
    <name>yarn.resourcemanager.scheduler.class</name>

    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    </property>

    Then, make sure that you have a fair-scheduler.xml in your YARN
    configuration directory.
    Its contents use the same format as they did in MR1, so you can look
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html for what to
    put in there. For a barebones one, you can try
    ---
    <?xml version="1.0"?>
    <allocations>
    </allocations>
    ---

    When you start the resourcemanager, the fair scheduler should be running.
    Hope that helps!
    -Sandy
    On Thursday, October 25, 2012 10:49:49 AM UTC-7, Willy Chang wrote:

    Santhosh, Would be greatly appreciated if u could provide the
    instructions
    to setup the fair scheduler in CDH4.1. Thanks!

    On Thursday, October 25, 2012 10:34:45 AM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    The fair share scheduler functionality has been ported in part in
    CDH4.1.
    Its missing the Web UI (with metrics), making it difficult to use. We
    are
    porting the Web UI as part of CDH4.2.

    If you can do without Web UI for now, you can give it a shot in CDH4.1

    Thanks,
    Santhosh

    On Wed, Oct 24, 2012 at 11:54 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    Santhosh, Can you please send any artifacts for Fair Scheduler in
    MR2 at
    willy.c...@gmail.com? We need this urgently. Thanks!


    On Tuesday, October 23, 2012 1:36:54 PM UTC-7, Daniel Oxenhandler
    wrote:
    Santhosh,

    Is there any update on this? we are running CDH 4.1 and would like
    to
    use fair scheduler. Is this available somewhere now? I checked the
    release
    notes for CDH4.1.1 and there is no mention of fair scheduler.

    Thanks!
    -Daniel

    On Wednesday, August 15, 2012 10:51:25 AM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    At this point in time, we have not published any artifacts for Fair
    Scheduler in MR2 for public consumption. Before releasing
    features, Cloudera
    gives a lot of importance to testing out the various components to
    safeguard
    customers from using untested code. Further, given that CDH 4.1
    will be
    released soon, it might be best to wait for CDH4.1.

    Thanks,
    Santhosh

    On Tue, Aug 14, 2012 at 9:54 AM, Willy Chang <willy.c...@gmail.com
    wrote:
    But can someone please post the link where to get the fair
    scheduler
    jar for MR2 CDH4? Thanks!


    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    Fair scheduler was added to MR2 last month (see -
    https://issues.apache.org/jira/browse/MAPREDUCE-3451). It will
    be available
    in CDH-4.1

    Thanks,
    Santhosh

    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang <
    willy.c...@gmail.com>
    wrote:
    Also, where do i download the fair scheduler jar file for mr2 &
    CDH4.

    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang wrote:

    Thanks Harsh! We're using MR2. Can u send me the link for that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html
    should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang
    wrote:
    All - I really need help/pointers to setup Fair Scheduler in
    CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --

    --

    --

    --


    --
  • Sandy Ryza at Oct 29, 2012 at 6:59 pm
    How were you able to do this in the past? Through the web UI?

    -Sandy
    On Mon, Oct 29, 2012 at 10:51 AM, Willy Chang wrote:

    Sandy, Though it is working, and your recommendation helps but it stills
    doesn't allow us to dynamically change to a different pool while the job is
    running. Or there is a way to do this?

    On Thursday, October 25, 2012 6:54:15 PM UTC-7, sandy wrote:

    If that page no longer shows up and you're able to successfully run jobs
    on the cluster, then it should be working.

    -Sandy
    On Thu, Oct 25, 2012 at 6:09 PM, Willy Chang wrote:

    Thanks Sandra. Can you also provide a command to confirm it is
    working? As expected, the {hostname}:8088/cluster/**scheduler no longer
    works after adding the configs below.
    On Thu, Oct 25, 2012 at 5:47 PM, sandy wrote:
    Hey Willy,

    Put this into your yarn-site.xml:
    <property>
    <name>yarn.resourcemanager.**scheduler.class</name>

    <value>org.apache.hadoop.yarn.**server.resourcemanager.**
    scheduler.fair.FairScheduler</**value>
    </property>

    Then, make sure that you have a fair-scheduler.xml in your YARN
    configuration directory.
    Its contents use the same format as they did in MR1, so you can look
    http://archive.cloudera.com/**cdh4/cdh/4/mr1/fair_scheduler.**html<http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html>for what to
    put in there. For a barebones one, you can try
    ---
    <?xml version="1.0"?>
    <allocations>
    </allocations>
    ---

    When you start the resourcemanager, the fair scheduler should be running.
    Hope that helps!
    -Sandy
    On Thursday, October 25, 2012 10:49:49 AM UTC-7, Willy Chang wrote:

    Santhosh, Would be greatly appreciated if u could provide the
    instructions
    to setup the fair scheduler in CDH4.1. Thanks!

    On Thursday, October 25, 2012 10:34:45 AM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    The fair share scheduler functionality has been ported in part in
    CDH4.1.
    Its missing the Web UI (with metrics), making it difficult to use.
    We are
    porting the Web UI as part of CDH4.2.

    If you can do without Web UI for now, you can give it a shot in
    CDH4.1
    Thanks,
    Santhosh

    On Wed, Oct 24, 2012 at 11:54 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    Santhosh, Can you please send any artifacts for Fair Scheduler in
    MR2 at
    willy.c...@gmail.com? We need this urgently. Thanks!


    On Tuesday, October 23, 2012 1:36:54 PM UTC-7, Daniel Oxenhandler
    wrote:
    Santhosh,

    Is there any update on this? we are running CDH 4.1 and would like
    to
    use fair scheduler. Is this available somewhere now? I checked the
    release
    notes for CDH4.1.1 and there is no mention of fair scheduler.

    Thanks!
    -Daniel

    On Wednesday, August 15, 2012 10:51:25 AM UTC-7, Santhosh
    Srinivasan
    wrote:
    Hi Willy,

    At this point in time, we have not published any artifacts for
    Fair
    Scheduler in MR2 for public consumption. Before releasing
    features, Cloudera
    gives a lot of importance to testing out the various components
    to safeguard
    customers from using untested code. Further, given that CDH 4.1
    will be
    released soon, it might be best to wait for CDH4.1.

    Thanks,
    Santhosh

    On Tue, Aug 14, 2012 at 9:54 AM, Willy Chang <
    willy.c...@gmail.com>
    wrote:
    But can someone please post the link where to get the fair
    scheduler
    jar for MR2 CDH4? Thanks!


    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh Srinivasan
    wrote:
    It will be available
    in CDH-4.1

    Thanks,
    Santhosh

    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang <
    willy.c...@gmail.com>
    wrote:
    Also, where do i download the fair scheduler jar file for mr2 &
    CDH4.


    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang
    wrote:
    Thanks Harsh! We're using MR2. Can u send me the link for
    that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at
    http://archive.cloudera.com/**cdh4/cdh/4/mr1/fair_scheduler.
    **html <http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html>
    should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang
    wrote:
    All - I really need help/pointers to setup Fair Scheduler
    in
    CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --

    --

    --

    --


    --

    --
  • Willy Chang at Oct 29, 2012 at 6:04 pm
    Yes Sandy.
    On Mon, Oct 29, 2012 at 10:55 AM, Sandy Ryza wrote:
    How were you able to do this in the past? Through the web UI?

    -Sandy

    On Mon, Oct 29, 2012 at 10:51 AM, Willy Chang wrote:

    Sandy, Though it is working, and your recommendation helps but it stills
    doesn't allow us to dynamically change to a different pool while the job is
    running. Or there is a way to do this?

    On Thursday, October 25, 2012 6:54:15 PM UTC-7, sandy wrote:

    If that page no longer shows up and you're able to successfully run jobs
    on the cluster, then it should be working.

    -Sandy

    On Thu, Oct 25, 2012 at 6:09 PM, Willy Chang <willy.c...@gmail.com>
    wrote:
    Thanks Sandra. Can you also provide a command to confirm it is
    working? As expected, the {hostname}:8088/cluster/scheduler no longer
    works after adding the configs below.
    On Thu, Oct 25, 2012 at 5:47 PM, sandy wrote:
    Hey Willy,

    Put this into your yarn-site.xml:
    <property>
    <name>yarn.resourcemanager.scheduler.class</name>


    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    </property>

    Then, make sure that you have a fair-scheduler.xml in your YARN
    configuration directory.
    Its contents use the same format as they did in MR1, so you can look
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html for
    what to
    put in there. For a barebones one, you can try
    ---
    <?xml version="1.0"?>
    <allocations>
    </allocations>
    ---

    When you start the resourcemanager, the fair scheduler should be
    running.

    Hope that helps!
    -Sandy
    On Thursday, October 25, 2012 10:49:49 AM UTC-7, Willy Chang wrote:

    Santhosh, Would be greatly appreciated if u could provide the
    instructions
    to setup the fair scheduler in CDH4.1. Thanks!

    On Thursday, October 25, 2012 10:34:45 AM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    The fair share scheduler functionality has been ported in part in
    CDH4.1.
    Its missing the Web UI (with metrics), making it difficult to use.
    We are
    porting the Web UI as part of CDH4.2.

    If you can do without Web UI for now, you can give it a shot in
    CDH4.1

    Thanks,
    Santhosh

    On Wed, Oct 24, 2012 at 11:54 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    Santhosh, Can you please send any artifacts for Fair Scheduler in
    MR2 at
    willy.c...@gmail.com? We need this urgently. Thanks!


    On Tuesday, October 23, 2012 1:36:54 PM UTC-7, Daniel Oxenhandler
    wrote:
    Santhosh,

    Is there any update on this? we are running CDH 4.1 and would like
    to
    use fair scheduler. Is this available somewhere now? I checked the
    release
    notes for CDH4.1.1 and there is no mention of fair scheduler.

    Thanks!
    -Daniel

    On Wednesday, August 15, 2012 10:51:25 AM UTC-7, Santhosh
    Srinivasan
    wrote:
    Hi Willy,

    At this point in time, we have not published any artifacts for
    Fair
    Scheduler in MR2 for public consumption. Before releasing
    features, Cloudera
    gives a lot of importance to testing out the various components
    to safeguard
    customers from using untested code. Further, given that CDH 4.1
    will be
    released soon, it might be best to wait for CDH4.1.

    Thanks,
    Santhosh

    On Tue, Aug 14, 2012 at 9:54 AM, Willy Chang
    <willy.c...@gmail.com>
    wrote:
    But can someone please post the link where to get the fair
    scheduler
    jar for MR2 CDH4? Thanks!


    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh Srinivasan
    wrote:
    Hi Willy,

    Fair scheduler was added to MR2 last month (see -
    https://issues.apache.org/jira/browse/MAPREDUCE-3451). It will
    be available
    in CDH-4.1

    Thanks,
    Santhosh

    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang
    <willy.c...@gmail.com>
    wrote:
    Also, where do i download the fair scheduler jar file for mr2
    &
    CDH4.


    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang
    wrote:
    Thanks Harsh! We're using MR2. Can u send me the link for
    that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at

    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html
    should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang
    wrote:
    All - I really need help/pointers to setup Fair Scheduler
    in
    CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --

    --

    --

    --

    --


    --

    --
  • Sandy Ryza at Oct 29, 2012 at 10:50 pm
    I don't believe it's currently supported. Do you mind me asking what you
    use it for?

    thanks,
    Sandy
    On Mon, Oct 29, 2012 at 10:57 AM, Willy Chang wrote:

    Yes Sandy.
    On Mon, Oct 29, 2012 at 10:55 AM, Sandy Ryza wrote:
    How were you able to do this in the past? Through the web UI?

    -Sandy


    On Mon, Oct 29, 2012 at 10:51 AM, Willy Chang <willy.chang.ic@gmail.com>
    wrote:
    Sandy, Though it is working, and your recommendation helps but it stills
    doesn't allow us to dynamically change to a different pool while the
    job is
    running. Or there is a way to do this?

    On Thursday, October 25, 2012 6:54:15 PM UTC-7, sandy wrote:

    If that page no longer shows up and you're able to successfully run
    jobs
    on the cluster, then it should be working.

    -Sandy

    On Thu, Oct 25, 2012 at 6:09 PM, Willy Chang <willy.c...@gmail.com>
    wrote:
    Thanks Sandra. Can you also provide a command to confirm it is
    working? As expected, the {hostname}:8088/cluster/scheduler no longer
    works after adding the configs below.
    On Thu, Oct 25, 2012 at 5:47 PM, sandy wrote:
    Hey Willy,

    Put this into your yarn-site.xml:
    <property>
    <name>yarn.resourcemanager.scheduler.class</name>

    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    </property>

    Then, make sure that you have a fair-scheduler.xml in your YARN
    configuration directory.
    Its contents use the same format as they did in MR1, so you can look
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html for
    what to
    put in there. For a barebones one, you can try
    ---
    <?xml version="1.0"?>
    <allocations>
    </allocations>
    ---

    When you start the resourcemanager, the fair scheduler should be
    running.

    Hope that helps!
    -Sandy
    On Thursday, October 25, 2012 10:49:49 AM UTC-7, Willy Chang wrote:

    Santhosh, Would be greatly appreciated if u could provide the
    instructions
    to setup the fair scheduler in CDH4.1. Thanks!

    On Thursday, October 25, 2012 10:34:45 AM UTC-7, Santhosh
    Srinivasan
    wrote:
    Hi Willy,

    The fair share scheduler functionality has been ported in part in
    CDH4.1.
    Its missing the Web UI (with metrics), making it difficult to use.
    We are
    porting the Web UI as part of CDH4.2.

    If you can do without Web UI for now, you can give it a shot in
    CDH4.1

    Thanks,
    Santhosh

    On Wed, Oct 24, 2012 at 11:54 AM, Willy Chang <
    willy.c...@gmail.com>
    wrote:
    Santhosh, Can you please send any artifacts for Fair Scheduler in
    MR2 at
    willy.c...@gmail.com? We need this urgently. Thanks!


    On Tuesday, October 23, 2012 1:36:54 PM UTC-7, Daniel Oxenhandler
    wrote:
    Santhosh,

    Is there any update on this? we are running CDH 4.1 and would
    like
    to
    use fair scheduler. Is this available somewhere now? I checked
    the
    release
    notes for CDH4.1.1 and there is no mention of fair scheduler.

    Thanks!
    -Daniel

    On Wednesday, August 15, 2012 10:51:25 AM UTC-7, Santhosh
    Srinivasan
    wrote:
    Hi Willy,

    At this point in time, we have not published any artifacts for
    Fair
    Scheduler in MR2 for public consumption. Before releasing
    features, Cloudera
    gives a lot of importance to testing out the various components
    to safeguard
    customers from using untested code. Further, given that CDH 4.1
    will be
    released soon, it might be best to wait for CDH4.1.

    Thanks,
    Santhosh

    On Tue, Aug 14, 2012 at 9:54 AM, Willy Chang
    <willy.c...@gmail.com>
    wrote:
    But can someone please post the link where to get the fair
    scheduler
    jar for MR2 CDH4? Thanks!


    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh
    Srinivasan
    wrote:
    Hi Willy,

    Fair scheduler was added to MR2 last month (see -
    https://issues.apache.org/jira/browse/MAPREDUCE-3451). It
    will
    be available
    in CDH-4.1

    Thanks,
    Santhosh

    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang
    <willy.c...@gmail.com>
    wrote:
    Also, where do i download the fair scheduler jar file for
    mr2
    &
    CDH4.


    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang
    wrote:
    Thanks Harsh! We're using MR2. Can u send me the link for
    that?
    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J wrote:

    Hi Willy,

    The instructions at
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html
    should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang
    wrote:
    All - I really need help/pointers to setup Fair
    Scheduler
    in
    CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --

    --

    --

    --

    --


    --

    --


    --
  • Willy Chang at Oct 30, 2012 at 9:15 pm
    It is useful to switch jobs to a different pool or queue, in case the
    primary pool is busy. We're in big data business.
    On Monday, October 29, 2012 3:50:20 PM UTC-7, sandy wrote:

    I don't believe it's currently supported. Do you mind me asking what you
    use it for?

    thanks,
    Sandy

    On Mon, Oct 29, 2012 at 10:57 AM, Willy Chang <willy.c...@gmail.com<javascript:>
    wrote:
    Yes Sandy.

    On Mon, Oct 29, 2012 at 10:55 AM, Sandy Ryza <sandy...@cloudera.com<javascript:>>
    wrote:
    How were you able to do this in the past? Through the web UI?

    -Sandy


    On Mon, Oct 29, 2012 at 10:51 AM, Willy Chang <willy.c...@gmail.com<javascript:>

    wrote:
    Sandy, Though it is working, and your recommendation helps but it
    stills
    doesn't allow us to dynamically change to a different pool while the
    job is
    running. Or there is a way to do this?

    On Thursday, October 25, 2012 6:54:15 PM UTC-7, sandy wrote:

    If that page no longer shows up and you're able to successfully run
    jobs
    on the cluster, then it should be working.

    -Sandy

    On Thu, Oct 25, 2012 at 6:09 PM, Willy Chang <willy.c...@gmail.com>
    wrote:
    Thanks Sandra. Can you also provide a command to confirm it is
    working? As expected, the {hostname}:8088/cluster/scheduler no longer
    works after adding the configs below.

    On Thu, Oct 25, 2012 at 5:47 PM, sandy <sandy...@cloudera.com>
    wrote:
    Hey Willy,

    Put this into your yarn-site.xml:
    <property>
    <name>yarn.resourcemanager.scheduler.class</name>

    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    </property>

    Then, make sure that you have a fair-scheduler.xml in your YARN
    configuration directory.
    Its contents use the same format as they did in MR1, so you can
    look
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html for
    what to
    put in there. For a barebones one, you can try
    ---
    <?xml version="1.0"?>
    <allocations>
    </allocations>
    ---

    When you start the resourcemanager, the fair scheduler should be
    running.

    Hope that helps!
    -Sandy
    On Thursday, October 25, 2012 10:49:49 AM UTC-7, Willy Chang wrote:

    Santhosh, Would be greatly appreciated if u could provide the
    instructions
    to setup the fair scheduler in CDH4.1. Thanks!

    On Thursday, October 25, 2012 10:34:45 AM UTC-7, Santhosh
    Srinivasan
    wrote:
    Hi Willy,

    The fair share scheduler functionality has been ported in part in
    CDH4.1.
    Its missing the Web UI (with metrics), making it difficult to
    use.
    We are
    porting the Web UI as part of CDH4.2.

    If you can do without Web UI for now, you can give it a shot in
    CDH4.1

    Thanks,
    Santhosh

    On Wed, Oct 24, 2012 at 11:54 AM, Willy Chang <
    willy.c...@gmail.com>
    wrote:
    Santhosh, Can you please send any artifacts for Fair Scheduler
    in
    MR2 at
    willy.c...@gmail.com? We need this urgently. Thanks!


    On Tuesday, October 23, 2012 1:36:54 PM UTC-7, Daniel
    Oxenhandler
    wrote:
    Santhosh,

    Is there any update on this? we are running CDH 4.1 and would
    like
    to
    use fair scheduler. Is this available somewhere now? I checked
    the
    release
    notes for CDH4.1.1 and there is no mention of fair scheduler.

    Thanks!
    -Daniel

    On Wednesday, August 15, 2012 10:51:25 AM UTC-7, Santhosh
    Srinivasan
    wrote:
    Hi Willy,

    At this point in time, we have not published any artifacts for
    Fair
    Scheduler in MR2 for public consumption. Before releasing
    features, Cloudera
    gives a lot of importance to testing out the various
    components
    to safeguard
    customers from using untested code. Further, given that CDH
    4.1
    will be
    released soon, it might be best to wait for CDH4.1.

    Thanks,
    Santhosh

    On Tue, Aug 14, 2012 at 9:54 AM, Willy Chang
    <willy.c...@gmail.com>
    wrote:
    But can someone please post the link where to get the fair
    scheduler
    jar for MR2 CDH4? Thanks!


    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh
    Srinivasan
    wrote:
    Hi Willy,

    Fair scheduler was added to MR2 last month (see -
    https://issues.apache.org/jira/browse/MAPREDUCE-3451). It
    will
    be available
    in CDH-4.1

    Thanks,
    Santhosh

    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang
    <willy.c...@gmail.com>
    wrote:
    Also, where do i download the fair scheduler jar file for
    mr2
    &
    CDH4.


    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang
    wrote:
    Thanks Harsh! We're using MR2. Can u send me the link for
    that?

    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J
    wrote:
    Hi Willy,

    The instructions at
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html
    should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang
    wrote:
    All - I really need help/pointers to setup Fair
    Scheduler
    in
    CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --

    --

    --

    --

    --


    --

    --


    --
  • Willy Chang at Oct 30, 2012 at 10:52 pm
    Sandy - With your recommendation, we've enabled the fair scheduler but
    after running the flows for sometime the RM manager stops with the
    following errors below.


    2012-10-29 22:30:52,901 FATAL
    org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
    handling event type NODE_UPDATE to the scheduler
    java.lang.NullPointerException
             at
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerApp.unreserve(FSSchedulerApp.java:356)
             at
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.unreserve(AppSchedulable.java:214)
             at
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.assignContainer(AppSchedulable.java:266)
             at
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.assignContainer(AppSchedulable.java:330)
             at
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueueSchedulable.assignContainer(FSQueueSchedulable.java:161)
             at
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:759)
             at
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:836)
             at
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1)
             at
    org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:329)
             at java.lang.Thread.run(Thread.java:662)
    2012-10-29 22:30:52,903 INFO
    org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting,
    bbye..
    2012-10-29 22:30:52,911 INFO org.mortbay.log: Stopped
    selectchannelconnector@0.0.0.0:8088
    2012-10-29 22:30:53,015 ERROR
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    InterruptedExcpetion recieved for ExpiredTokenRemover thread
    java.lang.InterruptedException: sleep interrupted
    2012-10-29 22:30:53,015 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping ResourceManager
    metrics system...
    2012-10-29 22:30:53,020 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics
    system stopped.
    2012-10-29 22:30:53,021 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics
    system shutdown complete.
    2012-10-29 22:30:53,021 WARN
    org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher:
    org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
    interrupted. Returning.
    2012-10-29 22:30:53,022 INFO
    org.apache.hadoop.yarn.service.AbstractService:
    Service:org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher
    is stopped.
    2012-10-29 22:30:53,022 INFO org.apache.hadoop.ipc.Server: Stopping server
    on 8033
    2012-10-29 22:30:53,023 INFO org.apache.hadoop.ipc.Server: Stopping IPC
    Server listener on 8033
    2012-10-29 22:30:53,023 INFO org.apache.hadoop.ipc.Server: Stopping IPC
    Server Responder
    2012-10-29 22:30:53,023 INFO
    org.apache.hadoop.yarn.service.AbstractService:
    Service:org.apache.hadoop.yarn.server.resourcemanager.AdminService is
    stopped.
    2012-10-29 22:30:53,023 INFO org.apache.hadoop.ipc.Server: Stopping server
    on 8040
    2012-10-29 22:30:53,027 INFO org.apache.hadoop.ipc.Server: Stopping IPC
    Server listener on 8040
    2012-10-29 22:30:53,028 INFO
    org.apache.hadoop.yarn.service.AbstractService:
    Service:org.apache.hadoop.yarn.server.resourcemanager.ClientRMService is
    stopped.
    2012-10-29 22:30:53,028 INFO org.apache.hadoop.ipc.Server: Stopping IPC
    Server Responder
    2012-10-29 22:30:53,029 INFO org.apache.hadoop.ipc.Server: Stopping server
    on 8030
    2012-10-29 22:30:53,031 INFO org.apache.hadoop.ipc.Server: Stopping IPC
    Server listener on 8030
    2012-10-29 22:30:53,032 INFO org.apache.hadoop.ipc.Server: Stopping IPC
    Server Responder
    On Monday, October 29, 2012 3:50:20 PM UTC-7, sandy wrote:

    I don't believe it's currently supported. Do you mind me asking what you
    use it for?

    thanks,
    Sandy

    On Mon, Oct 29, 2012 at 10:57 AM, Willy Chang <willy.c...@gmail.com<javascript:>
    wrote:
    Yes Sandy.

    On Mon, Oct 29, 2012 at 10:55 AM, Sandy Ryza <sandy...@cloudera.com<javascript:>>
    wrote:
    How were you able to do this in the past? Through the web UI?

    -Sandy


    On Mon, Oct 29, 2012 at 10:51 AM, Willy Chang <willy.c...@gmail.com<javascript:>

    wrote:
    Sandy, Though it is working, and your recommendation helps but it
    stills
    doesn't allow us to dynamically change to a different pool while the
    job is
    running. Or there is a way to do this?

    On Thursday, October 25, 2012 6:54:15 PM UTC-7, sandy wrote:

    If that page no longer shows up and you're able to successfully run
    jobs
    on the cluster, then it should be working.

    -Sandy

    On Thu, Oct 25, 2012 at 6:09 PM, Willy Chang <willy.c...@gmail.com>
    wrote:
    Thanks Sandra. Can you also provide a command to confirm it is
    working? As expected, the {hostname}:8088/cluster/scheduler no longer
    works after adding the configs below.

    On Thu, Oct 25, 2012 at 5:47 PM, sandy <sandy...@cloudera.com>
    wrote:
    Hey Willy,

    Put this into your yarn-site.xml:
    <property>
    <name>yarn.resourcemanager.scheduler.class</name>

    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    </property>

    Then, make sure that you have a fair-scheduler.xml in your YARN
    configuration directory.
    Its contents use the same format as they did in MR1, so you can
    look
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html for
    what to
    put in there. For a barebones one, you can try
    ---
    <?xml version="1.0"?>
    <allocations>
    </allocations>
    ---

    When you start the resourcemanager, the fair scheduler should be
    running.

    Hope that helps!
    -Sandy
    On Thursday, October 25, 2012 10:49:49 AM UTC-7, Willy Chang wrote:

    Santhosh, Would be greatly appreciated if u could provide the
    instructions
    to setup the fair scheduler in CDH4.1. Thanks!

    On Thursday, October 25, 2012 10:34:45 AM UTC-7, Santhosh
    Srinivasan
    wrote:
    Hi Willy,

    The fair share scheduler functionality has been ported in part in
    CDH4.1.
    Its missing the Web UI (with metrics), making it difficult to
    use.
    We are
    porting the Web UI as part of CDH4.2.

    If you can do without Web UI for now, you can give it a shot in
    CDH4.1

    Thanks,
    Santhosh

    On Wed, Oct 24, 2012 at 11:54 AM, Willy Chang <
    willy.c...@gmail.com>
    wrote:
    Santhosh, Can you please send any artifacts for Fair Scheduler
    in
    MR2 at
    willy.c...@gmail.com? We need this urgently. Thanks!


    On Tuesday, October 23, 2012 1:36:54 PM UTC-7, Daniel
    Oxenhandler
    wrote:
    Santhosh,

    Is there any update on this? we are running CDH 4.1 and would
    like
    to
    use fair scheduler. Is this available somewhere now? I checked
    the
    release
    notes for CDH4.1.1 and there is no mention of fair scheduler.

    Thanks!
    -Daniel

    On Wednesday, August 15, 2012 10:51:25 AM UTC-7, Santhosh
    Srinivasan
    wrote:
    Hi Willy,

    At this point in time, we have not published any artifacts for
    Fair
    Scheduler in MR2 for public consumption. Before releasing
    features, Cloudera
    gives a lot of importance to testing out the various
    components
    to safeguard
    customers from using untested code. Further, given that CDH
    4.1
    will be
    released soon, it might be best to wait for CDH4.1.

    Thanks,
    Santhosh

    On Tue, Aug 14, 2012 at 9:54 AM, Willy Chang
    <willy.c...@gmail.com>
    wrote:
    But can someone please post the link where to get the fair
    scheduler
    jar for MR2 CDH4? Thanks!


    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh
    Srinivasan
    wrote:
    Hi Willy,

    Fair scheduler was added to MR2 last month (see -
    https://issues.apache.org/jira/browse/MAPREDUCE-3451). It
    will
    be available
    in CDH-4.1

    Thanks,
    Santhosh

    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang
    <willy.c...@gmail.com>
    wrote:
    Also, where do i download the fair scheduler jar file for
    mr2
    &
    CDH4.


    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang
    wrote:
    Thanks Harsh! We're using MR2. Can u send me the link for
    that?

    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J
    wrote:
    Hi Willy,

    The instructions at
    http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html
    should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang
    wrote:
    All - I really need help/pointers to setup Fair
    Scheduler
    in
    CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --

    --

    --

    --

    --


    --

    --


    --
  • Sandy Ryza at Nov 1, 2012 at 5:02 am
    Hi Willy,

    We haven't come across that before. Thanks for letting us know! I filed a
    JIRA for it: https://issues.apache.org/jira/browse/YARN-192. Do you have
    any idea how we might reproduce it?

    thanks,
    Sandy
    On Tue, Oct 30, 2012 at 3:52 PM, Willy Chang wrote:

    Sandy - With your recommendation, we've enabled the fair scheduler but
    after running the flows for sometime the RM manager stops with the
    following errors below.


    2012-10-29 22:30:52,901 FATAL
    org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
    handling event type NODE_UPDATE to the scheduler
    java.lang.NullPointerException
    at
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerApp.unreserve(FSSchedulerApp.java:356)
    at
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.unreserve(AppSchedulable.java:214)
    at
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.assignContainer(AppSchedulable.java:266)
    at
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.assignContainer(AppSchedulable.java:330)
    at
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueueSchedulable.assignContainer(FSQueueSchedulable.java:161)
    at
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:759)
    at
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:836)
    at
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1)
    at
    org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:329)
    at java.lang.Thread.run(Thread.java:662)
    2012-10-29 22:30:52,903 INFO
    org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting,
    bbye..
    2012-10-29 22:30:52,911 INFO org.mortbay.log: Stopped
    selectchannelconnector@0.0.0.0:8088
    2012-10-29 22:30:53,015 ERROR
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    InterruptedExcpetion recieved for ExpiredTokenRemover thread
    java.lang.InterruptedException: sleep interrupted
    2012-10-29 22:30:53,015 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping ResourceManager
    metrics system...
    2012-10-29 22:30:53,020 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics
    system stopped.
    2012-10-29 22:30:53,021 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics
    system shutdown complete.
    2012-10-29 22:30:53,021 WARN
    org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher:
    org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
    interrupted. Returning.
    2012-10-29 22:30:53,022 INFO
    org.apache.hadoop.yarn.service.AbstractService:
    Service:org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher
    is stopped.
    2012-10-29 22:30:53,022 INFO org.apache.hadoop.ipc.Server: Stopping server
    on 8033
    2012-10-29 22:30:53,023 INFO org.apache.hadoop.ipc.Server: Stopping IPC
    Server listener on 8033
    2012-10-29 22:30:53,023 INFO org.apache.hadoop.ipc.Server: Stopping IPC
    Server Responder
    2012-10-29 22:30:53,023 INFO
    org.apache.hadoop.yarn.service.AbstractService:
    Service:org.apache.hadoop.yarn.server.resourcemanager.AdminService is
    stopped.
    2012-10-29 22:30:53,023 INFO org.apache.hadoop.ipc.Server: Stopping server
    on 8040
    2012-10-29 22:30:53,027 INFO org.apache.hadoop.ipc.Server: Stopping IPC
    Server listener on 8040
    2012-10-29 22:30:53,028 INFO
    org.apache.hadoop.yarn.service.AbstractService:
    Service:org.apache.hadoop.yarn.server.resourcemanager.ClientRMService is
    stopped.
    2012-10-29 22:30:53,028 INFO org.apache.hadoop.ipc.Server: Stopping IPC
    Server Responder
    2012-10-29 22:30:53,029 INFO org.apache.hadoop.ipc.Server: Stopping server
    on 8030
    2012-10-29 22:30:53,031 INFO org.apache.hadoop.ipc.Server: Stopping IPC
    Server listener on 8030
    2012-10-29 22:30:53,032 INFO org.apache.hadoop.ipc.Server: Stopping IPC
    Server Responder
    On Monday, October 29, 2012 3:50:20 PM UTC-7, sandy wrote:

    I don't believe it's currently supported. Do you mind me asking what you
    use it for?

    thanks,
    Sandy
    On Mon, Oct 29, 2012 at 10:57 AM, Willy Chang wrote:

    Yes Sandy.

    On Mon, Oct 29, 2012 at 10:55 AM, Sandy Ryza <sandy...@cloudera.com>
    wrote:
    How were you able to do this in the past? Through the web UI?

    -Sandy


    On Mon, Oct 29, 2012 at 10:51 AM, Willy Chang <willy.c...@gmail.com>
    wrote:
    Sandy, Though it is working, and your recommendation helps but it
    stills
    doesn't allow us to dynamically change to a different pool while the
    job is
    running. Or there is a way to do this?

    On Thursday, October 25, 2012 6:54:15 PM UTC-7, sandy wrote:

    If that page no longer shows up and you're able to successfully run
    jobs
    on the cluster, then it should be working.

    -Sandy

    On Thu, Oct 25, 2012 at 6:09 PM, Willy Chang <willy.c...@gmail.com>
    wrote:
    Thanks Sandra. Can you also provide a command to confirm it is
    working? As expected, the {hostname}:8088/cluster/**scheduler no
    longer
    works after adding the configs below.

    On Thu, Oct 25, 2012 at 5:47 PM, sandy <sandy...@cloudera.com>
    wrote:
    Hey Willy,

    Put this into your yarn-site.xml:
    <property>
    <name>yarn.resourcemanager.**scheduler.class</name>


    <value>org.apache.hadoop.yarn.**server.resourcemanager.**
    scheduler.fair.FairScheduler</**value>
    </property>

    Then, make sure that you have a fair-scheduler.xml in your YARN
    configuration directory.
    Its contents use the same format as they did in MR1, so you can
    look
    html <http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html>for
    what to
    put in there. For a barebones one, you can try
    ---
    <?xml version="1.0"?>
    <allocations>
    </allocations>
    ---

    When you start the resourcemanager, the fair scheduler should be
    running.

    Hope that helps!
    -Sandy

    On Thursday, October 25, 2012 10:49:49 AM UTC-7, Willy Chang
    wrote:
    Santhosh, Would be greatly appreciated if u could provide the
    instructions
    to setup the fair scheduler in CDH4.1. Thanks!

    On Thursday, October 25, 2012 10:34:45 AM UTC-7, Santhosh
    Srinivasan
    wrote:
    Hi Willy,

    The fair share scheduler functionality has been ported in part
    in
    CDH4.1.
    Its missing the Web UI (with metrics), making it difficult to
    use.
    We are
    porting the Web UI as part of CDH4.2.

    If you can do without Web UI for now, you can give it a shot in
    CDH4.1

    Thanks,
    Santhosh

    On Wed, Oct 24, 2012 at 11:54 AM, Willy Chang <
    willy.c...@gmail.com>
    wrote:
    Santhosh, Can you please send any artifacts for Fair Scheduler
    in
    MR2 at
    willy.c...@gmail.com? We need this urgently. Thanks!


    On Tuesday, October 23, 2012 1:36:54 PM UTC-7, Daniel
    Oxenhandler
    wrote:
    Santhosh,

    Is there any update on this? we are running CDH 4.1 and would
    like
    to
    use fair scheduler. Is this available somewhere now? I
    checked the
    release
    notes for CDH4.1.1 and there is no mention of fair scheduler.

    Thanks!
    -Daniel

    On Wednesday, August 15, 2012 10:51:25 AM UTC-7, Santhosh
    Srinivasan
    wrote:
    Hi Willy,

    At this point in time, we have not published any artifacts
    for
    Fair
    Scheduler in MR2 for public consumption. Before releasing
    features, Cloudera
    gives a lot of importance to testing out the various
    components
    to safeguard
    customers from using untested code. Further, given that CDH
    4.1
    will be
    released soon, it might be best to wait for CDH4.1.

    Thanks,
    Santhosh

    On Tue, Aug 14, 2012 at 9:54 AM, Willy Chang
    <willy.c...@gmail.com>
    wrote:
    But can someone please post the link where to get the fair
    scheduler
    jar for MR2 CDH4? Thanks!


    On Monday, August 13, 2012 6:04:57 PM UTC-7, Santhosh
    Srinivasan
    It will
    be available
    in CDH-4.1

    Thanks,
    Santhosh

    On Mon, Aug 13, 2012 at 3:12 PM, Willy Chang
    <willy.c...@gmail.com>
    wrote:
    Also, where do i download the fair scheduler jar file for
    mr2
    &
    CDH4.


    On Monday, August 13, 2012 12:39:13 PM UTC-7, Willy Chang
    wrote:
    Thanks Harsh! We're using MR2. Can u send me the link for
    that?

    On Friday, August 10, 2012 4:08:10 AM UTC-7, Harsh J
    wrote:
    Hi Willy,

    The instructions at

    http://archive.cloudera.com/**
    cdh4/cdh/4/mr1/fair_scheduler.**html<http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html>
    should
    still apply to MR1. Is this what you're looking for?

    On Fri, Aug 10, 2012 at 3:15 AM, Willy Chang
    wrote:
    All - I really need help/pointers to setup Fair
    Scheduler
    in
    CDH4. Any help
    would be greatly appreciated. Thanks Willy

    --



    --
    Harsh J
    --

    --

    --

    --

    --

    --


    --

    --


    --

    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcdh-user @
categorieshadoop
postedAug 9, '12 at 9:51p
activeNov 1, '12 at 5:02a
posts25
users5
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase