Hey,

We use capacity scheduler and divide our map slots among queues. For a
particular kind of job, we want to schedule at most one task per task
tracker. How does one do this? We are using Hadoop 0.20.205.0.

Thanks,

--
Nitin Khandelwal

Search Discussions

  • Harsh J at Dec 20, 2011 at 12:08 pm
    Nitin,

    AFAIK there's no way to do this directly. Why are you looking to do
    that though? What gains do you see you'll get that way against letting
    slots be used in the regular way, given multi-core CPUs and multi-disk
    DFS setups?

    On Tue, Dec 20, 2011 at 5:16 PM, Nitin Khandelwal
    wrote:
    Hey,

    We use capacity scheduler and divide our map slots among queues. For a
    particular kind of job, we want to schedule at most one task per task
    tracker. How does one do this?  We are using Hadoop 0.20.205.0.

    Thanks,

    --
    Nitin Khandelwal


    --
    Harsh J
  • Nitin Khandelwal at Dec 20, 2011 at 12:14 pm
    Hi Harsh,
    Thanks for your reply.

    The task makes web request which is IP limited. So , I need only one task
    per slave.

    Thanks,
    Nitin
    On 20 December 2011 17:37, Harsh J wrote:

    Nitin,

    AFAIK there's no way to do this directly. Why are you looking to do
    that though? What gains do you see you'll get that way against letting
    slots be used in the regular way, given multi-core CPUs and multi-disk
    DFS setups?

    On Tue, Dec 20, 2011 at 5:16 PM, Nitin Khandelwal
    wrote:
    Hey,

    We use capacity scheduler and divide our map slots among queues. For a
    particular kind of job, we want to schedule at most one task per task
    tracker. How does one do this? We are using Hadoop 0.20.205.0.

    Thanks,

    --
    Nitin Khandelwal


    --
    Harsh J


    --


    Nitin Khandelwal
  • Arun C Murthy at Dec 20, 2011 at 4:14 pm
    Just use multiple slots per each map.

    See: http://hadoop.apache.org/common/docs/stable/capacity_scheduler.html#Resource+based+scheduling

    Arun
    On Dec 20, 2011, at 3:46 AM, Nitin Khandelwal wrote:

    Hey,

    We use capacity scheduler and divide our map slots among queues. For a particular kind of job, we want to schedule at most one task per task tracker. How does one do this? We are using Hadoop 0.20.205.0.

    Thanks,

    --
    Nitin Khandelwal
  • Nitin Khandelwal at Dec 21, 2011 at 5:03 am
    Hi Arun,

    I didnot get you completely. Are you trying to say that i should assign
    memory limits to map task so , that more than one wont be posted on each TT
    ? Also, I have multiple type of jobs (each in separate queue) which are
    IP limited. I want one map task for each type of job in a TT. So, if i
    assign memory limit to each type of job, will it work ? What i want to ask
    is that if one type of job is not running, is it possible at scheduler will
    assign 2 slots on same machine to some other type of job ?


    Thanks,
    Nitin

    On 20 December 2011 21:43, Arun C Murthy wrote:

    Just use multiple slots per each map.

    See:
    http://hadoop.apache.org/common/docs/stable/capacity_scheduler.html#Resource+based+scheduling

    Arun
    On Dec 20, 2011, at 3:46 AM, Nitin Khandelwal wrote:

    Hey,

    We use capacity scheduler and divide our map slots among queues. For a
    particular kind of job, we want to schedule at most one task per task
    tracker. How does one do this? We are using Hadoop 0.20.205.0.
    Thanks,

    --
    Nitin Khandelwal

    --


    Nitin Khandelwal

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedDec 20, '11 at 11:47a
activeDec 21, '11 at 5:03a
posts5
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase