FAQ
Hi,

How can I modify the task assignment strategy in hadoop which is used to
assign tasks to different worker nodes? (Not the job scheduler)

Big thanks,

Shen

Search Discussions

  • Jeff Zhang at Oct 8, 2010 at 1:34 am
    I believe it is possible. But what is your purpose? I believe current
    solution is good enough.




    On Fri, Oct 8, 2010 at 2:57 AM, Shen LI wrote:
    Hi,
    How can I modify the task assignment strategy in hadoop which is used to
    assign tasks to different worker nodes? (Not the job scheduler)
    Big thanks,
    Shen


    --
    Best Regards

    Jeff Zhang
  • Shen LI at Oct 8, 2010 at 2:38 am
    Hi, Thanks you very much for your reply. I want to run my own algorithm for
    this part to see if we can achieve better outcome in specific scenario. So
    how can I modify it?

    Thanks a lot!

    Shen
    On Thu, Oct 7, 2010 at 6:33 PM, Jeff Zhang wrote:

    I believe it is possible. But what is your purpose? I believe current
    solution is good enough.




    On Fri, Oct 8, 2010 at 2:57 AM, Shen LI wrote:
    Hi,
    How can I modify the task assignment strategy in hadoop which is used to
    assign tasks to different worker nodes? (Not the job scheduler)
    Big thanks,
    Shen


    --
    Best Regards

    Jeff Zhang
  • Jeff Zhang at Oct 8, 2010 at 4:31 am
    Currently, TaskTracker send heatbeat to JobTracker, and JobTracker
    will send actions to TaskTracker according TaskTracker's status. I
    think you should do some hacking about the heatbeat part of JobTracker

    public synchronized HeartbeatResponse heartbeat(TaskTrackerStatus status,
    boolean restarted,
    boolean initialContact,
    boolean acceptNewTasks,
    short responseId)


    On Fri, Oct 8, 2010 at 10:38 AM, Shen LI wrote:
    Hi, Thanks you very much for your reply. I want to run my own algorithm for
    this part  to see if we can achieve better outcome in specific scenario. So
    how can I modify it?
    Thanks a lot!
    Shen
    On Thu, Oct 7, 2010 at 6:33 PM, Jeff Zhang wrote:

    I believe it is possible. But what is your purpose? I believe current
    solution is good enough.




    On Fri, Oct 8, 2010 at 2:57 AM, Shen LI wrote:
    Hi,
    How can I modify the task assignment strategy in hadoop which is used to
    assign tasks to different worker nodes? (Not the job scheduler)
    Big thanks,
    Shen


    --
    Best Regards

    Jeff Zhang


    --
    Best Regards

    Jeff Zhang
  • Ken Goodhope at Oct 10, 2010 at 7:22 pm
    Another option might be to extend fair scheduler's task selector. The
    code is already set up to make this a configurable abstraction point.
    One word of caution, if you are using the cloudera distro, you will
    find that fair scheduler has been modified to not use those classes
    anymore. Something that wasn't documented very well. Still works with
    20.2. Not sure about 21.
    On Thursday, October 7, 2010, Jeff Zhang wrote:
    Currently, TaskTracker send heatbeat to JobTracker, and JobTracker
    will send actions to TaskTracker according TaskTracker's status. I
    think you should do some hacking about the heatbeat part of JobTracker

    public synchronized HeartbeatResponse heartbeat(TaskTrackerStatus status,
    boolean restarted,
    boolean initialContact,
    boolean acceptNewTasks,
    short responseId)


    On Fri, Oct 8, 2010 at 10:38 AM, Shen LI wrote:
    Hi, Thanks you very much for your reply. I want to run my own algorithm for
    this part  to see if we can achieve better outcome in specific scenario. So
    how can I modify it?
    Thanks a lot!
    Shen
    On Thu, Oct 7, 2010 at 6:33 PM, Jeff Zhang wrote:

    I believe it is possible. But what is your purpose? I believe current
    solution is good enough.




    On Fri, Oct 8, 2010 at 2:57 AM, Shen LI wrote:
    Hi,
    How can I modify the task assignment strategy in hadoop which is used to
    assign tasks to different worker nodes? (Not the job scheduler)
    Big thanks,
    Shen


    --
    Best Regards

    Jeff Zhang


    --
    Best Regards

    Jeff Zhang
  • Todd Lipcon at Oct 14, 2010 at 8:17 pm

    On Sun, Oct 10, 2010 at 7:22 PM, Ken Goodhope wrote:
    Another option might be to extend fair scheduler's task selector. The
    code is already set up to make this a configurable abstraction point.
    One word of caution, if you are using the cloudera distro, you will
    find that fair scheduler has been modified to not use those classes
    anymore. Something that wasn't documented very well. Still works with
    20.2. Not sure about 21.
    Just to clarify, we don't have any custom non-Apache code in our fair
    scheduler in CDH. Rather, we've backported newer fair scheduler work
    from 0.21 and trunk into 0.20. Most importantly, we backported
    MAPREDUCE-706 which changed a lot of the internals.

    Thanks
    -Todd
    On Thursday, October 7, 2010, Jeff Zhang wrote:
    Currently, TaskTracker send heatbeat to JobTracker, and JobTracker
    will send actions to TaskTracker according TaskTracker's status. I
    think you should do some hacking about the heatbeat part of JobTracker

    public synchronized HeartbeatResponse heartbeat(TaskTrackerStatus status,
    boolean restarted,
    boolean initialContact,
    boolean acceptNewTasks,
    short responseId)


    On Fri, Oct 8, 2010 at 10:38 AM, Shen LI wrote:
    Hi, Thanks you very much for your reply. I want to run my own algorithm for
    this part  to see if we can achieve better outcome in specific scenario. So
    how can I modify it?
    Thanks a lot!
    Shen
    On Thu, Oct 7, 2010 at 6:33 PM, Jeff Zhang wrote:

    I believe it is possible. But what is your purpose? I believe current
    solution is good enough.




    On Fri, Oct 8, 2010 at 2:57 AM, Shen LI wrote:
    Hi,
    How can I modify the task assignment strategy in hadoop which is used to
    assign tasks to different worker nodes? (Not the job scheduler)
    Big thanks,
    Shen


    --
    Best Regards

    Jeff Zhang


    --
    Best Regards

    Jeff Zhang


    --
    Todd Lipcon
    Software Engineer, Cloudera

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedOct 7, '10 at 6:58p
activeOct 14, '10 at 8:17p
posts6
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase