FAQ
When I run the sort job, I found when there are 70 reduce tasks running and
no one completed, the progress bar shows that it has finished about 80%, so
how the mapreduce mechnism to caculate this?

Also, when I run a job, as we know, we can determine the number of total
reduce tasks through setNumReduceTasks() function, but how to determine the
reducer number(I mean the tasktracker number which run the reduce task)
being used?

Thanks!
Stan. Lee

Search Discussions

  • 原攀峰 at May 17, 2010 at 4:37 am
    For a reduce task, the execution is divided into three phases, each of which accounts for 1/3 of the score:
    • The copy phase, when the task fetches map outputs.
    • The sort phase, when map outputs are sorted by key.
    • The reduce phase, when a user-defined function is applied to the list of map outputs with each key.



    在2010-05-17 09:44:38,"stan lee" <lee.stan065@gmail.com> 写道:
    When I run the sort job, I found when there are 70 reduce tasks running and
    no one completed, the progress bar shows that it has finished about 80%, so
    how the mapreduce mechnism to caculate this?

    Also, when I run a job, as we know, we can determine the number of total
    reduce tasks through setNumReduceTasks() function, but how to determine the
    reducer number(I mean the tasktracker number which run the reduce task)
    being used?

    Thanks!
    Stan. Lee
  • 原攀峰 at May 17, 2010 at 4:45 am
    For a reduce task, the execution is divided into three phases, each of which accounts for 1/3 of the score:
    • The copy phase, when the task fetches map outputs.
    • The sort phase, when map outputs are sorted by key.
    • The reduce phase, when a user-defined function is applied to the list of map outputs with each key.
    --

    Yuan Panfeng(原攀峰) | BeiHang University

    TEL: +86-13426166934

    MSN: ypf412@hotmail.com

    EMAIL: ypf412@gmail.com

    QQ: 362889262




    在2010-05-17 09:44:38,"stan lee" <lee.stan065@gmail.com> 写道:
    When I run the sort job, I found when there are 70 reduce tasks running and
    no one completed, the progress bar shows that it has finished about 80%, so
    how the mapreduce mechnism to caculate this?

    Also, when I run a job, as we know, we can determine the number of total
    reduce tasks through setNumReduceTasks() function, but how to determine the
    reducer number(I mean the tasktracker number which run the reduce task)
    being used?

    Thanks!
    Stan. Lee
  • Stan lee at May 18, 2010 at 3:33 pm
    Thanks PanFeng, do you have more detailed explanation on this? Is it
    caculated by how many reduce files has completed each phase?

    Also, what's the answer for my second question? Thanks!
    On Mon, May 17, 2010 at 12:44 PM, 原攀峰 wrote:

    For a reduce task, the execution is divided into three phases, each of
    which accounts for 1/3 of the score:
    • The copy phase, when the task fetches map outputs.
    • The sort phase, when map outputs are sorted by key.
    • The reduce phase, when a user-defined function is applied to the list of
    map outputs with each key.
    --

    Yuan Panfeng(原攀峰) | BeiHang University

    TEL: +86-13426166934

    MSN: ypf412@hotmail.com

    EMAIL: ypf412@gmail.com

    QQ: 362889262




    在2010-05-17 09:44:38,"stan lee" <lee.stan065@gmail.com> 写道:
    When I run the sort job, I found when there are 70 reduce tasks running and
    no one completed, the progress bar shows that it has finished about 80%, so
    how the mapreduce mechnism to caculate this?

    Also, when I run a job, as we know, we can determine the number of total
    reduce tasks through setNumReduceTasks() function, but how to determine the
    reducer number(I mean the tasktracker number which run the reduce task)
    being used?

    Thanks!
    Stan. Lee

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMay 17, '10 at 1:45a
activeMay 18, '10 at 3:33p
posts4
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Stan lee: 2 posts 原攀峰: 2 posts

People

Translate

site design / logo © 2022 Grokbase