FAQ
Hello guys,

Can anybody tell me which is the relation between map task and combine
tasks?
I would like to know if there is a 1:1 relation between them, or is a *:1
(many to one).

For example:
If I have *2 map tasks* ran on the same machine, will I have *1 combine task
* on that machine to combine the maps outputs, *or 2 combine tasks*?

Best Regards,
--
Lucian

Search Discussions

  • Devaraj K at Jul 1, 2011 at 11:48 am
    Hi Lucian,



    For every map task, combiner will be executed multiple times
    before writing the map output. Combine step is not a separate task and it is
    part of map task execution. Reducer will copy the output of the map task
    which is reduced by the combiner.


    For example:
    If I have 2 map tasks ran on the same machine, will I have 1 combine task
    on that machine to combine the maps outputs, or 2 combine tasks?



    In this case, combiner will be executed for each map task independent to
    each other. This combiner step will execute multiple times till it gets same
    output for 1 or more runs of combiner.





    You can go through Combiner section here for more info :
    http://wiki.apache.org/hadoop/HadoopMapReduce



    Devaraj K

    ----------------------------------------------------------------------------
    ---------------------------------------------------------
    This e-mail and its attachments contain confidential information from
    HUAWEI, which
    is intended only for the person or entity whose address is listed above. Any
    use of the
    information contained herein in any way (including, but not limited to,
    total or partial
    disclosure, reproduction, or dissemination) by persons other than the
    intended
    recipient(s) is prohibited. If you receive this e-mail in error, please
    notify the sender by
    phone or email immediately and delete it!ss



    _____

    From: Lucian Iordache
    Sent: Friday, July 01, 2011 2:25 PM
    To: mapreduce-user@hadoop.apache.org
    Subject: Relation between Mapper and Combiner



    Hello guys,

    Can anybody tell me which is the relation between map task and combine
    tasks?
    I would like to know if there is a 1:1 relation between them, or is a *:1
    (many to one).

    For example:
    If I have 2 map tasks ran on the same machine, will I have 1 combine task on
    that machine to combine the maps outputs, or 2 combine tasks?

    Best Regards,
    --
    Lucian
  • Lucian Iordache at Jul 1, 2011 at 2:24 pm
    Ok, that is what I wanted to know.

    Thanks you!

    Best Regards,
    Lucian
    On Fri, Jul 1, 2011 at 2:47 PM, Devaraj K wrote:

    Hi Lucian,****

    ** **

    For every map task, combiner will be executed multiple times
    before writing the map output. Combine step is not a separate task and it is
    part of map task execution. Reducer will copy the output of the map task
    which is reduced by the combiner.****

    ** **
    For example:
    If I have *2 map tasks* ran on the same machine, will I have *1 combine
    task* on that machine to combine the maps outputs, *or 2 combine tasks*?**
    **

    ** **

    In this case, combiner will be executed for each map task independent to
    each other. This combiner step will execute multiple times till it gets same
    output for 1 or more runs of combiner.

    ****

    ** **

    You can go through Combiner section here for more info :
    http://wiki.apache.org/hadoop/HadoopMapReduce****

    ** **

    Devaraj K ****


    -------------------------------------------------------------------------------------------------------------------------------------
    This e-mail and its attachments contain confidential information from
    HUAWEI, which
    is intended only for the person or entity whose address is listed above.
    Any use of the
    information contained herein in any way (including, but not limited to,
    total or partial
    disclosure, reproduction, or dissemination) by persons other than the
    intended
    recipient(s) is prohibited. If you receive this e-mail in error, please
    notify the sender by
    phone or email immediately and delete it!ss****

    ****
    ------------------------------

    *From:* Lucian Iordache
    *Sent:* Friday, July 01, 2011 2:25 PM
    *To:* mapreduce-user@hadoop.apache.org
    *Subject:* Relation between Mapper and Combiner****

    ** **

    Hello guys,

    Can anybody tell me which is the relation between map task and combine
    tasks?
    I would like to know if there is a 1:1 relation between them, or is a *:1
    (many to one).

    For example:
    If I have *2 map tasks* ran on the same machine, will I have *1 combine
    task* on that machine to combine the maps outputs, *or 2 combine tasks*?

    Best Regards,
    --
    Lucian****

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedJul 1, '11 at 8:56a
activeJul 1, '11 at 2:24p
posts3
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Lucian Iordache: 2 posts Devaraj K: 1 post

People

Translate

site design / logo © 2022 Grokbase