FAQ
Hi,

I have some questions related to basic functionality in Hadoop.

1. When a Mapper process the intermediate output data, how it knows how many partitions to do(how many reducers will be) and how much data to go in each  partition for each reducer ?

2. A JobTracker when assigns a task to a reducer, it will also specify the locations of intermediate output data where it should retrieve it right ? But how a reducer will know from each remote location with intermediate output what portion it has to retrieve only ?

Could somebody help me with these questions together with pointing me out where I can find the java code doing that ? I am running Hadoop 1.0.3.


Thanks,
Robert

Search Discussions

  • Harsh J at Jul 8, 2012 at 5:35 am
    Hi Robert,

    Inline. (Answer is specific to Hadoop 1.x since you asked for that
    alone, but certain things may vary for Hadoop 2.x).
    On Sun, Jul 8, 2012 at 7:07 AM, Grandl Robert wrote:
    Hi,

    I have some questions related to basic functionality in Hadoop.

    1. When a Mapper process the intermediate output data, how it knows how many
    partitions to do(how many reducers will be) and how much data to go in each
    partition for each reducer ?
    The number of reducers is non-dynamic and is user-specified, and is
    set in the job configuration. Hence the Partitioner knows about the
    value it needs to use for its numPartitions (== numReduces for the
    job).

    For this one in 1.x code, look at MapTask.java, in the constructors of
    internal classes OldOutputCollector (Stable API) and
    NewOutputCollector (New API).

    The data estimated to be going into a partition, for limit/scheduling
    checks, is currently a naive computation, done by summing upon the
    estimate output sizes of each map. See
    ResourceEstimator#getEstimatedReduceInputSize for the overall
    estimation across maps, and see Task#calculateOutputSize for the
    per-map estimation code.
    2. A JobTracker when assigns a task to a reducer, it will also specify the
    locations of intermediate output data where it should retrieve it right ?
    But how a reducer will know from each remote location with intermediate
    output what portion it has to retrieve only ?
    The JT does not send in the information of locations when a reduce is
    scheduled. When the reducers begin their shuffle phase, they query the
    TaskTracker to get the map completion events, via
    TaskTracker#getMapCompletionEvents protocol call. The TaskTracker by
    itself calls the JobTracker#getTaskCompletionEvents protocol call to
    get this info underneath. The returned structure carries the host that
    has completed the map successfully, which the Reduce's copier relies
    on to fetch the data from the right host's TT.

    The reduce merely asks the data assigned for it for the specific
    completed maps at each TT. Note that a reduce task ID is also its
    partition ID, so it merely has to ask the data for its own task ID #
    and the TT serves, over HTTP, the right parts of the intermediate data
    to it.

    Feel free to ping back if you need some more clarification! :)

    --
    Harsh J
  • Harsh J at Jul 9, 2012 at 1:17 am
    The changes should appear in your Task's userlogs (not the TaskTracker
    logs). Have you deployed your changed code properly (i.e. do you
    generate a new tarball, or perhaps use the MRMiniCluster to do this)?
    On Mon, Jul 9, 2012 at 4:57 AM, Grandl Robert wrote:
    Hi Harsh,

    Your comments were extremely helpful.

    Still I am wondering why if I add LOG.info entries into MapTask.java or
    ReduceTask.java in most of the functions(including Old/NewOutputCollector),
    the logs are not shown. In this way it's hard for me to track which
    functions are called and which not. Even more in ReduceTask.java.

    Do you have any ideas ?

    Thanks a lot for your answer,
    Robert

    ________________________________
    From: Harsh J <harsh@cloudera.com>
    To: mapreduce-user@hadoop.apache.org; Grandl Robert <rgrandl@yahoo.com>
    Sent: Sunday, July 8, 2012 1:34 AM

    Subject: Re: Basic question on how reducer works

    Hi Robert,

    Inline. (Answer is specific to Hadoop 1.x since you asked for that
    alone, but certain things may vary for Hadoop 2.x).
    On Sun, Jul 8, 2012 at 7:07 AM, Grandl Robert wrote:
    Hi,

    I have some questions related to basic functionality in Hadoop.

    1. When a Mapper process the intermediate output data, how it knows how
    many
    partitions to do(how many reducers will be) and how much data to go in
    each
    partition for each reducer ?
    The number of reducers is non-dynamic and is user-specified, and is
    set in the job configuration. Hence the Partitioner knows about the
    value it needs to use for its numPartitions (== numReduces for the
    job).

    For this one in 1.x code, look at MapTask.java, in the constructors of
    internal classes OldOutputCollector (Stable API) and
    NewOutputCollector (New API).

    The data estimated to be going into a partition, for limit/scheduling
    checks, is currently a naive computation, done by summing upon the
    estimate output sizes of each map. See
    ResourceEstimator#getEstimatedReduceInputSize for the overall
    estimation across maps, and see Task#calculateOutputSize for the
    per-map estimation code.
    2. A JobTracker when assigns a task to a reducer, it will also specify the
    locations of intermediate output data where it should retrieve it right ?
    But how a reducer will know from each remote location with intermediate
    output what portion it has to retrieve only ?
    The JT does not send in the information of locations when a reduce is
    scheduled. When the reducers begin their shuffle phase, they query the
    TaskTracker to get the map completion events, via
    TaskTracker#getMapCompletionEvents protocol call. The TaskTracker by
    itself calls the JobTracker#getTaskCompletionEvents protocol call to
    get this info underneath. The returned structure carries the host that
    has completed the map successfully, which the Reduce's copier relies
    on to fetch the data from the right host's TT.

    The reduce merely asks the data assigned for it for the specific
    completed maps at each TT. Note that a reduce task ID is also its
    partition ID, so it merely has to ask the data for its own task ID #
    and the TT serves, over HTTP, the right parts of the intermediate data
    to it.

    Feel free to ping back if you need some more clarification! :)

    --
    Harsh J


    --
    Harsh J
  • Grandl Robert at Jul 9, 2012 at 1:27 am
    I see. I was looking into tasktracker log :).

    Thanks a lot,
    Robert



    ________________________________
    From: Harsh J <harsh@cloudera.com>
    To: Grandl Robert <rgrandl@yahoo.com>; mapreduce-user <mapreduce-user@hadoop.apache.org>
    Sent: Sunday, July 8, 2012 9:16 PM
    Subject: Re: Basic question on how reducer works

    The changes should appear in your Task's userlogs (not the TaskTracker
    logs). Have you deployed your changed code properly (i.e. do you
    generate a new tarball, or perhaps use the MRMiniCluster to do this)?
    On Mon, Jul 9, 2012 at 4:57 AM, Grandl Robert wrote:
    Hi Harsh,

    Your comments were extremely helpful.

    Still I am wondering why if I add LOG.info entries into MapTask.java or
    ReduceTask.java in most of the functions(including Old/NewOutputCollector),
    the logs are not shown. In this way it's hard for me to track which
    functions are called and which not. Even more in ReduceTask.java.

    Do you have any ideas ?

    Thanks a lot for your answer,
    Robert

    ________________________________
    From: Harsh J <harsh@cloudera.com>
    To: mapreduce-user@hadoop.apache.org; Grandl Robert <rgrandl@yahoo.com>
    Sent: Sunday, July 8, 2012 1:34 AM

    Subject: Re: Basic question on how reducer works

    Hi Robert,

    Inline. (Answer is specific to Hadoop 1.x since you asked for that
    alone, but certain things may vary for Hadoop 2.x).
    On Sun, Jul 8, 2012 at 7:07 AM, Grandl Robert wrote:
    Hi,

    I have some questions related to basic functionality in Hadoop.

    1. When a Mapper process the intermediate output data, how it knows how
    many
    partitions to do(how many reducers will be) and how much data to go in
    each
    partition for each reducer ?
    The number of reducers is non-dynamic and is user-specified, and is
    set in the job configuration. Hence the Partitioner knows about the
    value it needs to use for its numPartitions (== numReduces for the
    job).

    For this one in 1.x code, look at MapTask.java, in the constructors of
    internal classes OldOutputCollector (Stable API) and
    NewOutputCollector (New API).

    The data estimated to be going into a partition, for limit/scheduling
    checks, is currently a naive computation, done by summing upon the
    estimate output sizes of each map. See
    ResourceEstimator#getEstimatedReduceInputSize for the overall
    estimation across maps, and see Task#calculateOutputSize for the
    per-map estimation code.
    2. A JobTracker when assigns a task to a reducer, it will also specify the
    locations of intermediate output data where it should retrieve it right ?
    But how a reducer will know from each remote location with intermediate
    output what portion it has to retrieve only ?
    The JT does not send in the information of locations when a reduce is
    scheduled. When the reducers begin their shuffle phase, they query the
    TaskTracker to get the map completion events, via
    TaskTracker#getMapCompletionEvents protocol call. The TaskTracker by
    itself calls the JobTracker#getTaskCompletionEvents protocol call to
    get this info underneath. The returned structure carries the host that
    has completed the map successfully, which the Reduce's copier relies
    on to fetch the data from the right host's TT.

    The reduce merely asks the data assigned for it for the specific
    completed maps at each TT. Note that a reduce task ID is also its
    partition ID, so it merely has to ask the data for its own task ID #
    and the TT serves, over HTTP, the right parts of the intermediate data
    to it.

    Feel free to ping back if you need some more clarification! :)

    --
    Harsh J


    --
    Harsh J
  • Pavan Kulkarni at Jul 9, 2012 at 2:57 am
    I too had similar problems.
    I guess we should also set the debug mode for
    that specific class in the log4j.properties file .Isn't it?

    And I didn't quite get what you mean by task's userlogs?
    where are these logs located ? In the logs directory I only see
    logs for all the daemons.Thanks

    On Sun, Jul 8, 2012 at 6:27 PM, Grandl Robert wrote:

    I see. I was looking into tasktracker log :).

    Thanks a lot,
    Robert

    ------------------------------
    *From:* Harsh J <harsh@cloudera.com>
    *To:* Grandl Robert <rgrandl@yahoo.com>; mapreduce-user <
    mapreduce-user@hadoop.apache.org>
    *Sent:* Sunday, July 8, 2012 9:16 PM

    *Subject:* Re: Basic question on how reducer works

    The changes should appear in your Task's userlogs (not the TaskTracker
    logs). Have you deployed your changed code properly (i.e. do you
    generate a new tarball, or perhaps use the MRMiniCluster to do this)?
    On Mon, Jul 9, 2012 at 4:57 AM, Grandl Robert wrote:
    Hi Harsh,

    Your comments were extremely helpful.

    Still I am wondering why if I add LOG.info <http://log.info/> entries
    into MapTask.java or
    ReduceTask.java in most of the functions(including
    Old/NewOutputCollector),
    the logs are not shown. In this way it's hard for me to track which
    functions are called and which not. Even more in ReduceTask.java.

    Do you have any ideas ?

    Thanks a lot for your answer,
    Robert

    ________________________________
    From: Harsh J <harsh@cloudera.com>
    To: mapreduce-user@hadoop.apache.org; Grandl Robert <rgrandl@yahoo.com>
    Sent: Sunday, July 8, 2012 1:34 AM

    Subject: Re: Basic question on how reducer works

    Hi Robert,

    Inline. (Answer is specific to Hadoop 1.x since you asked for that
    alone, but certain things may vary for Hadoop 2.x).
    On Sun, Jul 8, 2012 at 7:07 AM, Grandl Robert wrote:
    Hi,

    I have some questions related to basic functionality in Hadoop.

    1. When a Mapper process the intermediate output data, how it knows how
    many
    partitions to do(how many reducers will be) and how much data to go in
    each
    partition for each reducer ?
    The number of reducers is non-dynamic and is user-specified, and is
    set in the job configuration. Hence the Partitioner knows about the
    value it needs to use for its numPartitions (== numReduces for the
    job).

    For this one in 1.x code, look at MapTask.java, in the constructors of
    internal classes OldOutputCollector (Stable API) and
    NewOutputCollector (New API).

    The data estimated to be going into a partition, for limit/scheduling
    checks, is currently a naive computation, done by summing upon the
    estimate output sizes of each map. See
    ResourceEstimator#getEstimatedReduceInputSize for the overall
    estimation across maps, and see Task#calculateOutputSize for the
    per-map estimation code.
    2. A JobTracker when assigns a task to a reducer, it will also specify
    the
    locations of intermediate output data where it should retrieve it right
    ?
    But how a reducer will know from each remote location with intermediate
    output what portion it has to retrieve only ?
    The JT does not send in the information of locations when a reduce is
    scheduled. When the reducers begin their shuffle phase, they query the
    TaskTracker to get the map completion events, via
    TaskTracker#getMapCompletionEvents protocol call. The TaskTracker by
    itself calls the JobTracker#getTaskCompletionEvents protocol call to
    get this info underneath. The returned structure carries the host that
    has completed the map successfully, which the Reduce's copier relies
    on to fetch the data from the right host's TT.

    The reduce merely asks the data assigned for it for the specific
    completed maps at each TT. Note that a reduce task ID is also its
    partition ID, so it merely has to ask the data for its own task ID #
    and the TT serves, over HTTP, the right parts of the intermediate data
    to it.

    Feel free to ping back if you need some more clarification! :)

    --
    Harsh J


    --
    Harsh J


    --

    --With Regards
    Pavan Kulkarni
  • Harsh J at Jul 9, 2012 at 3:39 am
    Pavan,

    This is covered in the MR tutorial doc:
    http://hadoop.apache.org/common/docs/stable/mapred_tutorial.html#Task+Logs
    On Mon, Jul 9, 2012 at 8:26 AM, Pavan Kulkarni wrote:
    I too had similar problems.
    I guess we should also set the debug mode for
    that specific class in the log4j.properties file .Isn't it?

    And I didn't quite get what you mean by task's userlogs?
    where are these logs located ? In the logs directory I only see
    logs for all the daemons.Thanks

    On Sun, Jul 8, 2012 at 6:27 PM, Grandl Robert wrote:

    I see. I was looking into tasktracker log :).

    Thanks a lot,
    Robert

    ________________________________
    From: Harsh J <harsh@cloudera.com>
    To: Grandl Robert <rgrandl@yahoo.com>; mapreduce-user
    <mapreduce-user@hadoop.apache.org>
    Sent: Sunday, July 8, 2012 9:16 PM

    Subject: Re: Basic question on how reducer works

    The changes should appear in your Task's userlogs (not the TaskTracker
    logs). Have you deployed your changed code properly (i.e. do you
    generate a new tarball, or perhaps use the MRMiniCluster to do this)?
    On Mon, Jul 9, 2012 at 4:57 AM, Grandl Robert wrote:
    Hi Harsh,

    Your comments were extremely helpful.

    Still I am wondering why if I add LOG.info entries into MapTask.java or
    ReduceTask.java in most of the functions(including
    Old/NewOutputCollector),
    the logs are not shown. In this way it's hard for me to track which
    functions are called and which not. Even more in ReduceTask.java.

    Do you have any ideas ?

    Thanks a lot for your answer,
    Robert

    ________________________________
    From: Harsh J <harsh@cloudera.com>
    To: mapreduce-user@hadoop.apache.org; Grandl Robert <rgrandl@yahoo.com>
    Sent: Sunday, July 8, 2012 1:34 AM

    Subject: Re: Basic question on how reducer works

    Hi Robert,

    Inline. (Answer is specific to Hadoop 1.x since you asked for that
    alone, but certain things may vary for Hadoop 2.x).
    On Sun, Jul 8, 2012 at 7:07 AM, Grandl Robert wrote:
    Hi,

    I have some questions related to basic functionality in Hadoop.

    1. When a Mapper process the intermediate output data, how it knows how
    many
    partitions to do(how many reducers will be) and how much data to go in
    each
    partition for each reducer ?
    The number of reducers is non-dynamic and is user-specified, and is
    set in the job configuration. Hence the Partitioner knows about the
    value it needs to use for its numPartitions (== numReduces for the
    job).

    For this one in 1.x code, look at MapTask.java, in the constructors of
    internal classes OldOutputCollector (Stable API) and
    NewOutputCollector (New API).

    The data estimated to be going into a partition, for limit/scheduling
    checks, is currently a naive computation, done by summing upon the
    estimate output sizes of each map. See
    ResourceEstimator#getEstimatedReduceInputSize for the overall
    estimation across maps, and see Task#calculateOutputSize for the
    per-map estimation code.
    2. A JobTracker when assigns a task to a reducer, it will also specify
    the
    locations of intermediate output data where it should retrieve it right
    ?
    But how a reducer will know from each remote location with intermediate
    output what portion it has to retrieve only ?
    The JT does not send in the information of locations when a reduce is
    scheduled. When the reducers begin their shuffle phase, they query the
    TaskTracker to get the map completion events, via
    TaskTracker#getMapCompletionEvents protocol call. The TaskTracker by
    itself calls the JobTracker#getTaskCompletionEvents protocol call to
    get this info underneath. The returned structure carries the host that
    has completed the map successfully, which the Reduce's copier relies
    on to fetch the data from the right host's TT.

    The reduce merely asks the data assigned for it for the specific
    completed maps at each TT. Note that a reduce task ID is also its
    partition ID, so it merely has to ask the data for its own task ID #
    and the TT serves, over HTTP, the right parts of the intermediate data
    to it.

    Feel free to ping back if you need some more clarification! :)

    --
    Harsh J


    --
    Harsh J


    --

    --With Regards
    Pavan Kulkarni


    --
    Harsh J
  • Pavan Kulkarni at Jul 9, 2012 at 4:12 am
    Oh.Thanks a lot Harsh .
    On Sun, Jul 8, 2012 at 11:38 PM, Harsh J wrote:

    Pavan,

    This is covered in the MR tutorial doc:
    http://hadoop.apache.org/common/docs/stable/mapred_tutorial.html#Task+Logs
    On Mon, Jul 9, 2012 at 8:26 AM, Pavan Kulkarni wrote:
    I too had similar problems.
    I guess we should also set the debug mode for
    that specific class in the log4j.properties file .Isn't it?

    And I didn't quite get what you mean by task's userlogs?
    where are these logs located ? In the logs directory I only see
    logs for all the daemons.Thanks

    On Sun, Jul 8, 2012 at 6:27 PM, Grandl Robert wrote:

    I see. I was looking into tasktracker log :).

    Thanks a lot,
    Robert

    ________________________________
    From: Harsh J <harsh@cloudera.com>
    To: Grandl Robert <rgrandl@yahoo.com>; mapreduce-user
    <mapreduce-user@hadoop.apache.org>
    Sent: Sunday, July 8, 2012 9:16 PM

    Subject: Re: Basic question on how reducer works

    The changes should appear in your Task's userlogs (not the TaskTracker
    logs). Have you deployed your changed code properly (i.e. do you
    generate a new tarball, or perhaps use the MRMiniCluster to do this)?
    On Mon, Jul 9, 2012 at 4:57 AM, Grandl Robert wrote:
    Hi Harsh,

    Your comments were extremely helpful.

    Still I am wondering why if I add LOG.info entries into MapTask.java
    or
    ReduceTask.java in most of the functions(including
    Old/NewOutputCollector),
    the logs are not shown. In this way it's hard for me to track which
    functions are called and which not. Even more in ReduceTask.java.

    Do you have any ideas ?

    Thanks a lot for your answer,
    Robert

    ________________________________
    From: Harsh J <harsh@cloudera.com>
    To: mapreduce-user@hadoop.apache.org; Grandl Robert <
    rgrandl@yahoo.com>
    Sent: Sunday, July 8, 2012 1:34 AM

    Subject: Re: Basic question on how reducer works

    Hi Robert,

    Inline. (Answer is specific to Hadoop 1.x since you asked for that
    alone, but certain things may vary for Hadoop 2.x).
    On Sun, Jul 8, 2012 at 7:07 AM, Grandl Robert wrote:
    Hi,

    I have some questions related to basic functionality in Hadoop.

    1. When a Mapper process the intermediate output data, how it knows
    how
    many
    partitions to do(how many reducers will be) and how much data to go
    in
    each
    partition for each reducer ?
    The number of reducers is non-dynamic and is user-specified, and is
    set in the job configuration. Hence the Partitioner knows about the
    value it needs to use for its numPartitions (== numReduces for the
    job).

    For this one in 1.x code, look at MapTask.java, in the constructors of
    internal classes OldOutputCollector (Stable API) and
    NewOutputCollector (New API).

    The data estimated to be going into a partition, for limit/scheduling
    checks, is currently a naive computation, done by summing upon the
    estimate output sizes of each map. See
    ResourceEstimator#getEstimatedReduceInputSize for the overall
    estimation across maps, and see Task#calculateOutputSize for the
    per-map estimation code.
    2. A JobTracker when assigns a task to a reducer, it will also
    specify
    the
    locations of intermediate output data where it should retrieve it
    right
    ?
    But how a reducer will know from each remote location with
    intermediate
    output what portion it has to retrieve only ?
    The JT does not send in the information of locations when a reduce is
    scheduled. When the reducers begin their shuffle phase, they query the
    TaskTracker to get the map completion events, via
    TaskTracker#getMapCompletionEvents protocol call. The TaskTracker by
    itself calls the JobTracker#getTaskCompletionEvents protocol call to
    get this info underneath. The returned structure carries the host that
    has completed the map successfully, which the Reduce's copier relies
    on to fetch the data from the right host's TT.

    The reduce merely asks the data assigned for it for the specific
    completed maps at each TT. Note that a reduce task ID is also its
    partition ID, so it merely has to ask the data for its own task ID #
    and the TT serves, over HTTP, the right parts of the intermediate data
    to it.

    Feel free to ping back if you need some more clarification! :)

    --
    Harsh J


    --
    Harsh J


    --

    --With Regards
    Pavan Kulkarni


    --
    Harsh J


    --

    --With Regards
    Pavan Kulkarni
  • Arun C Murthy at Jul 9, 2012 at 1:25 pm
    Robert,
    On Jul 7, 2012, at 6:37 PM, Grandl Robert wrote:

    Hi,

    I have some questions related to basic functionality in Hadoop.

    1. When a Mapper process the intermediate output data, how it knows how many partitions to do(how many reducers will be) and how much data to go in each partition for each reducer ?

    2. A JobTracker when assigns a task to a reducer, it will also specify the locations of intermediate output data where it should retrieve it right ? But how a reducer will know from each remote location with intermediate output what portion it has to retrieve only ?
    To add to Harsh's comment. Essentially the TT *knows* where the output of a given map-id/reduce-id pair is present via an output-file/index-file combination.

    Arun

    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/
  • Manoj Babu at Jul 9, 2012 at 5:52 pm
    Hi,

    It would be more helpful, If you could more details for the below doubts.

    1, How the partitioner knows which reducer needs to be called?
    2, When we are using more than one reducers, the output gets separated.
    Actually for what scenario we have to go for multiple reducers?

    Cheers!
    Manoj.


    On Mon, Jul 9, 2012 at 6:54 PM, Arun C Murthy wrote:

    Robert,

    On Jul 7, 2012, at 6:37 PM, Grandl Robert wrote:

    Hi,

    I have some questions related to basic functionality in Hadoop.

    1. When a Mapper process the intermediate output data, how it knows how
    many partitions to do(how many reducers will be) and how much data to go in
    each partition for each reducer ?

    2. A JobTracker when assigns a task to a reducer, it will also specify the
    locations of intermediate output data where it should retrieve it right ?
    But how a reducer will know from each remote location with intermediate
    output what portion it has to retrieve only ?


    To add to Harsh's comment. Essentially the TT *knows* where the output of
    a given map-id/reduce-id pair is present via an output-file/index-file
    combination.

    Arun

    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/

  • Harsh J at Jul 9, 2012 at 5:58 pm
    Manoj,

    Think of it this way, and you shouldn't be confused: A reducer == a partition.

    For (1) - Partitioners do not 'call' a reduce, just write the data
    with a proper partition ID. The reducer thats same as the partition
    ID, picks it up for itself later. This we have already explained
    earlier.

    For (2) - For what scenario do you _not_ want multiple reducers
    handling each partition uniquely, when it is possible to scale that
    way?
    On Mon, Jul 9, 2012 at 11:22 PM, Manoj Babu wrote:
    Hi,

    It would be more helpful, If you could more details for the below doubts.

    1, How the partitioner knows which reducer needs to be called?
    2, When we are using more than one reducers, the output gets separated.
    Actually for what scenario we have to go for multiple reducers?

    Cheers!
    Manoj.


    On Mon, Jul 9, 2012 at 6:54 PM, Arun C Murthy wrote:

    Robert,

    On Jul 7, 2012, at 6:37 PM, Grandl Robert wrote:

    Hi,

    I have some questions related to basic functionality in Hadoop.

    1. When a Mapper process the intermediate output data, how it knows how
    many partitions to do(how many reducers will be) and how much data to go in
    each partition for each reducer ?

    2. A JobTracker when assigns a task to a reducer, it will also specify the
    locations of intermediate output data where it should retrieve it right ?
    But how a reducer will know from each remote location with intermediate
    output what portion it has to retrieve only ?


    To add to Harsh's comment. Essentially the TT *knows* where the output of
    a given map-id/reduce-id pair is present via an output-file/index-file
    combination.

    Arun

    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/


    --
    Harsh J
  • Manoj Babu at Jul 9, 2012 at 6:08 pm
    Hi Harsh,

    Thanks for clarifying. I was in thought earlier that Partitioner is picking
    the reducer.

    My cluster setup provides options for multiple reducers so i want to know
    when and in which scenario we have go for multiple reducers?

    Cheers!
    Manoj.


    On Mon, Jul 9, 2012 at 11:27 PM, Harsh J wrote:

    Manoj,

    Think of it this way, and you shouldn't be confused: A reducer == a
    partition.

    For (1) - Partitioners do not 'call' a reduce, just write the data
    with a proper partition ID. The reducer thats same as the partition
    ID, picks it up for itself later. This we have already explained
    earlier.

    For (2) - For what scenario do you _not_ want multiple reducers
    handling each partition uniquely, when it is possible to scale that
    way?
    On Mon, Jul 9, 2012 at 11:22 PM, Manoj Babu wrote:
    Hi,

    It would be more helpful, If you could more details for the below doubts.

    1, How the partitioner knows which reducer needs to be called?
    2, When we are using more than one reducers, the output gets separated.
    Actually for what scenario we have to go for multiple reducers?

    Cheers!
    Manoj.


    On Mon, Jul 9, 2012 at 6:54 PM, Arun C Murthy wrote:

    Robert,

    On Jul 7, 2012, at 6:37 PM, Grandl Robert wrote:

    Hi,

    I have some questions related to basic functionality in Hadoop.

    1. When a Mapper process the intermediate output data, how it knows how
    many partitions to do(how many reducers will be) and how much data to
    go in
    each partition for each reducer ?

    2. A JobTracker when assigns a task to a reducer, it will also specify
    the
    locations of intermediate output data where it should retrieve it right
    ?
    But how a reducer will know from each remote location with intermediate
    output what portion it has to retrieve only ?


    To add to Harsh's comment. Essentially the TT *knows* where the output
    of
    a given map-id/reduce-id pair is present via an output-file/index-file
    combination.

    Arun

    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/


    --
    Harsh J
  • Karthik Kambatla at Jul 9, 2012 at 6:13 pm
    Hi Manoj,

    As Harsh said, we would almost always need multiple reducers. As each
    reduce is potentially executed on a different core (same machine or a
    different one), in most cases, we would want at least as many reduces as
    the number of cores for maximum parallelism/performance.

    Karthik
    On Mon, Jul 9, 2012 at 11:07 AM, Manoj Babu wrote:

    Hi Harsh,

    Thanks for clarifying. I was in thought earlier that Partitioner is
    picking the reducer.

    My cluster setup provides options for multiple reducers so i want to know
    when and in which scenario we have go for multiple reducers?

    Cheers!
    Manoj.


    On Mon, Jul 9, 2012 at 11:27 PM, Harsh J wrote:

    Manoj,

    Think of it this way, and you shouldn't be confused: A reducer == a
    partition.

    For (1) - Partitioners do not 'call' a reduce, just write the data
    with a proper partition ID. The reducer thats same as the partition
    ID, picks it up for itself later. This we have already explained
    earlier.

    For (2) - For what scenario do you _not_ want multiple reducers
    handling each partition uniquely, when it is possible to scale that
    way?
    On Mon, Jul 9, 2012 at 11:22 PM, Manoj Babu wrote:
    Hi,

    It would be more helpful, If you could more details for the below doubts.
    1, How the partitioner knows which reducer needs to be called?
    2, When we are using more than one reducers, the output gets separated.
    Actually for what scenario we have to go for multiple reducers?

    Cheers!
    Manoj.



    On Mon, Jul 9, 2012 at 6:54 PM, Arun C Murthy <acm@hortonworks.com>
    wrote:
    Robert,

    On Jul 7, 2012, at 6:37 PM, Grandl Robert wrote:

    Hi,

    I have some questions related to basic functionality in Hadoop.

    1. When a Mapper process the intermediate output data, how it knows how
    many partitions to do(how many reducers will be) and how much data to
    go in
    each partition for each reducer ?

    2. A JobTracker when assigns a task to a reducer, it will also specify
    the
    locations of intermediate output data where it should retrieve it
    right ?
    But how a reducer will know from each remote location with intermediate
    output what portion it has to retrieve only ?


    To add to Harsh's comment. Essentially the TT *knows* where the output
    of
    a given map-id/reduce-id pair is present via an output-file/index-file
    combination.

    Arun

    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/


    --
    Harsh J
  • Grandl Robert at Jul 9, 2012 at 7:55 pm
    Thanks a lot guys for answers.


    Still I am not able to find exactly the code for the following things:

    1. reducer to read from a Map output only its partition. I looked into ReduceTask#getMapOutput which do the actual read in ReduceTask#shuffleInMemory, but I don't see where it specify which partition to read(reduceID).

    2. still don't understand very well in which part of the code(MapTask.java) the intermediate data is written do which partition. So MapOutputBuffer is the one who actually writes the data to buffer and spill after buffer is full. Could you please elaborate a bit on how the data is written to which partition ?

    Thanks,
    Robert



    ________________________________
    From: Arun C Murthy <acm@hortonworks.com>
    To: mapreduce-user@hadoop.apache.org
    Sent: Monday, July 9, 2012 9:24 AM
    Subject: Re: Basic question on how reducer works


    Robert,


    On Jul 7, 2012, at 6:37 PM, Grandl Robert wrote:

    Hi,

    I have some questions related to basic functionality in Hadoop.


    1. When a Mapper process the intermediate output data, how it knows how many partitions to do(how many reducers will be) and how much data to go in each  partition for each reducer ?


    2. A JobTracker when assigns a task to a reducer, it will also specify the locations of intermediate output data where it should retrieve it right ? But how a reducer will know from each remote location with intermediate output what portion it has to retrieve only ?
    To add to Harsh's comment. Essentially the TT *knows* where the output of a given map-id/reduce-id pair is present via an output-file/index-file combination.

    Arun

    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/
  • Arun C Murthy at Jul 9, 2012 at 8:34 pm

    On Jul 9, 2012, at 12:55 PM, Grandl Robert wrote:

    Thanks a lot guys for answers.

    Still I am not able to find exactly the code for the following things:

    1. reducer to read from a Map output only its partition. I looked into ReduceTask#getMapOutput which do the actual read in ReduceTask#shuffleInMemory, but I don't see where it specify which partition to read(reduceID).
    Look at TaskTracker.MapOutputServlet.
    2. still don't understand very well in which part of the code(MapTask.java) the intermediate data is written do which partition. So MapOutputBuffer is the one who actually writes the data to buffer and spill after buffer is full. Could you please elaborate a bit on how the data is written to which partition ?
    Essentially you can think of the partition-id as the 'primary key' and the actual 'key' in the map-output of <key, value> as the 'secondary key'.

    hth,
    Arun
    Thanks,
    Robert

    From: Arun C Murthy <acm@hortonworks.com>
    To: mapreduce-user@hadoop.apache.org
    Sent: Monday, July 9, 2012 9:24 AM
    Subject: Re: Basic question on how reducer works

    Robert,
    On Jul 7, 2012, at 6:37 PM, Grandl Robert wrote:

    Hi,

    I have some questions related to basic functionality in Hadoop.

    1. When a Mapper process the intermediate output data, how it knows how many partitions to do(how many reducers will be) and how much data to go in each partition for each reducer ?

    2. A JobTracker when assigns a task to a reducer, it will also specify the locations of intermediate output data where it should retrieve it right ? But how a reducer will know from each remote location with intermediate output what portion it has to retrieve only ?
    To add to Harsh's comment. Essentially the TT *knows* where the output of a given map-id/reduce-id pair is present via an output-file/index-file combination.

    Arun

    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/


    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/
  • Grandl Robert at Jul 10, 2012 at 3:16 am
    Thanks Arun.

    So just for my clarification. The map will create partitions according to the number of reducers s.t. each reducer to get almost same number of keys in its partition. However, each key can have different number of values so the "weight" of each partition will depend on that. Also when a new <key, value> is added into a partition a hash on the partition ID will be computed to find the corresponding partition ?

    Robert



    ________________________________
    From: Arun C Murthy <acm@hortonworks.com>
    To: mapreduce-user@hadoop.apache.org
    Sent: Monday, July 9, 2012 4:33 PM
    Subject: Re: Basic question on how reducer works




    On Jul 9, 2012, at 12:55 PM, Grandl Robert wrote:

    Thanks a lot guys for answers.


    Still I am not able to find exactly the code for the following things:


    1. reducer to read from a Map output only its partition. I looked into ReduceTask#getMapOutput which do the actual read in ReduceTask#shuffleInMemory, but I don't see where it specify which partition to read(reduceID).
    Look at TaskTracker.MapOutputServlet.


    2. still don't understand very well in which part of the code(MapTask.java) the intermediate data is written do which partition. So MapOutputBuffer is the one who actually writes the data to buffer and spill after buffer is full. Could you please elaborate a bit on how the data is written to which partition ?
    Essentially you can think of the partition-id as the 'primary key' and the actual 'key' in the map-output of <key, value> as the 'secondary key'.

    hth,
    Arun


    Thanks,
    Robert



    ________________________________
    From: Arun C Murthy <acm@hortonworks.com>
    To: mapreduce-user@hadoop.apache.org
    Sent: Monday, July 9, 2012 9:24 AM
    Subject: Re: Basic question on how reducer works


    Robert,


    On Jul 7, 2012, at 6:37 PM, Grandl Robert wrote:

    Hi,

    I have some questions related to basic functionality in Hadoop.


    1. When a Mapper process the intermediate output data, how it knows how many partitions to do(how many reducers will be) and how much data to go in each  partition for each reducer ?


    2. A JobTracker when assigns a task to a reducer, it will also specify the locations of intermediate output data where it should retrieve it right ? But how a reducer will know from each remote location with intermediate output what portion it has to retrieve only ?
    To add to Harsh's comment. Essentially the TT *knows* where the output of a given map-id/reduce-id pair is present via an output-file/index-file combination.


    Arun


    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/



    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/
  • Karthik Kambatla at Jul 10, 2012 at 3:34 am
    The partitioner is configurable. The default partitioner, from what I
    remember, computes the partition as the hashcode modulo number of
    reducers/partitions. For random input, it is balanced, but some cases can
    have very skewed key distribution. Also, as you have pointed out, the
    number of values per key can also vary. Together, both of them determine
    "weight" of each partition as you call it.

    Karthik
    On Mon, Jul 9, 2012 at 8:15 PM, Grandl Robert wrote:

    Thanks Arun.

    So just for my clarification. The map will create partitions according to
    the number of reducers s.t. each reducer to get almost same number of keys
    in its partition. However, each key can have different number of values so
    the "weight" of each partition will depend on that. Also when a new <key,
    value> is added into a partition a hash on the partition ID will be
    computed to find the corresponding partition ?

    Robert

    ------------------------------
    *From:* Arun C Murthy <acm@hortonworks.com>
    *To:* mapreduce-user@hadoop.apache.org
    *Sent:* Monday, July 9, 2012 4:33 PM

    *Subject:* Re: Basic question on how reducer works


    On Jul 9, 2012, at 12:55 PM, Grandl Robert wrote:

    Thanks a lot guys for answers.

    Still I am not able to find exactly the code for the following things:

    1. reducer to read from a Map output only its partition. I looked into
    ReduceTask#getMapOutput which do the actual read in
    ReduceTask#shuffleInMemory, but I don't see where it specify which
    partition to read(reduceID).


    Look at TaskTracker.MapOutputServlet.

    2. still don't understand very well in which part of the
    code(MapTask.java) the intermediate data is written do which partition. So
    MapOutputBuffer is the one who actually writes the data to buffer and spill
    after buffer is full. Could you please elaborate a bit on how the data is
    written to which partition ?


    Essentially you can think of the partition-id as the 'primary key' and the
    actual 'key' in the map-output of <key, value> as the 'secondary key'.

    hth,
    Arun

    Thanks,
    Robert

    ------------------------------
    *From:* Arun C Murthy <acm@hortonworks.com>
    *To:* mapreduce-user@hadoop.apache.org
    *Sent:* Monday, July 9, 2012 9:24 AM
    *Subject:* Re: Basic question on how reducer works

    Robert,

    On Jul 7, 2012, at 6:37 PM, Grandl Robert wrote:

    Hi,

    I have some questions related to basic functionality in Hadoop.

    1. When a Mapper process the intermediate output data, how it knows how
    many partitions to do(how many reducers will be) and how much data to go in
    each partition for each reducer ?

    2. A JobTracker when assigns a task to a reducer, it will also specify the
    locations of intermediate output data where it should retrieve it right ?
    But how a reducer will know from each remote location with intermediate
    output what portion it has to retrieve only ?


    To add to Harsh's comment. Essentially the TT *knows* where the output of
    a given map-id/reduce-id pair is present via an output-file/index-file
    combination.

    Arun

    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/





    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/



  • Subir S at Jul 10, 2012 at 3:30 pm
    Is there any property to convey the maximum amount of data each
    reducer/partition may take for processing. Like the bytes_per_reducer
    of pig, so that the count of reducers can be controlled based on size
    of intermediate map output data size?
    On 7/10/12, Karthik Kambatla wrote:
    The partitioner is configurable. The default partitioner, from what I
    remember, computes the partition as the hashcode modulo number of
    reducers/partitions. For random input, it is balanced, but some cases can
    have very skewed key distribution. Also, as you have pointed out, the
    number of values per key can also vary. Together, both of them determine
    "weight" of each partition as you call it.

    Karthik
    On Mon, Jul 9, 2012 at 8:15 PM, Grandl Robert wrote:

    Thanks Arun.

    So just for my clarification. The map will create partitions according to
    the number of reducers s.t. each reducer to get almost same number of
    keys
    in its partition. However, each key can have different number of values
    so
    the "weight" of each partition will depend on that. Also when a new <key,
    value> is added into a partition a hash on the partition ID will be
    computed to find the corresponding partition ?

    Robert

    ------------------------------
    *From:* Arun C Murthy <acm@hortonworks.com>
    *To:* mapreduce-user@hadoop.apache.org
    *Sent:* Monday, July 9, 2012 4:33 PM

    *Subject:* Re: Basic question on how reducer works


    On Jul 9, 2012, at 12:55 PM, Grandl Robert wrote:

    Thanks a lot guys for answers.

    Still I am not able to find exactly the code for the following things:

    1. reducer to read from a Map output only its partition. I looked into
    ReduceTask#getMapOutput which do the actual read in
    ReduceTask#shuffleInMemory, but I don't see where it specify which
    partition to read(reduceID).


    Look at TaskTracker.MapOutputServlet.

    2. still don't understand very well in which part of the
    code(MapTask.java) the intermediate data is written do which partition.
    So
    MapOutputBuffer is the one who actually writes the data to buffer and
    spill
    after buffer is full. Could you please elaborate a bit on how the data is
    written to which partition ?


    Essentially you can think of the partition-id as the 'primary key' and
    the
    actual 'key' in the map-output of <key, value> as the 'secondary key'.

    hth,
    Arun

    Thanks,
    Robert

    ------------------------------
    *From:* Arun C Murthy <acm@hortonworks.com>
    *To:* mapreduce-user@hadoop.apache.org
    *Sent:* Monday, July 9, 2012 9:24 AM
    *Subject:* Re: Basic question on how reducer works

    Robert,

    On Jul 7, 2012, at 6:37 PM, Grandl Robert wrote:

    Hi,

    I have some questions related to basic functionality in Hadoop.

    1. When a Mapper process the intermediate output data, how it knows how
    many partitions to do(how many reducers will be) and how much data to go
    in
    each partition for each reducer ?

    2. A JobTracker when assigns a task to a reducer, it will also specify
    the
    locations of intermediate output data where it should retrieve it right ?
    But how a reducer will know from each remote location with intermediate
    output what portion it has to retrieve only ?


    To add to Harsh's comment. Essentially the TT *knows* where the output of
    a given map-id/reduce-id pair is present via an output-file/index-file
    combination.

    Arun

    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/





    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/



  • Subir S at Jul 14, 2012 at 5:50 am
    Probably a wrong question in a wrong thread and wrong mailing list :)
    On 7/10/12, Subir S wrote:
    Is there any property to convey the maximum amount of data each
    reducer/partition may take for processing. Like the bytes_per_reducer
    of pig, so that the count of reducers can be controlled based on size
    of intermediate map output data size?
    On 7/10/12, Karthik Kambatla wrote:
    The partitioner is configurable. The default partitioner, from what I
    remember, computes the partition as the hashcode modulo number of
    reducers/partitions. For random input, it is balanced, but some cases can
    have very skewed key distribution. Also, as you have pointed out, the
    number of values per key can also vary. Together, both of them determine
    "weight" of each partition as you call it.

    Karthik
    On Mon, Jul 9, 2012 at 8:15 PM, Grandl Robert wrote:

    Thanks Arun.

    So just for my clarification. The map will create partitions according
    to
    the number of reducers s.t. each reducer to get almost same number of
    keys
    in its partition. However, each key can have different number of values
    so
    the "weight" of each partition will depend on that. Also when a new
    <key,
    value> is added into a partition a hash on the partition ID will be
    computed to find the corresponding partition ?

    Robert

    ------------------------------
    *From:* Arun C Murthy <acm@hortonworks.com>
    *To:* mapreduce-user@hadoop.apache.org
    *Sent:* Monday, July 9, 2012 4:33 PM

    *Subject:* Re: Basic question on how reducer works


    On Jul 9, 2012, at 12:55 PM, Grandl Robert wrote:

    Thanks a lot guys for answers.

    Still I am not able to find exactly the code for the following things:

    1. reducer to read from a Map output only its partition. I looked into
    ReduceTask#getMapOutput which do the actual read in
    ReduceTask#shuffleInMemory, but I don't see where it specify which
    partition to read(reduceID).


    Look at TaskTracker.MapOutputServlet.

    2. still don't understand very well in which part of the
    code(MapTask.java) the intermediate data is written do which partition.
    So
    MapOutputBuffer is the one who actually writes the data to buffer and
    spill
    after buffer is full. Could you please elaborate a bit on how the data
    is
    written to which partition ?


    Essentially you can think of the partition-id as the 'primary key' and
    the
    actual 'key' in the map-output of <key, value> as the 'secondary key'.

    hth,
    Arun

    Thanks,
    Robert

    ------------------------------
    *From:* Arun C Murthy <acm@hortonworks.com>
    *To:* mapreduce-user@hadoop.apache.org
    *Sent:* Monday, July 9, 2012 9:24 AM
    *Subject:* Re: Basic question on how reducer works

    Robert,

    On Jul 7, 2012, at 6:37 PM, Grandl Robert wrote:

    Hi,

    I have some questions related to basic functionality in Hadoop.

    1. When a Mapper process the intermediate output data, how it knows how
    many partitions to do(how many reducers will be) and how much data to go
    in
    each partition for each reducer ?

    2. A JobTracker when assigns a task to a reducer, it will also specify
    the
    locations of intermediate output data where it should retrieve it right
    ?
    But how a reducer will know from each remote location with intermediate
    output what portion it has to retrieve only ?


    To add to Harsh's comment. Essentially the TT *knows* where the output
    of
    a given map-id/reduce-id pair is present via an output-file/index-file
    combination.

    Arun

    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/





    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/



  • Harsh J at Jul 14, 2012 at 6:09 am
    If you wish to impose a limit on the max reducer input to be allowed
    in a job, you may set "mapreduce.reduce.input.limit" on your job, as
    total bytes allowed per reducer.

    But this is more of a hard limit, which I suspect your question wasn't
    about. Your question is indeed better off on the pig's user lists.
    On Tue, Jul 10, 2012 at 8:59 PM, Subir S wrote:
    Is there any property to convey the maximum amount of data each
    reducer/partition may take for processing. Like the bytes_per_reducer
    of pig, so that the count of reducers can be controlled based on size
    of intermediate map output data size?
    On 7/10/12, Karthik Kambatla wrote:
    The partitioner is configurable. The default partitioner, from what I
    remember, computes the partition as the hashcode modulo number of
    reducers/partitions. For random input, it is balanced, but some cases can
    have very skewed key distribution. Also, as you have pointed out, the
    number of values per key can also vary. Together, both of them determine
    "weight" of each partition as you call it.

    Karthik
    On Mon, Jul 9, 2012 at 8:15 PM, Grandl Robert wrote:

    Thanks Arun.

    So just for my clarification. The map will create partitions according to
    the number of reducers s.t. each reducer to get almost same number of
    keys
    in its partition. However, each key can have different number of values
    so
    the "weight" of each partition will depend on that. Also when a new <key,
    value> is added into a partition a hash on the partition ID will be
    computed to find the corresponding partition ?

    Robert

    ------------------------------
    *From:* Arun C Murthy <acm@hortonworks.com>
    *To:* mapreduce-user@hadoop.apache.org
    *Sent:* Monday, July 9, 2012 4:33 PM

    *Subject:* Re: Basic question on how reducer works


    On Jul 9, 2012, at 12:55 PM, Grandl Robert wrote:

    Thanks a lot guys for answers.

    Still I am not able to find exactly the code for the following things:

    1. reducer to read from a Map output only its partition. I looked into
    ReduceTask#getMapOutput which do the actual read in
    ReduceTask#shuffleInMemory, but I don't see where it specify which
    partition to read(reduceID).


    Look at TaskTracker.MapOutputServlet.

    2. still don't understand very well in which part of the
    code(MapTask.java) the intermediate data is written do which partition.
    So
    MapOutputBuffer is the one who actually writes the data to buffer and
    spill
    after buffer is full. Could you please elaborate a bit on how the data is
    written to which partition ?


    Essentially you can think of the partition-id as the 'primary key' and
    the
    actual 'key' in the map-output of <key, value> as the 'secondary key'.

    hth,
    Arun

    Thanks,
    Robert

    ------------------------------
    *From:* Arun C Murthy <acm@hortonworks.com>
    *To:* mapreduce-user@hadoop.apache.org
    *Sent:* Monday, July 9, 2012 9:24 AM
    *Subject:* Re: Basic question on how reducer works

    Robert,

    On Jul 7, 2012, at 6:37 PM, Grandl Robert wrote:

    Hi,

    I have some questions related to basic functionality in Hadoop.

    1. When a Mapper process the intermediate output data, how it knows how
    many partitions to do(how many reducers will be) and how much data to go
    in
    each partition for each reducer ?

    2. A JobTracker when assigns a task to a reducer, it will also specify
    the
    locations of intermediate output data where it should retrieve it right ?
    But how a reducer will know from each remote location with intermediate
    output what portion it has to retrieve only ?


    To add to Harsh's comment. Essentially the TT *knows* where the output of
    a given map-id/reduce-id pair is present via an output-file/index-file
    combination.

    Arun

    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/





    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/





    --
    Harsh J
  • Subir S at Jul 14, 2012 at 12:00 pm
    Harsh, Thanks I think this is what I was looking for. I have 3 related
    questions.

    1.) Will this work in 0.20.2-cdh3u3

    2.) What is the hard limit that you mean?

    3.)Can this be applied for streaming?

    Thanks, Subir
    On 7/14/12, Harsh J wrote:
    If you wish to impose a limit on the max reducer input to be allowed
    in a job, you may set "mapreduce.reduce.input.limit" on your job, as
    total bytes allowed per reducer.

    But this is more of a hard limit, which I suspect your question wasn't
    about. Your question is indeed better off on the pig's user lists.
    On Tue, Jul 10, 2012 at 8:59 PM, Subir S wrote:
    Is there any property to convey the maximum amount of data each
    reducer/partition may take for processing. Like the bytes_per_reducer
    of pig, so that the count of reducers can be controlled based on size
    of intermediate map output data size?
    On 7/10/12, Karthik Kambatla wrote:
    The partitioner is configurable. The default partitioner, from what I
    remember, computes the partition as the hashcode modulo number of
    reducers/partitions. For random input, it is balanced, but some cases
    can
    have very skewed key distribution. Also, as you have pointed out, the
    number of values per key can also vary. Together, both of them determine
    "weight" of each partition as you call it.

    Karthik
    On Mon, Jul 9, 2012 at 8:15 PM, Grandl Robert wrote:

    Thanks Arun.

    So just for my clarification. The map will create partitions according
    to
    the number of reducers s.t. each reducer to get almost same number of
    keys
    in its partition. However, each key can have different number of values
    so
    the "weight" of each partition will depend on that. Also when a new
    <key,
    value> is added into a partition a hash on the partition ID will be
    computed to find the corresponding partition ?

    Robert

    ------------------------------
    *From:* Arun C Murthy <acm@hortonworks.com>
    *To:* mapreduce-user@hadoop.apache.org
    *Sent:* Monday, July 9, 2012 4:33 PM

    *Subject:* Re: Basic question on how reducer works


    On Jul 9, 2012, at 12:55 PM, Grandl Robert wrote:

    Thanks a lot guys for answers.

    Still I am not able to find exactly the code for the following things:

    1. reducer to read from a Map output only its partition. I looked into
    ReduceTask#getMapOutput which do the actual read in
    ReduceTask#shuffleInMemory, but I don't see where it specify which
    partition to read(reduceID).


    Look at TaskTracker.MapOutputServlet.

    2. still don't understand very well in which part of the
    code(MapTask.java) the intermediate data is written do which partition.
    So
    MapOutputBuffer is the one who actually writes the data to buffer and
    spill
    after buffer is full. Could you please elaborate a bit on how the data
    is
    written to which partition ?


    Essentially you can think of the partition-id as the 'primary key' and
    the
    actual 'key' in the map-output of <key, value> as the 'secondary key'.

    hth,
    Arun

    Thanks,
    Robert

    ------------------------------
    *From:* Arun C Murthy <acm@hortonworks.com>
    *To:* mapreduce-user@hadoop.apache.org
    *Sent:* Monday, July 9, 2012 9:24 AM
    *Subject:* Re: Basic question on how reducer works

    Robert,

    On Jul 7, 2012, at 6:37 PM, Grandl Robert wrote:

    Hi,

    I have some questions related to basic functionality in Hadoop.

    1. When a Mapper process the intermediate output data, how it knows how
    many partitions to do(how many reducers will be) and how much data to
    go
    in
    each partition for each reducer ?

    2. A JobTracker when assigns a task to a reducer, it will also specify
    the
    locations of intermediate output data where it should retrieve it right
    ?
    But how a reducer will know from each remote location with intermediate
    output what portion it has to retrieve only ?


    To add to Harsh's comment. Essentially the TT *knows* where the output
    of
    a given map-id/reduce-id pair is present via an output-file/index-file
    combination.

    Arun

    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/





    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/





    --
    Harsh J
  • Harsh J at Jul 14, 2012 at 1:56 pm
    Subir,
    On Sat, Jul 14, 2012 at 5:30 PM, Subir S wrote:
    Harsh, Thanks I think this is what I was looking for. I have 3 related
    questions.

    1.) Will this work in 0.20.2-cdh3u3
    Yes, will work. (Btw, best to ask CDH-specific questions on the
    cdh-user@cloudera.org lists)
    2.) What is the hard limit that you mean?
    If a reducer gets more data than this value, due to the map's outputs
    growing large (for any partition), the job will begin to fail.
    3.)Can this be applied for streaming?
    Yes, streaming is still MR and this property is for MR (applied during
    scheduling, so not streaming/java specific).

    --
    Harsh J
  • Subir S at Jul 16, 2012 at 8:32 pm
    Just for reference of others who might see this thread. Jira
    corresponding to parameter on reduce input limit is MAPREDUCE-2324
    On 7/14/12, Harsh J wrote:
    Subir,
    On Sat, Jul 14, 2012 at 5:30 PM, Subir S wrote:
    Harsh, Thanks I think this is what I was looking for. I have 3 related
    questions.

    1.) Will this work in 0.20.2-cdh3u3
    Yes, will work. (Btw, best to ask CDH-specific questions on the
    cdh-user@cloudera.org lists)
    2.) What is the hard limit that you mean?
    If a reducer gets more data than this value, due to the map's outputs
    growing large (for any partition), the job will begin to fail.
    3.)Can this be applied for streaming?
    Yes, streaming is still MR and this property is for MR (applied during
    scheduling, so not streaming/java specific).

    --
    Harsh J

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedJul 8, '12 at 1:39a
activeJul 16, '12 at 8:32p
posts22
users7
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2021 Grokbase