FAQ
Hello,

Where exactly(Java class) does the network data transfer happen from a
mapper to a reducer ? Mapper seems to be writing data to the HDFS. Does
reducer read data from the mapper's HDFS over the network ? This is evading
me. Thanks in advance for the help.

Cheers,
Aishwarya

Search Discussions

  • Jagaran das at May 29, 2011 at 1:08 am
    Mappers write to local persistance and reducers write to hdfs

    - Jagaran



    ________________________________
    From: Aishwarya Venkataraman <avenkata@cs.ucsd.edu>
    To: common-dev@hadoop.apache.org
    Sent: Sat, 28 May, 2011 4:11:13 PM
    Subject: Question regarding network data transfer

    Hello,

    Where exactly(Java class) does the network data transfer happen from a
    mapper to a reducer ? Mapper seems to be writing data to the HDFS. Does
    reducer read data from the mapper's HDFS over the network ? This is evading
    me. Thanks in advance for the help.

    Cheers,
    Aishwarya
  • Aishwarya Venkataraman at May 29, 2011 at 1:20 am
    So how does reducer obtain the mapper's output ? Does it make a network call
    and read data from mappers local storage or does the mapper send the data ?

    Thanks,
    Aishwarya
    On Sat, May 28, 2011 at 6:08 PM, jagaran das wrote:

    Mappers write to local persistance and reducers write to hdfs

    - Jagaran



    ________________________________
    From: Aishwarya Venkataraman <avenkata@cs.ucsd.edu>
    To: common-dev@hadoop.apache.org
    Sent: Sat, 28 May, 2011 4:11:13 PM
    Subject: Question regarding network data transfer

    Hello,

    Where exactly(Java class) does the network data transfer happen from a
    mapper to a reducer ? Mapper seems to be writing data to the HDFS. Does
    reducer read data from the mapper's HDFS over the network ? This is evading
    me. Thanks in advance for the help.

    Cheers,
    Aishwarya


    --
    Thanks,
    Aishwarya Venkataraman
    avenkata@cs.ucsd.edu
    Graduate Student | Department of Computer Science
    University of California, San Diego
  • Harsh J at May 29, 2011 at 2:49 am
    Aishwarya,

    On Sun, May 29, 2011 at 6:49 AM, Aishwarya Venkataraman
    wrote:
    So how does reducer obtain the mapper's output ? Does it make a network call
    and read data from mappers local storage or does the mapper send the data ?
    The mappers store the files at a location that is accessibly by the
    TaskTracker's HTTP servlet. The reducer fetches all successful map
    attempt outputs from the TaskTrackers when they initialize.

    --
    Harsh J

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMay 28, '11 at 11:11p
activeMay 29, '11 at 2:49a
posts4
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase