FAQ
Hi all,

I want to know if anyone had already done an in-depth analysis of the
MapReduce mechanism. Has anyone really gone into bytecode level
understanding of the Map and Reduce mechanism. It would be good if we
can take a simple MapReduce (say WordCount) and then try the analysis.
Please send me pointers to if there s already some work done in this
respect. Or please help me with how to proceed with the same analysis
if you feel a specific technique/software/development environment has
ready plugins to help in this regard.

thanks,
Matthew John

Search Discussions

  • Matei Zaharia at Feb 17, 2011 at 7:50 am
    Do you mean profiling the data path in MapReduce? I think the general consensus is that a decent amount of time is spent in deserialization and in data copies in the HDFS stack, although of course there is work to improve this. For example, take a look at https://issues.apache.org/jira/browse/HDFS-347 for optimizations for the HDFS read path (at least for local data). My guess is that these are two of the more surprising things you'll see in a profile. Of course, for many jobs, the tasks might be IO-bound, so this may not matter.

    Matei

    On Feb 16, 2011, at 11:40 PM, Matthew John wrote:

    Hi all,

    I want to know if anyone had already done an in-depth analysis of the
    MapReduce mechanism. Has anyone really gone into bytecode level
    understanding of the Map and Reduce mechanism. It would be good if we
    can take a simple MapReduce (say WordCount) and then try the analysis.
    Please send me pointers to if there s already some work done in this
    respect. Or please help me with how to proceed with the same analysis
    if you feel a specific technique/software/development environment has
    ready plugins to help in this regard.

    thanks,
    Matthew John

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedFeb 17, '11 at 7:40a
activeFeb 17, '11 at 7:50a
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Matthew John: 1 post Matei Zaharia: 1 post

People

Translate

site design / logo © 2022 Grokbase