FAQ
hi all!

could someone please point out key differences between hadoop code and
Amazon's Elastic MapReduce. I am particularly interested in ways that
hadoop code is changed/optimized to run on efficiently EC2.

cheers!
momina

Search Discussions

  • Sirota, Peter at Sep 9, 2012 at 7:39 pm
    Hi,

    The major differences are in s3 file system that has been rewritten in EMR and in Hadoop interactions with S3. Other differences are in detecting various failure conditions.

    Outside these it's Apache Hadoop. Here is a list of patches EMR applied on top of 1.0.3 Hadoop
    http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/EnvironmentConfig_AMIHadoopPatches.html

    Regards,
    Peter


    On Sep 9, 2012, at 11:29 AM, "Momina Khan" wrote:

    hi all!

    could someone please point out key differences between hadoop code and
    Amazon's Elastic MapReduce. I am particularly interested in ways that
    hadoop code is changed/optimized to run on efficiently EC2.

    cheers!
    momina
  • Sudhir Kylasa at Sep 9, 2012 at 7:45 pm
    Hi,

    I am Sudhir Kylasa, a PhD student @ Purdue University.

    I just started with Hadoop MR and other related open projects with Apache.

    Can someone please point me to a place where I can find instructions on how to setup Eclipse with HadoopMR development environment.

    I want to download a stable load of Hadoop MR apply my own code snippets on top of it and install the executables on my cluster here.

    Please help.

    Thanks
    Sudhir
  • Karthik Kambatla at Sep 10, 2012 at 12:52 am
    Hi Sudhir

    http://wiki.apache.org/hadoop/EclipseEnvironment is a good place to start
    for your eclipse setup. For deploying a cluster off of your local changes,
    you can either package it (mvn package) or just build the jars and replace
    the jars in an existing setup.

    Hope that helps.
    Karthik
    On Sun, Sep 9, 2012 at 12:44 PM, Sudhir Kylasa wrote:

    Hi,

    I am Sudhir Kylasa, a PhD student @ Purdue University.

    I just started with Hadoop MR and other related open projects with Apache.

    Can someone please point me to a place where I can find instructions on
    how to setup Eclipse with HadoopMR development environment.

    I want to download a stable load of Hadoop MR apply my own code snippets
    on top of it and install the executables on my cluster here.

    Please help.

    Thanks
    Sudhir
  • Eli Collins at Sep 9, 2012 at 8:04 pm
    Peter,

    Thanks for the info. Do you guys plan to contribute the rewritten s3
    code (assume you're referring to org.apache.hadoop.fs.s3) back to
    Apache?

    Thanks,
    Eli
    On Sun, Sep 9, 2012 at 12:38 PM, Sirota, Peter wrote:
    Hi,

    The major differences are in s3 file system that has been rewritten in EMR and in Hadoop interactions with S3. Other differences are in detecting various failure conditions.

    Outside these it's Apache Hadoop. Here is a list of patches EMR applied on top of 1.0.3 Hadoop
    http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/EnvironmentConfig_AMIHadoopPatches.html

    Regards,
    Peter


    On Sep 9, 2012, at 11:29 AM, "Momina Khan" wrote:

    hi all!

    could someone please point out key differences between hadoop code and
    Amazon's Elastic MapReduce. I am particularly interested in ways that
    hadoop code is changed/optimized to run on efficiently EC2.

    cheers!
    momina
  • Momina Khan at Sep 10, 2012 at 4:41 am
    thank u that was very helpful!
    On Mon, Sep 10, 2012 at 1:04 AM, Eli Collins wrote:

    Peter,

    Thanks for the info. Do you guys plan to contribute the rewritten s3
    code (assume you're referring to org.apache.hadoop.fs.s3) back to
    Apache?

    Thanks,
    Eli
    On Sun, Sep 9, 2012 at 12:38 PM, Sirota, Peter wrote:
    Hi,

    The major differences are in s3 file system that has been rewritten in
    EMR and in Hadoop interactions with S3. Other differences are in detecting
    various failure conditions.
    Outside these it's Apache Hadoop. Here is a list of patches EMR applied
    on top of 1.0.3 Hadoop

    http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/EnvironmentConfig_AMIHadoopPatches.html
    Regards,
    Peter


    On Sep 9, 2012, at 11:29 AM, "Momina Khan" wrote:

    hi all!

    could someone please point out key differences between hadoop code and
    Amazon's Elastic MapReduce. I am particularly interested in ways that
    hadoop code is changed/optimized to run on efficiently EC2.

    cheers!
    momina

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedSep 9, '12 at 6:29p
activeSep 10, '12 at 4:41a
posts6
users5
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase