FAQ
Hi,
Is there any research currently going on where map reduce is applied to
nodes in normal internet scenarios?.In environments where network bandwidth
is at premium what are the tweaks applied to hadoop?
I would be very thankful if you can post me links in this direction.

--
Regards,
R.V.

Search Discussions

  • Alex Loddengaard at Aug 3, 2010 at 4:38 pm
    I don't know of any research, but such a scenario is likely not going to
    turn out so well. Hadoop is very network hungry and is designed to be run
    in a datacenter. Sorry I don't have more information for you.

    Alex
    On Mon, Aug 2, 2010 at 9:14 PM, Rahul.V. wrote:

    Hi,
    Is there any research currently going on where map reduce is applied to
    nodes in normal internet scenarios?.In environments where network bandwidth
    is at premium what are the tweaks applied to hadoop?
    I would be very thankful if you can post me links in this direction.

    --
    Regards,
    R.V.
  • He Chen at Aug 3, 2010 at 8:00 pm
    Condor has a hadoop subproject in UW-Madison, and there are also some
    scientists from VT. They worked on security Hadoop MapReduce on Internet.

    In my opinion, Alex is correct, Hadoop MR is communication intensive
    especially in the map and shuffle stage. In the map stage, every mapper
    needs input data from File System. If your data distributed among Internet,
    you may encounter heavy delay. Also in the shuffle stage, reducer collect
    mapper's intermediate results through Internet. This is another bottleneck
    we can not overlook.

    Hope this will help.

    Chen
    On Tue, Aug 3, 2010 at 11:37 AM, Alex Loddengaard wrote:

    I don't know of any research, but such a scenario is likely not going to
    turn out so well. Hadoop is very network hungry and is designed to be run
    in a datacenter. Sorry I don't have more information for you.

    Alex

    On Mon, Aug 2, 2010 at 9:14 PM, Rahul.V. <greatness.hardness@gmail.com
    wrote:
    Hi,
    Is there any research currently going on where map reduce is applied to
    nodes in normal internet scenarios?.In environments where network bandwidth
    is at premium what are the tweaks applied to hadoop?
    I would be very thankful if you can post me links in this direction.

    --
    Regards,
    R.V.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedAug 3, '10 at 4:14a
activeAug 3, '10 at 8:00p
posts3
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase