[MapReduce-user] Programming Multiple rounds of mapreduce

Hello,

I am trying to write a program where I need to write multiple rounds
of map and reduce.

The output of the last round of map-reduce must be fed into the input
of the next round.

Can anyone please guide me to any link / material that can teach me as
to how I can achieve this.

Thanks a lot in advance!

Thanks & regards
Arko

Hello,

I am trying to write a program where I need to write multiple rounds
of map and reduce.

The output of the last round of map-reduce must be fed into the input
of the next round.

Can anyone please guide me to any link / material that can teach me as
to how I can achieve this.

Thanks a lot in advance!

Thanks & regards
Arko

On Mon, Jun 13, 2011 at 3:01 PM, GOEKE, MATTHEW (AG/1000) wrote:

If you know for certain that it needs to be split into multiple work units
I would suggest looking into Oozie. Easy to install, light weight, low
learning curve... for my purposes it's been very helpful so far. I am also
fairly certain you can chain multiple job confs into the same run but I have
not actually tried that therefore I can't promise it is easy or possible.

http://www.cloudera.com/blog/2010/07/whats-new-in-cdh3-b2-oozie/

If you are not running CDH3u0 then you can also get the tarball and
documentation directly here:
https://ccp.cloudera.com/display/SUPPORT/CDH3+Downloadable+Tarballs

Matt

-----Original Message-----
From: Marcos Ortiz
Sent: Monday, June 13, 2011 4:57 PM
To: mapreduce-user@hadoop.apache.org
Cc: Arko Provo Mukherjee
Subject: Re: Programming Multiple rounds of mapreduce

Well, you can define a job for each round and then, you can define the
running workflow based in your implementation and to chain your jobs

El 6/13/2011 5:46 PM, Arko Provo Mukherjee escribió:
--
Marcos Luís Ortíz Valmaseda
Software Engineer (UCI)
http://marcosluis2186.posterous.com
http://twitter.com/marcosluis2186


This e-mail message may contain privileged and/or confidential information,
and is intended to be received only by persons entitled
to receive such information. If you have received this e-mail in error,
please notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use
of this e-mail by you is strictly prohibited.

All e-mails and attachments sent and received are subject to monitoring,
reading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for
checking for the presence of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage
caused by any such code transmitted by or accompanying
this e-mail or any attachment.


The information contained in this email may be subject to the export
control laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR)
and sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this
information you are obligated to comply with all
applicable U.S. export laws and regulations.

On Mon, Jun 13, 2011 at 5:46 PM, Arko Provo Mukherjee wrote:

Hello,

I am trying to write a program where I need to write multiple rounds of map
and reduce.

The output of the last round of map-reduce must be fed into the input of
the next round.

Can anyone please guide me to any link / material that can teach me as to
how I can achieve this.

Thanks a lot in advance!

Thanks & regards
Arko

On Mon, Jun 13, 2011 at 5:30 PM, Moustafa Gaber wrote:

I think HaLoop is a framework which can answer your question:
http://code.google.com/p/haloop/


On Mon, Jun 13, 2011 at 5:46 PM, Arko Provo Mukherjee <
arkoprovomukherjee@gmail.com> wrote:


--
Best Regards,
Mostafa Ead

On Mon, Jun 13, 2011 at 6:39 PM, Arko Provo Mukherjee wrote:

Hello,

Thanks everyone for your responses.

I am new to Hadoop, so this was a lot of new information for me. I will
surely go though all of these.

However, I was actually hoping that someone could point me to some example
codes where multiple rounds of map-reduce has been used.

Please let me know if anyone has any such examples as they are the best way
to learn for me :-)

Thanks much!
Cheers
Arko

Hello,

I am trying to write a program where I need to write multiple rounds of map
and reduce.

The output of the last round of map-reduce must be fed into the input of the
next round.

Can anyone please guide me to any link / material that can teach me as to
how I can achieve this.

Thanks a lot in advance!

Thanks & regards
Arko