[Hadoop-common-user] Cascading jobs in hadoop

Hi all,

I have a set of map red jobs which need to be cascaded ,i.e, output of MR
job1 is the input of MR job2. etc..

Can anyone point me to the corresponding classes in hadoop 0.20.0 API?

I have seen "x.addDependingJob(y)" function in the yahoo's hadoop tutorial
but that is for the older versions..
What is the similar thing in 0.20.0 API?

Any help is appreciated ,

Thanks
bharath.v
ug3
IIIT Hyderabad!

On Oct 2, 2009, at 3:29 AM, bharath v wrote:

Hi all,

I have a set of map red jobs which need to be cascaded ,i.e, output
of MR
job1 is the input of MR job2. etc..

Can anyone point me to the corresponding classes in hadoop 0.20.0 API?

I have seen "x.addDependingJob(y)" function in the yahoo's hadoop
tutorial
but that is for the older versions..
What is the similar thing in 0.20.0 API?

Any help is appreciated ,

Thanks
bharath.v
ug3
IIIT Hyderabad!

On 10/2/09, Chris K Wensel wrote:
You might find the Cascading project quite useful in this regard.

http://www.cascading.org/

using MapReduceFlow and CascadeConnector classes, you can chain
arbitrary MR jobs together. Cascading will determine the dependencies,
if any, and run the jobs in topological order (independent jobs will
be submitted to run in parallel).

you may also find writing your own MR jobs by hand tedious and
brittle. Cascading can help you there as well.

cheers,
chris
--
Chris K Wensel
chris@concurrentinc.com
http://www.concurrentinc.com

On Sat, Oct 3, 2009 at 10:29 AM, bharath vissapragada wrote:

Tom and Chris ,

Thanks for your replies .. I have seen thr o.a.h.mapred.jobcontrol.Job
and o.a.h.mapreduce.Job .. Only one of them has the above option of
adding a dependent Jobs .. Can anyone tell me the difference between
"mapred" and "mapreduce" packages ..

Thanks in advance