Grokbase › Groups › Hadoop › common-user › August 2007
FAQ
Hello:

According the Map/Reduce working flow, it runs map first, then reduce.
Could I run this pairs for several iterations(i.e. to run map/reduce several
times
and stop for certain criteria, like some iterative methods in
numerical-analysis)?
If the answer is yes, any example to do that?

---ChaoChun

--
View this message in context: http://www.nabble.com/How-to-run-Map-Reduce-for-iterative-algorithms-in-Hadoop--tf4303510.html#a12249705
Sent from the Hadoop Users mailing list archive at Nabble.com.

Search Discussions

  • Enis Soztutar at Aug 21, 2007 at 7:56 am ⇧
    Yes you can.

    If the job you are iterating is the same, then just put the job
    submission in a loop. For example :

    for(int i=0;i<something;i++) {
    if(isDone())
    break;

    JobConf job = new JobConf(conf);
    job.addInputPath("foo");
    job.setOutputPath("bar");
    //set job parameters accordingly
    JobClient.runJob(job);
    }



    ChaoChun Liang wrote:
    Hello:

    According the Map/Reduce working flow, it runs map first, then reduce.
    Could I run this pairs for several iterations(i.e. to run map/reduce several
    times
    and stop for certain criteria, like some iterative methods in
    numerical-analysis)?
    If the answer is yes, any example to do that?

    ---ChaoChun
  • ChaoChun Liang at Aug 21, 2007 at 3:41 pm ⇧

    Enis Soztutar wrote:

    for(int i=0;i<something;i++) {
    if(isDone())
    break;

    JobConf job = new JobConf(conf);
    job.addInputPath("foo");
    job.setOutputPath("bar");
    //set job parameters accordingly
    JobClient.runJob(job);
    }
    Can I get the result directly from the reduce and decide whether the next
    loop run or not?
    For example, I would like to get the residual value directly from the reduce
    and to run the next loop if this value satisfied some criteria?

    ChaoChun

    --
    View this message in context: http://www.nabble.com/How-to-run-Map-Reduce-for-iterative-algorithms-in-Hadoop--tf4303510.html#a12257260
    Sent from the Hadoop Users mailing list archive at Nabble.com.
  • Andrzej Bialecki at Aug 21, 2007 at 4:01 pm ⇧

    ChaoChun Liang wrote:

    Enis Soztutar wrote:
    for(int i=0;i<something;i++) {
    if(isDone())
    break;

    JobConf job = new JobConf(conf);
    job.addInputPath("foo");
    job.setOutputPath("bar");
    //set job parameters accordingly
    JobClient.runJob(job);
    }
    Can I get the result directly from the reduce and decide whether the next
    loop run or not?
    For example, I would like to get the residual value directly from the reduce
    and to run the next loop if this value satisfied some criteria?
    You can aggregate some (limited number) values using the Counters
    functionality, and retrieve them from the JobTracker when a job completes.


    --
    Best regards,
    Andrzej Bialecki <><
    ___. ___ ___ ___ _ _ __________________________________
    [__ || __|__/|__||\/| Information Retrieval, Semantic Web
    ___|||__|| \| || | Embedded Unix, System Integration
    http://www.sigram.com Contact: info at sigram dot com
  • Ted Dunning at Aug 21, 2007 at 4:12 pm ⇧
    There are status global variables that you can set that might help with
    that.

    On 8/21/07 8:40 AM, "ChaoChun Liang" wrote:



    Enis Soztutar wrote:
    for(int i=0;i<something;i++) {
    if(isDone())
    break;

    JobConf job = new JobConf(conf);
    job.addInputPath("foo");
    job.setOutputPath("bar");
    //set job parameters accordingly
    JobClient.runJob(job);
    }
    Can I get the result directly from the reduce and decide whether the next
    loop run or not?
    For example, I would like to get the residual value directly from the reduce
    and to run the next loop if this value satisfied some criteria?

    ChaoChun

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedAug 21, '07 at 7:34a
activeAug 21, '07 at 4:12p
posts5
users4
websitehadoop.apache.org...
irc#hadoop

4 users in discussion

ChaoChun Liang: 2 posts Enis Soztutar: 1 post Ted Dunning: 1 post Andrzej Bialecki: 1 post

Content

People

Support

Translate

site design / logo © 2023 Grokbase