FAQ
I was following this tutorial on version 0.19.1

http://v-lad.org/Tutorials/Hadoop/23%20-%20create%20the%20project.html

I however wanted to use the latest version of api 0.20.2

The original code in tutorial had following lines
conf.setMapperClass(org.apache.hadoop.mapred.lib.IdentityMapper.class);
conf.setReducerClass(org.apache.hadoop.mapred.lib.IdentityReducer.class);

both Identity classes are deprecated.. So seemed the solution was to create
mapper and reducer as follows:
public static class NOOPMapper
extends Mapper<Text, IntWritable, Text, IntWritable>{


public void map(Text key, IntWritable value, Context context
) throws IOException, InterruptedException {

context.write(key, value);

}
}

public static class NOOPReducer
extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable<IntWritable> values,
Context context
) throws IOException, InterruptedException {
context.write(key, result);
}
}


And then with code:
Configuration conf = new Configuration();
Job job = new Job(conf, "testdriver");

job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);

job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);

FileInputFormat.addInputPath(job, new Path("In"));
FileOutputFormat.setOutputPath(job, new Path("Out"));

job.setMapperClass(NOOPMapper.class);
job.setReducerClass(NOOPReducer.class);

job.waitForCompletion(true);


However I get this message
java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be
cast to org.apache.hadoop.io.Text
at TestDriver$NOOPMapper.map(TestDriver.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
11/08/01 16:41:01 INFO mapred.JobClient: map 0% reduce 0%
11/08/01 16:41:01 INFO mapred.JobClient: Job complete: job_local_0001
11/08/01 16:41:01 INFO mapred.JobClient: Counters: 0



Can anyone tell me what I need for this to work.

Attached is full code..
http://old.nabble.com/file/p32174859/TestDriver.java TestDriver.java
--
View this message in context: http://old.nabble.com/maprd-vs-mapreduce-api-tp32174859p32174859.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Search Discussions

  • Roger Chen at Aug 2, 2011 at 3:46 am
    Your reducer is writing IntWritable but your output format class is still
    Text. Change one of those so they match the other.
    On Mon, Aug 1, 2011 at 8:40 PM, garpinc wrote:


    I was following this tutorial on version 0.19.1

    http://v-lad.org/Tutorials/Hadoop/23%20-%20create%20the%20project.html

    I however wanted to use the latest version of api 0.20.2

    The original code in tutorial had following lines
    conf.setMapperClass(org.apache.hadoop.mapred.lib.IdentityMapper.class);
    conf.setReducerClass(org.apache.hadoop.mapred.lib.IdentityReducer.class);

    both Identity classes are deprecated.. So seemed the solution was to create
    mapper and reducer as follows:
    public static class NOOPMapper
    extends Mapper<Text, IntWritable, Text, IntWritable>{


    public void map(Text key, IntWritable value, Context context
    ) throws IOException, InterruptedException {

    context.write(key, value);

    }
    }

    public static class NOOPReducer
    extends Reducer<Text,IntWritable,Text,IntWritable> {
    private IntWritable result = new IntWritable();

    public void reduce(Text key, Iterable<IntWritable> values,
    Context context
    ) throws IOException, InterruptedException {
    context.write(key, result);
    }
    }


    And then with code:
    Configuration conf = new Configuration();
    Job job = new Job(conf, "testdriver");

    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);

    job.setInputFormatClass(TextInputFormat.class);
    job.setOutputFormatClass(TextOutputFormat.class);

    FileInputFormat.addInputPath(job, new Path("In"));
    FileOutputFormat.setOutputPath(job, new Path("Out"));

    job.setMapperClass(NOOPMapper.class);
    job.setReducerClass(NOOPReducer.class);

    job.waitForCompletion(true);


    However I get this message
    java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be
    cast to org.apache.hadoop.io.Text
    at TestDriver$NOOPMapper.map(TestDriver.java:1)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at
    org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
    11/08/01 16:41:01 INFO mapred.JobClient: map 0% reduce 0%
    11/08/01 16:41:01 INFO mapred.JobClient: Job complete: job_local_0001
    11/08/01 16:41:01 INFO mapred.JobClient: Counters: 0



    Can anyone tell me what I need for this to work.

    Attached is full code..
    http://old.nabble.com/file/p32174859/TestDriver.java TestDriver.java
    --
    View this message in context:
    http://old.nabble.com/maprd-vs-mapreduce-api-tp32174859p32174859.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.

    --
    Roger Chen
    UC Davis Genome Center
  • Stevens, Keith D. at Aug 5, 2011 at 10:43 pm
    The Mapper and Reducer class in org.apache.hadoop.mapreduce implement the identity function. So you should be able to just do

    conf.setMapperClass(org.apache.hadoop.mapreduce.Mapper.class);
    conf.setReducerClass(org.apache.hadoop.mapreduce.Reducer.class);

    without having to implement your own no-op classes.

    I recommend reading the javadoc for differences between the old api and the new api, for example http://hadoop.apache.org/common/docs/r0.20.2/api/index.html indicates the different functionality of Mapper in the new api and it's dual use as the identity mapper.

    Cheers,
    --Keith
    On Aug 5, 2011, at 1:15 PM, garpinc wrote:


    I was following this tutorial on version 0.19.1

    http://v-lad.org/Tutorials/Hadoop/23%20-%20create%20the%20project.html

    I however wanted to use the latest version of api 0.20.2

    The original code in tutorial had following lines
    conf.setMapperClass(org.apache.hadoop.mapred.lib.IdentityMapper.class);
    conf.setReducerClass(org.apache.hadoop.mapred.lib.IdentityReducer.class);

    both Identity classes are deprecated.. So seemed the solution was to create
    mapper and reducer as follows:
    public static class NOOPMapper
    extends Mapper<Text, IntWritable, Text, IntWritable>{


    public void map(Text key, IntWritable value, Context context
    ) throws IOException, InterruptedException {

    context.write(key, value);

    }
    }

    public static class NOOPReducer
    extends Reducer<Text,IntWritable,Text,IntWritable> {
    private IntWritable result = new IntWritable();

    public void reduce(Text key, Iterable<IntWritable> values,
    Context context
    ) throws IOException, InterruptedException {
    context.write(key, result);
    }
    }


    And then with code:
    Configuration conf = new Configuration();
    Job job = new Job(conf, "testdriver");

    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);

    job.setInputFormatClass(TextInputFormat.class);
    job.setOutputFormatClass(TextOutputFormat.class);

    FileInputFormat.addInputPath(job, new Path("In"));
    FileOutputFormat.setOutputPath(job, new Path("Out"));

    job.setMapperClass(NOOPMapper.class);
    job.setReducerClass(NOOPReducer.class);

    job.waitForCompletion(true);


    However I get this message
    java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be
    cast to org.apache.hadoop.io.Text
    at TestDriver$NOOPMapper.map(TestDriver.java:1)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
    11/08/01 16:41:01 INFO mapred.JobClient: map 0% reduce 0%
    11/08/01 16:41:01 INFO mapred.JobClient: Job complete: job_local_0001
    11/08/01 16:41:01 INFO mapred.JobClient: Counters: 0



    Can anyone tell me what I need for this to work.

    Attached is full code..
    http://old.nabble.com/file/p32174859/TestDriver.java TestDriver.java
    --
    View this message in context: http://old.nabble.com/maprd-vs-mapreduce-api-tp32174859p32174859.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
  • Mohit Anchlia at Aug 5, 2011 at 11:26 pm
    On Fri, Aug 5, 2011 at 3:42 PM, Stevens, Keith D. wrote:
    The Mapper and Reducer class in org.apache.hadoop.mapreduce implement the identity function.  So you should be able to just do

    conf.setMapperClass(org.apache.hadoop.mapreduce.Mapper.class);
    conf.setReducerClass(org.apache.hadoop.mapreduce.Reducer.class);

    without having to implement your own no-op classes.

    I recommend reading the javadoc for differences between the old api and the new api, for example http://hadoop.apache.org/common/docs/r0.20.2/api/index.html indicates the different functionality of Mapper in the new api and it's dual use as the identity mapper.
    Sorry for asking on this thread :) Does Definitive Guide 2 cover the new api?
    Cheers,
    --Keith
    On Aug 5, 2011, at 1:15 PM, garpinc wrote:


    I was following this tutorial on version 0.19.1

    http://v-lad.org/Tutorials/Hadoop/23%20-%20create%20the%20project.html

    I however wanted to use the latest version of api 0.20.2

    The original code in tutorial had following lines
    conf.setMapperClass(org.apache.hadoop.mapred.lib.IdentityMapper.class);
    conf.setReducerClass(org.apache.hadoop.mapred.lib.IdentityReducer.class);

    both Identity classes are deprecated.. So seemed the solution was to create
    mapper and reducer as follows:
    public static class NOOPMapper
    extends Mapper<Text, IntWritable, Text, IntWritable>{


    public void map(Text key, IntWritable value, Context context
    ) throws IOException, InterruptedException {

    context.write(key, value);

    }
    }

    public static class NOOPReducer
    extends Reducer<Text,IntWritable,Text,IntWritable> {
    private IntWritable result = new IntWritable();

    public void reduce(Text key, Iterable<IntWritable> values,
    Context context
    ) throws IOException, InterruptedException {
    context.write(key, result);
    }
    }


    And then with code:
    Configuration conf = new Configuration();
    Job job = new Job(conf, "testdriver");

    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);

    job.setInputFormatClass(TextInputFormat.class);
    job.setOutputFormatClass(TextOutputFormat.class);

    FileInputFormat.addInputPath(job, new Path("In"));
    FileOutputFormat.setOutputPath(job, new Path("Out"));

    job.setMapperClass(NOOPMapper.class);
    job.setReducerClass(NOOPReducer.class);

    job.waitForCompletion(true);


    However I get this message
    java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be
    cast to org.apache.hadoop.io.Text
    at TestDriver$NOOPMapper.map(TestDriver.java:1)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
    11/08/01 16:41:01 INFO mapred.JobClient:  map 0% reduce 0%
    11/08/01 16:41:01 INFO mapred.JobClient: Job complete: job_local_0001
    11/08/01 16:41:01 INFO mapred.JobClient: Counters: 0



    Can anyone tell me what I need for this to work.

    Attached is full code..
    http://old.nabble.com/file/p32174859/TestDriver.java TestDriver.java
    --
    View this message in context: http://old.nabble.com/maprd-vs-mapreduce-api-tp32174859p32174859.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedAug 2, '11 at 3:41a
activeAug 5, '11 at 11:26p
posts4
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase