FAQ
Hi,
I am new to Hadoop so this might be an easy question for someone to
help me with.

I continually am getting this exception (my code follows below)

java.io.IOException: Type mismatch in key from map: expected
org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
at org.apache.hadoop.mapred.MapTask
$MapOutputBuffer.collect(MapTask.java:807)
at org.apache.hadoop.mapred.MapTask
$NewOutputCollector.write(MapTask.java:504)
at
org
.apache
.hadoop
.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)


Running 0.20.1

I have a text file with lines of data separated by carriage returns.
This is properly stored in a directory within HDFS. I only have a
Mapping task for processing this file, after the mapping is done it
should go straight to output, No reduce or combiner functions.

I am just trying to test to see if this will run. The mapper just
takes the data sent to it and adds it back to the collector as 2 text
values.

--------------------
My Mapper:
--------------------

public class MyTestMapper extends Mapper<LongWritable,Text,Text,Text> {

public void map(Object key, Text value, Context context) throws
IOException, InterruptedException {
String key = key.toString();
String line = value.toString();

String id = extractId(line);
String reformattedLine = reformatLine(line);

context.write(new Text(id), new Text(reformattedLine));
}
}

--------------------
My job submission code:
-------------------------------------

Job job = new Job(conf);
job.setJarByClass(MyTestMapper.class);
job.setMapperClass(MyTestMapper.class);

FileInputFormat.addInputPath(job, new Path("/myDir/sample.txt"));
FileOutputFormat.setOutputPath(job, new Path("/myDir/output/
results-"+System.currentTimeMillis()+".txt"));

job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);

job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);

job.submit();

Search Discussions

  • Ahad Rana at Oct 16, 2009 at 12:40 am
    Hi,
    Unfortunately Mapper is now a class and from your call stack it seems that
    Mapper's default map implementation is being called (instead of the one you
    defined in your class), which is passing the LongWritable key to the
    collector. You should use @Override to have the compiler help you figure out
    why your map function doesn't have the same signature as the one defined in
    the base class.

    Best of luck.

    Ahad.
    On Thu, Oct 15, 2009 at 10:29 AM, yz5od2 wrote:

    Hi,
    I am new to Hadoop so this might be an easy question for someone to help me
    with.

    I continually am getting this exception (my code follows below)

    java.io.IOException: Type mismatch in key from map: expected
    org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
    at
    org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:807)
    at
    org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:504)
    at
    org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
    at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)


    Running 0.20.1

    I have a text file with lines of data separated by carriage returns. This
    is properly stored in a directory within HDFS. I only have a Mapping task
    for processing this file, after the mapping is done it should go straight to
    output, No reduce or combiner functions.

    I am just trying to test to see if this will run. The mapper just takes the
    data sent to it and adds it back to the collector as 2 text values.

    --------------------
    My Mapper:
    --------------------

    public class MyTestMapper extends Mapper<LongWritable,Text,Text,Text> {

    public void map(Object key, Text value, Context context) throws
    IOException, InterruptedException {
    String key = key.toString();
    String line = value.toString();

    String id = extractId(line);
    String reformattedLine = reformatLine(line);

    context.write(new Text(id), new Text(reformattedLine));
    }
    }

    --------------------
    My job submission code:
    -------------------------------------

    Job job = new Job(conf);
    job.setJarByClass(MyTestMapper.class);
    job.setMapperClass(MyTestMapper.class);

    FileInputFormat.addInputPath(job, new Path("/myDir/sample.txt"));
    FileOutputFormat.setOutputPath(job, new
    Path("/myDir/output/results-"+System.currentTimeMillis()+".txt"));

    job.setMapOutputKeyClass(Text.class);
    job.setMapOutputValueClass(Text.class);

    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(Text.class);

    job.submit();
  • .ke. sivakumar at Oct 16, 2009 at 7:54 am
    if u use any of the java IDE's, try clearing all the warnings in the code,
    @override will be added automatically.
  • Aaron at Oct 17, 2009 at 8:21 pm
    Yep that was it. I changed the method signature to LongWriteable for
    the 1st parameter (also annotated it w/ Override to see the compile
    error)

    thanks for your help!

    On Oct 15, 2009, at 6:40 PM, Ahad Rana wrote:

    Hi,
    Unfortunately Mapper is now a class and from your call stack it
    seems that
    Mapper's default map implementation is being called (instead of the
    one you
    defined in your class), which is passing the LongWritable key to the
    collector. You should use @Override to have the compiler help you
    figure out
    why your map function doesn't have the same signature as the one
    defined in
    the base class.

    Best of luck.

    Ahad.

    On Thu, Oct 15, 2009 at 10:29 AM, yz5od2 <woods5242-
    outdoors@yahoo.com>wrote:
    Hi,
    I am new to Hadoop so this might be an easy question for someone to
    help me
    with.

    I continually am getting this exception (my code follows below)

    java.io.IOException: Type mismatch in key from map: expected
    org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
    at
    org.apache.hadoop.mapred.MapTask
    $MapOutputBuffer.collect(MapTask.java:807)
    at
    org.apache.hadoop.mapred.MapTask
    $NewOutputCollector.write(MapTask.java:504)
    at
    org
    .apache
    .hadoop
    .mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:
    80)
    at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:
    583)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)


    Running 0.20.1

    I have a text file with lines of data separated by carriage
    returns. This
    is properly stored in a directory within HDFS. I only have a
    Mapping task
    for processing this file, after the mapping is done it should go
    straight to
    output, No reduce or combiner functions.

    I am just trying to test to see if this will run. The mapper just
    takes the
    data sent to it and adds it back to the collector as 2 text values.

    --------------------
    My Mapper:
    --------------------

    public class MyTestMapper extends
    Mapper<LongWritable,Text,Text,Text> {

    public void map(Object key, Text value, Context context) throws
    IOException, InterruptedException {
    String key = key.toString();
    String line = value.toString();

    String id = extractId(line);
    String reformattedLine = reformatLine(line);

    context.write(new Text(id), new Text(reformattedLine));
    }
    }

    --------------------
    My job submission code:
    -------------------------------------

    Job job = new Job(conf);
    job.setJarByClass(MyTestMapper.class);
    job.setMapperClass(MyTestMapper.class);

    FileInputFormat.addInputPath(job, new Path("/myDir/sample.txt"));
    FileOutputFormat.setOutputPath(job, new
    Path("/myDir/output/results-"+System.currentTimeMillis()+".txt"));

    job.setMapOutputKeyClass(Text.class);
    job.setMapOutputValueClass(Text.class);

    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(Text.class);

    job.submit();
  • Amandeep Khurana at Oct 16, 2009 at 12:45 am
    Comments inline
    On Thu, Oct 15, 2009 at 10:29 AM, yz5od2 wrote:

    Hi,
    I am new to Hadoop so this might be an easy question for someone to help me
    with.

    I continually am getting this exception (my code follows below)

    java.io.IOException: Type mismatch in key from map: expected
    org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
    at
    org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:807)
    at
    org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:504)
    at
    org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
    at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)


    Running 0.20.1

    I have a text file with lines of data separated by carriage returns. This
    is properly stored in a directory within HDFS. I only have a Mapping task
    for processing this file, after the mapping is done it should go straight to
    output, No reduce or combiner functions.

    I am just trying to test to see if this will run. The mapper just takes the
    data sent to it and adds it back to the collector as 2 text values.

    --------------------
    My Mapper:
    --------------------

    public class MyTestMapper extends Mapper<LongWritable,Text,Text,Text> {
    public void map(Object key, Text value, Context context) throws
    IOException, InterruptedException {
    Change the Object to Text and try again..

    String key = key.toString();
    String line = value.toString();

    String id = extractId(line);
    String reformattedLine = reformatLine(line);

    context.write(new Text(id), new Text(reformattedLine));
    }
    }

    --------------------
    My job submission code:
    -------------------------------------

    Job job = new Job(conf);
    job.setJarByClass(MyTestMapper.class);
    job.setMapperClass(MyTestMapper.class);

    FileInputFormat.addInputPath(job, new Path("/myDir/sample.txt"));
    FileOutputFormat.setOutputPath(job, new
    Path("/myDir/output/results-"+System.currentTimeMillis()+".txt"));

    job.setMapOutputKeyClass(Text.class);
    job.setMapOutputValueClass(Text.class);

    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(Text.class);

    job.submit();

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedOct 15, '09 at 7:50p
activeOct 17, '09 at 8:21p
posts5
users5
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase