Hello everyone,

I want to find out how to get an array as output in mapreduce.

I just simply modified the reduce method of wordcount example as shown below:

public static class IntSumReducer extends
Reducer<Text, IntWritable, Text, ArrayWritable> {
private IntWritable[] iw = new IntWritable[2];
private ArrayWritable result = new ArrayWritable

public void reduce(Text key, Iterable<IntWritable> values,
Context context) throws IOException,
InterruptedException {
iw[0] = new IntWritable(); // initialize
iw[1] = new IntWritable(); // initialize
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
context.write(key, result);

Map outputs <Text ,ArrayWritable>.

Reduce takes <Text,Iterable<IntWritable>> as inputs.

The moment I'm trying to use the ArrayWritable in reduce to store the results
I get the following error :

11/03/08 17:03:07 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
11/03/08 17:03:07 WARN mapred.JobClient: No job jar file set. User classes
may not be found. See JobConf(Class) or JobConf#setJar(String).
11/03/08 17:03:07 INFO input.FileInputFormat: Total input paths to process : 1
11/03/08 17:03:07 INFO mapred.JobClient: Running job: job_local_0001
11/03/08 17:03:07 INFO input.FileInputFormat: Total input paths to process : 1
11/03/08 17:03:07 INFO mapred.MapTask: io.sort.mb = 100
11/03/08 17:03:07 INFO mapred.MapTask: data buffer = 79691776/99614720
11/03/08 17:03:07 INFO mapred.MapTask: record buffer = 262144/327680
11/03/08 17:03:08 INFO mapred.MapTask: Starting flush of map output
11/03/08 17:03:08 WARN mapred.LocalJobRunner: job_local_0001
java.io.IOException: wrong value class: class
org.apache.hadoop.io.ArrayWritable is not class
at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:167)
at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write
at org.apache.hadoop.examples.WordCount$IntSumReducer.reduce
at org.apache.hadoop.examples.WordCount$IntSumReducer.reduce
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:623)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run
11/03/08 17:03:08 INFO mapred.JobClient: map 0% reduce 0%
11/03/08 17:03:08 INFO mapred.JobClient: Job complete: job_local_0001
11/03/08 17:03:08 INFO mapred.JobClient: Counters: 0

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
postedMar 9, '11 at 1:50a
activeMar 9, '11 at 1:50a

1 user in discussion

Ran Wei: 1 post



site design / logo © 2022 Grokbase