FAQ
Hi,
i have been trying to implement custom input and output formats. i
was successful enough in creating an custom output format. but whn i call a
mapreduce method which takes in a file, using custom input format, i get an
exception.
java.lang.NullPointerException
at Beans.Content.write(Content.java:54)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:613)
at CustomInputFormat.SampleCustomInputMap.map(SampleCustomInputMap.java:32)
at CustomInputFormat.SampleCustomInputMap.map(SampleCustomInputMap.java:1)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)
0 Hello World0 Hello World0 Hello World0

id:: null media:: null url:: null content::null

id:: null media:: null url:: null content::null
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
at
CustomInputFormat.SampleCustomInputMapReduce.run(SampleCustomInputMapReduce.java:53)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at
CustomInputFormat.SampleCustomInputMapReduce.main(SampleCustomInputMapReduce.java:59)


I have attached the following files
Content => Custom object that implements writercomparible
ContentInputFormatV2 => inputformat which implements SequenceFileInputFormat
ContentRecordReader => implementation of RecordReader (Not required though,
it should work w/o it i assume).
SampleCustomInputMap => mapper class
SampleCustomInputReduce => reducer class
SampleCustomInputMapReduce => class which contains the main method and the
job configurations.
data and index => my inputfiles for the main function
For the 1st record it works fine, but for next record and records after
that, i get null. where could i go wrong??
Regards,
Raakhi

Search Discussions

  • Antonio D'Ettole at Feb 2, 2010 at 8:57 am
    Rakhi,
    I've recently had to implement a custom InputFormat that's pretty basic
    (every split is a list of integers basically).
    You can check it out here http://github.com/codazzo/MultiRow
    One guy also implemented a custom InputFormat and wrote about it on his
    blog http://codedemigod.com/blog/?p=120
    Hope that helps.
    Antonio
    On Fri, Jan 29, 2010 at 2:04 PM, Rakhi Khatwani wrote:

    Hi,
    i have been trying to implement custom input and output formats. i
    was successful enough in creating an custom output format. but whn i call a
    mapreduce method which takes in a file, using custom input format, i get an
    exception.
    java.lang.NullPointerException
    at Beans.Content.write(Content.java:54)
    at
    org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
    at
    org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
    at
    org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:613)
    at
    CustomInputFormat.SampleCustomInputMap.map(SampleCustomInputMap.java:32)
    at CustomInputFormat.SampleCustomInputMap.map(SampleCustomInputMap.java:1)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at
    org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)
    0 Hello World0 Hello World0 Hello World0

    id:: null media:: null url:: null content::null

    id:: null media:: null url:: null content::null
    java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
    at
    CustomInputFormat.SampleCustomInputMapReduce.run(SampleCustomInputMapReduce.java:53)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at
    CustomInputFormat.SampleCustomInputMapReduce.main(SampleCustomInputMapReduce.java:59)


    I have attached the following files
    Content => Custom object that implements writercomparible
    ContentInputFormatV2 => inputformat which implements
    SequenceFileInputFormat
    ContentRecordReader => implementation of RecordReader (Not required though,
    it should work w/o it i assume).
    SampleCustomInputMap => mapper class
    SampleCustomInputReduce => reducer class
    SampleCustomInputMapReduce => class which contains the main method and the
    job configurations.
    data and index => my inputfiles for the main function
    For the 1st record it works fine, but for next record and records after
    that, i get null. where could i go wrong??
    Regards,
    Raakhi

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJan 29, '10 at 1:05p
activeFeb 2, '10 at 8:57a
posts2
users2
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase