FAQ
Hello,

I want to output a class which I have written as the value of the map phase.
The obvious was is to implement the Writable interface but the problem is
the class has other classes as its member properties. The DataInput and
DataOutput interfaces used by the read and write methods of the Writable
class do not support object serialization. Is there any other way I can
achieve this?

Thanks,
Farhan

Search Discussions

  • Ted Yu at Apr 27, 2010 at 6:20 pm
    Take a look at the sample given in Javadoc of Writable.java
    You need to serialize your data yourself:
    @Override
    public void readFields(DataInput in) throws IOException {
    h = Text.readString(in);
    sc = in.readFloat();
    ran = in.readInt ();
    }


    On Tue, Apr 27, 2010 at 10:53 AM, Farhan Husain
    wrote:
    Hello,

    I want to output a class which I have written as the value of the map
    phase.
    The obvious was is to implement the Writable interface but the problem is
    the class has other classes as its member properties. The DataInput and
    DataOutput interfaces used by the read and write methods of the Writable
    class do not support object serialization. Is there any other way I can
    achieve this?

    Thanks,
    Farhan
  • Farhan Husain at Apr 27, 2010 at 10:01 pm
    Can I use the Serializable interface? Alternatively, is there any way to
    specify OutputFormatter for mappers like we can do for reducers?

    Thanks,
    Farhan
    On Tue, Apr 27, 2010 at 1:19 PM, Ted Yu wrote:

    Take a look at the sample given in Javadoc of Writable.java
    You need to serialize your data yourself:
    @Override
    public void readFields(DataInput in) throws IOException {
    h = Text.readString(in);
    sc = in.readFloat();
    ran = in.readInt ();
    }


    On Tue, Apr 27, 2010 at 10:53 AM, Farhan Husain
    wrote:
    Hello,

    I want to output a class which I have written as the value of the map
    phase.
    The obvious was is to implement the Writable interface but the problem is
    the class has other classes as its member properties. The DataInput and
    DataOutput interfaces used by the read and write methods of the Writable
    class do not support object serialization. Is there any other way I can
    achieve this?

    Thanks,
    Farhan
  • Farhan Husain at Apr 27, 2010 at 10:56 pm
    I tried to use a class which implements the Serializable interface and got
    the following error:

    java.lang.NullPointerException
    at
    org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73)
    at
    org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:487)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:575)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)

    On Tue, Apr 27, 2010 at 12:53 PM, Farhan Husain
    wrote:
    Hello,

    I want to output a class which I have written as the value of the map
    phase. The obvious was is to implement the Writable interface but the
    problem is the class has other classes as its member properties. The
    DataInput and DataOutput interfaces used by the read and write methods of
    the Writable class do not support object serialization. Is there any other
    way I can achieve this?

    Thanks,
    Farhan
  • Ted Yu at Apr 28, 2010 at 1:13 am
    Can you try adding 'org.apache.hadoop.io.serializer.JavaSerialization,' to
    the following config ?
    "C:\hadoop-0.20.2\src\core\core-default.xml"(87,9):
    <name>io.serializations</name>

    By default, only org.apache.hadoop.io.serializer.WritableSerialization is
    included.
    On Tue, Apr 27, 2010 at 3:55 PM, Farhan Husain wrote:

    I tried to use a class which implements the Serializable interface and got
    the following error:

    java.lang.NullPointerException
    at

    org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73)
    at
    org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:759)
    at

    org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:487)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:575)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)

    On Tue, Apr 27, 2010 at 12:53 PM, Farhan Husain
    wrote:
    Hello,

    I want to output a class which I have written as the value of the map
    phase. The obvious was is to implement the Writable interface but the
    problem is the class has other classes as its member properties. The
    DataInput and DataOutput interfaces used by the read and write methods of
    the Writable class do not support object serialization. Is there any other
    way I can achieve this?

    Thanks,
    Farhan

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedApr 27, '10 at 5:54p
activeApr 28, '10 at 1:13a
posts5
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Farhan Husain: 3 posts Ted Yu: 2 posts

People

Translate

site design / logo © 2022 Grokbase