FAQ
I have a custom Writable called DigestWritable. The DigestWritable is
stored in a SequenceFile.



For illustration, assume it is like this:

public class DigestWritable implements org.apache.hadoop.io.Writable {

String str;

int I;

}



I started to write a SerDe in order to read the DigestWritable from the
SequenceFile and pass the String/int fields to Hive.



However, in

public Object deserialize(Writable field) throws SerDeException



the Writable param seems to only be allowed to be Text or ByteWritable.



Is this correct, or am I doing something wrong in my SerDe?



How do I get the Writable to be a DigestWritable?



Is the Writable only allowed to be a primitive?



Thanks

John

Search Discussions

  • Rodriguez, John at Sep 15, 2010 at 6:50 pm
    To answer my own question, the Writable can be any class that implements
    Writable.



    My "create external table" statement was not correct and I was not
    getting the SequenceFile.reader which was causing my problems.



    Also, even though the key in a SequenceFile is not used, it has to be
    both Writable and WritableComparable. This is because of the method
    signature from

    src/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java

    private RecordReader<WritableComparable, Writable> getRecordReader()



    In my SerDe, I now have access to my custom Writable class and I can
    translate the values for Hive.



    This is just an experiment with fewer fields than my custom Writable has
    but here is the correct DDL.



    create external table if not exists t10 (

    str string, flagWord int

    )

    partitioned by (id int)

    row format

    serde 'com.utils.DigestSerde'

    STORED AS inputformat

    'org.apache.hadoop.mapred.SequenceFileInputFormat'

    outputformat

    'org.apache.hadoop.mapred.SequenceFileOutputFormat'

    location '/user/jrodrigu/hiveTables';



    ALTER TABLE t10 ADD PARTITION (id=1) LOCATION
    '/user/jrodrigu/hiveTables' PARTITION (id=2) LOCATION
    '/user/jrodrigu/hiveTables1';



    From: Rodriguez, John
    Sent: Tuesday, September 14, 2010 2:16 PM
    To: hive-user@hadoop.apache.org
    Subject: Is deserialize(Writable field) able to handle any writable
    class



    I have a custom Writable called DigestWritable. The DigestWritable is
    stored in a SequenceFile.



    For illustration, assume it is like this:

    public class DigestWritable implements org.apache.hadoop.io.Writable {

    String str;

    int I;

    }



    I started to write a SerDe in order to read the DigestWritable from the
    SequenceFile and pass the String/int fields to Hive.



    However, in

    public Object deserialize(Writable field) throws SerDeException



    the Writable param seems to only be allowed to be Text or ByteWritable.



    Is this correct, or am I doing something wrong in my SerDe?



    How do I get the Writable to be a DigestWritable?



    Is the Writable only allowed to be a primitive?



    Thanks

    John

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedSep 14, '10 at 7:16p
activeSep 15, '10 at 6:50p
posts2
users1
websitehive.apache.org

1 user in discussion

Rodriguez, John: 2 posts

People

Translate

site design / logo © 2022 Grokbase