Grokbase Groups Hive user May 2011
FAQ
Hi ,

I have a four column external table.
One of the col is array of struct of five fields.
I use a custom serde, to read out hadoop seq file and populate values on to hive row.
The array of struct is ArrayList of Object[5].

All queries run fine.
Except the explode query which gives a class cast exception and saying object array is being casted as string


I dont knw wht is the issue, but the other serde along same lines works super fine.

pl guide me


at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaStringObjectInspector.getPrimitiveWritableObject(JavaStringObjectInspector.java:35)
at org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:206)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:480)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:426)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:555)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.LateralViewJoinOperator.processOp(LateralViewJoinOperator.java:133)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:111)
at org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:44)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:82)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDTFExplode.process(GenericUDTFExplode.java:75)



-Sagar

Search Discussions

  • Edward Capriolo at May 13, 2011 at 1:52 pm
    You must have a mistake in your object inspector. I am doing something
    similar with nested structs in arrays. This is a struct that has a few
    top level properties, and then one property is a list of structs one
    level of nesting only.

    public ObjectInspector initialize(ObjectInspector[] arguments)
    throws UDFArgumentException {
    ret = new Object[2];
    oi=arguments;

    List<String> fieldNames = new ArrayList<String>();
    List<ObjectInspector> fieldInspectors = new ArrayList<ObjectInspector>();


    List<String> structFieldNames = new ArrayList<String>();
    List<ObjectInspector> structFieldInspectors = new
    ArrayList<ObjectInspector>();
    structFieldNames.add("id");
    structFieldNames.add("acount");
    structFieldNames.add("bcount");

    structFieldInspectors.add(PrimitiveObjectInspectorFactory.javaLongObjectInspector);
    structFieldInspectors.add(PrimitiveObjectInspectorFactory.javaLongObjectInspector);
    structFieldInspectors.add(PrimitiveObjectInspectorFactory.javaLongObjectInspector);

    fieldNames.add("redirectList");
    fieldInspectors.add(
    ObjectInspectorFactory.getStandardListObjectInspector(
    ObjectInspectorFactory.getStandardStructObjectInspector(structFieldNames,
    structFieldInspectors)
    )
    );

    //sl.getRecordCount()
    fieldNames.add("cCount");
    fieldInspectors.add(PrimitiveObjectInspectorFactory.javaLongObjectInspector);

    StructObjectInspector soi =
    ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames,
    fieldInspectors);
    return soi;

    I hope this helps.
    On Thu, May 12, 2011 at 9:45 PM, Sagar Naik wrote:
    Hi ,

    I have a four column external table.
    One of the col is array of struct of five fields.
    I use a custom serde, to read out hadoop seq file and populate values on to hive row.
    The array of struct is ArrayList of Object[5].

    All queries run fine.
    Except the explode query which gives a class cast exception and saying object array is being casted as string


    I dont knw wht is the issue, but the other serde along same lines works super fine.

    pl guide  me


    at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaStringObjectInspector.getPrimitiveWritableObject(JavaStringObjectInspector.java:35)
    at org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:206)
    at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:480)
    at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:426)
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:555)
    at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
    at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
    at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
    at org.apache.hadoop.hive.ql.exec.LateralViewJoinOperator.processOp(LateralViewJoinOperator.java:133)
    at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
    at org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:111)
    at org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:44)
    at org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:82)
    at org.apache.hadoop.hive.ql.udf.generic.GenericUDTFExplode.process(GenericUDTFExplode.java:75)



    -Sagar
  • Sagar Naik at May 13, 2011 at 9:12 pm
    Does that mean I need to override UDTFExplode ?
    In that case, every nested struct /list serde needs to implement this ?

    Correct if I m wrong



    -Sagar
    On May 13, 2011, at 6:51 AM, Edward Capriolo wrote:

    You must have a mistake in your object inspector. I am doing something
    similar with nested structs in arrays. This is a struct that has a few
    top level properties, and then one property is a list of structs one
    level of nesting only.

    public ObjectInspector initialize(ObjectInspector[] arguments)
    throws UDFArgumentException {
    ret = new Object[2];
    oi=arguments;

    List<String> fieldNames = new ArrayList<String>();
    List<ObjectInspector> fieldInspectors = new ArrayList<ObjectInspector>();


    List<String> structFieldNames = new ArrayList<String>();
    List<ObjectInspector> structFieldInspectors = new
    ArrayList<ObjectInspector>();
    structFieldNames.add("id");
    structFieldNames.add("acount");
    structFieldNames.add("bcount");

    structFieldInspectors.add(PrimitiveObjectInspectorFactory.javaLongObjectInspector);
    structFieldInspectors.add(PrimitiveObjectInspectorFactory.javaLongObjectInspector);
    structFieldInspectors.add(PrimitiveObjectInspectorFactory.javaLongObjectInspector);

    fieldNames.add("redirectList");
    fieldInspectors.add(
    ObjectInspectorFactory.getStandardListObjectInspector(
    ObjectInspectorFactory.getStandardStructObjectInspector(structFieldNames,
    structFieldInspectors)
    )
    );

    //sl.getRecordCount()
    fieldNames.add("cCount");
    fieldInspectors.add(PrimitiveObjectInspectorFactory.javaLongObjectInspector);

    StructObjectInspector soi =
    ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames,
    fieldInspectors);
    return soi;

    I hope this helps.
    On Thu, May 12, 2011 at 9:45 PM, Sagar Naik wrote:
    Hi ,

    I have a four column external table.
    One of the col is array of struct of five fields.
    I use a custom serde, to read out hadoop seq file and populate values on to hive row.
    The array of struct is ArrayList of Object[5].

    All queries run fine.
    Except the explode query which gives a class cast exception and saying object array is being casted as string


    I dont knw wht is the issue, but the other serde along same lines works super fine.

    pl guide me


    at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaStringObjectInspector.getPrimitiveWritableObject(JavaStringObjectInspector.java:35)
    at org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:206)
    at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:480)
    at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:426)
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:555)
    at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
    at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
    at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
    at org.apache.hadoop.hive.ql.exec.LateralViewJoinOperator.processOp(LateralViewJoinOperator.java:133)
    at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:457)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:697)
    at org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:111)
    at org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:44)
    at org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:82)
    at org.apache.hadoop.hive.ql.udf.generic.GenericUDTFExplode.process(GenericUDTFExplode.java:75)



    -Sagar

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedMay 13, '11 at 1:46a
activeMay 13, '11 at 9:12p
posts3
users2
websitehive.apache.org

2 users in discussion

Sagar Naik: 2 posts Edward Capriolo: 1 post

People

Translate

site design / logo © 2021 Grokbase