Impala not reading Avro data

Hi Lenni,

I tried that, but unfortunately it had no effect.

Thanks,
Tom

I don't have problem querying the avro table in my Impala 1.0 cluster. But
I'm really worrying about this problem because we are going to use avro to
store data before parquet impala support is mature.

The only problem is that I have to run the create table statement in HIVE.
Impala doesn't support ROW FORMAT serde.

Here are my avro table:

create external table t_avro (
   A string,
   B int,
   C double)
ROW FORMAT SERDE
   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
LOCATION '/user/benyiw/t_avro'
TBLPROPERTIES (
   'avro.schema.literal'='{ "type": "record",
       "name": "TestRecord",
       "version": 2,
       "fields": [
         { "name": "A", "type": ["null", "string"] },
         { "name": "B", "type": ["null", "int" ] },
         { "name": "C", "type": ["null", "double"], "default": null }
       ] }
');

[:21000] > desc t_avro
Query: describe t_avro
Query finished, fetching results ...
+------+--------+-------------------+
+------+--------+-------------------+
Returned 3 row(s) in 0.45s
[:21000] > select * from t_avro;
Query: select * from t_avro
Query finished, fetching results ...
+-------+---+----+
+-------+---+----+
Returned 1 row(s) in 0.56s
[:21000] > select count(*) from t_avro;
Query: select count(*) from t_avro
Query finished, fetching results ...
+----------+
+----------+
Returned 1 row(s) in 0.27s

Hi Tom,

Can you send the impalad log file? Or if you're able to send the data file
I'll see if I can reproduce the problem. Also, what's the Avro schema of
your table?

Thanks,
Skye

I think I found what the underlying problem is - it looks like a
limitation in the Impala Avro schema parser not being able to parse

{ "name": "A", "type":{"type":"string"} }

The more common case of

{ "name": "A", "type":"string" }

works fine. The nested case is typically used when you want to add
extra properties to the type, e.g. the Java code generator adds
"avro.java.string":"String" to say that the type is a String (rather
than Avro Java's Utf8 type).

I opened https://issues.cloudera.org/browse/IMPALA-399 - hopefully
someone can fix it soon :)

Thanks bewang for the working example - that helped me narrow down the problem.

Cheers,
Tom