FAQ
Can someone explain to me what this error means?

First error while processing:
hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
at offset: 269585170
Format error in record or block header at end of file.
First error while processing:
hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000012_0.avro
at offset: 364234195
Backend 5:Format error in record or block header at end of file.
First error while processing:
hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
at offset: 269585170
Backend 5:Format error in record or block header at end of file.
First error while processing:
hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
at offset: 269585170
Format error in record or block header at end of file.
First error while processing:
hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000012_0.avro
at offset: 364234195
Backend 5:Format error in record or block header at end of file.
First error while processing:
hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
at offset: 269585170
Backend 5:Format error in record or block header at end of file.
First error while processing:
hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
at offset: 269585170
Format error in record or block header at end of file.
First error while processing:
hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000012_0.avro
at offset: 364234195
Backend 5:Format error in record or block header at end of file.
First error while processing:
hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
at offset: 269585170

Thanks,
Ben

Search Discussions

  • Skye Wanderman-Milne at Apr 17, 2013 at 7:47 pm
    These messages indicate that Impala ran into a problem parsing the Avro
    files. There should be more detailed information in the log -- if you send
    me your log files, or one of your Avro data files, I can take a look.

    Thanks,
    Skye

    On Wed, Apr 17, 2013 at 11:57 AM, Benjamin Kim wrote:

    Can someone explain to me what this error means?

    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
    at offset: 269585170
    Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000012_0.avro
    at offset: 364234195
    Backend 5:Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
    at offset: 269585170
    Backend 5:Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
    at offset: 269585170
    Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000012_0.avro
    at offset: 364234195
    Backend 5:Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
    at offset: 269585170
    Backend 5:Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
    at offset: 269585170
    Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000012_0.avro
    at offset: 364234195
    Backend 5:Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
    at offset: 269585170

    Thanks,
    Ben
  • Benjamin Kim at Apr 17, 2013 at 8:56 pm
    Hi Skye,

    The avro file is 270MB compressed. It's the one listed below in the error
    messages.

    So...

    Where would I find the impala log?

    Thanks,
    Ben
    On Wednesday, April 17, 2013 12:47:09 PM UTC-7, Skye Wanderman-Milne wrote:

    These messages indicate that Impala ran into a problem parsing the Avro
    files. There should be more detailed information in the log -- if you send
    me your log files, or one of your Avro data files, I can take a look.

    Thanks,
    Skye


    On Wed, Apr 17, 2013 at 11:57 AM, Benjamin Kim <bbui...@gmail.com<javascript:>
    wrote:
    Can someone explain to me what this error means?

    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
    at offset: 269585170
    Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000012_0.avro
    at offset: 364234195
    Backend 5:Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
    at offset: 269585170
    Backend 5:Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
    at offset: 269585170
    Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000012_0.avro
    at offset: 364234195
    Backend 5:Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
    at offset: 269585170
    Backend 5:Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
    at offset: 269585170
    Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000012_0.avro
    at offset: 364234195
    Backend 5:Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
    at offset: 269585170

    Thanks,
    Ben
  • Ishaan Joshi at Apr 17, 2013 at 9:21 pm
    Benjamin,

    You should be able to find the logs on the debug webpage. The address is:
    http://<hostname_of_impalad_you_connected_to>:25000
    There is a link to go look at the logs. Alternatively, if your cluster is
    CM managed, you can look into /var/log/impalad/impalad.INFO for the latest
    logs.

    Thanks,

    -- Ishaan

    On Wed, Apr 17, 2013 at 1:56 PM, Benjamin Kim wrote:

    Hi Skye,

    The avro file is 270MB compressed. It's the one listed below in the error
    messages.

    So...

    Where would I find the impala log?

    Thanks,
    Ben

    On Wednesday, April 17, 2013 12:47:09 PM UTC-7, Skye Wanderman-Milne wrote:

    These messages indicate that Impala ran into a problem parsing the Avro
    files. There should be more detailed information in the log -- if you send
    me your log files, or one of your Avro data files, I can take a look.

    Thanks,
    Skye

    On Wed, Apr 17, 2013 at 11:57 AM, Benjamin Kim wrote:

    Can someone explain to me what this error means?

    First error while processing: hdfs://nameservice1/amg/adr/**
    bi/event/events/1.0.0/ymdh/**2013/03/31/00/1364828298062-**
    965be4009adc11e2ade9782bcb2198**c4/000020_0.avro at offset: 269585170
    Format error in record or block header at end of file.
    First error while processing: hdfs://nameservice1/amg/adr/**
    bi/event/events/1.0.0/ymdh/**2013/03/31/00/1364828298062-**
    965be4009adc11e2ade9782bcb2198**c4/000012_0.avro at offset: 364234195
    Backend 5:Format error in record or block header at end of file.
    First error while processing: hdfs://nameservice1/amg/adr/**
    bi/event/events/1.0.0/ymdh/**2013/03/31/00/1364828298062-**
    965be4009adc11e2ade9782bcb2198**c4/000020_0.avro at offset: 269585170
    Backend 5:Format error in record or block header at end of file.
    First error while processing: hdfs://nameservice1/amg/adr/**
    bi/event/events/1.0.0/ymdh/**2013/03/31/00/1364828298062-**
    965be4009adc11e2ade9782bcb2198**c4/000020_0.avro at offset: 269585170
    Format error in record or block header at end of file.
    First error while processing: hdfs://nameservice1/amg/adr/**
    bi/event/events/1.0.0/ymdh/**2013/03/31/00/1364828298062-**
    965be4009adc11e2ade9782bcb2198**c4/000012_0.avro at offset: 364234195
    Backend 5:Format error in record or block header at end of file.
    First error while processing: hdfs://nameservice1/amg/adr/**
    bi/event/events/1.0.0/ymdh/**2013/03/31/00/1364828298062-**
    965be4009adc11e2ade9782bcb2198**c4/000020_0.avro at offset: 269585170
    Backend 5:Format error in record or block header at end of file.
    First error while processing: hdfs://nameservice1/amg/adr/**
    bi/event/events/1.0.0/ymdh/**2013/03/31/00/1364828298062-**
    965be4009adc11e2ade9782bcb2198**c4/000020_0.avro at offset: 269585170
    Format error in record or block header at end of file.
    First error while processing: hdfs://nameservice1/amg/adr/**
    bi/event/events/1.0.0/ymdh/**2013/03/31/00/1364828298062-**
    965be4009adc11e2ade9782bcb2198**c4/000012_0.avro at offset: 364234195
    Backend 5:Format error in record or block header at end of file.
    First error while processing: hdfs://nameservice1/amg/adr/**
    bi/event/events/1.0.0/ymdh/**2013/03/31/00/1364828298062-**
    965be4009adc11e2ade9782bcb2198**c4/000020_0.avro at offset: 269585170

    Thanks,
    Ben
  • Benjamin Kim at Apr 17, 2013 at 9:45 pm
    Skye,

    I attached the log file.

    Hope this helps.

    Thanks,
    Ben
    On Wednesday, April 17, 2013 12:47:09 PM UTC-7, Skye Wanderman-Milne wrote:

    These messages indicate that Impala ran into a problem parsing the Avro
    files. There should be more detailed information in the log -- if you send
    me your log files, or one of your Avro data files, I can take a look.

    Thanks,
    Skye


    On Wed, Apr 17, 2013 at 11:57 AM, Benjamin Kim <bbui...@gmail.com<javascript:>
    wrote:
    Can someone explain to me what this error means?

    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
    at offset: 269585170
    Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000012_0.avro
    at offset: 364234195
    Backend 5:Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
    at offset: 269585170
    Backend 5:Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
    at offset: 269585170
    Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000012_0.avro
    at offset: 364234195
    Backend 5:Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
    at offset: 269585170
    Backend 5:Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
    at offset: 269585170
    Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000012_0.avro
    at offset: 364234195
    Backend 5:Format error in record or block header at end of file.
    First error while processing:
    hdfs://nameservice1/amg/adr/bi/event/events/1.0.0/ymdh/2013/03/31/00/1364828298062-965be4009adc11e2ade9782bcb2198c4/000020_0.avro
    at offset: 269585170

    Thanks,
    Ben
  • Benjamin Kim at Apr 19, 2013 at 4:04 pm
    Hi Skye,

    Here it is.

    Thanks,
    Ben
    On Thursday, April 18, 2013 6:48:53 PM UTC-7, Skye Wanderman-Milne wrote:

    Hey Ben, I took a look at your log but didn't see anything that clearly
    corresponded to the errors you're getting. Could you try restarting Impala
    (in order to get a fresh log) and running just the problematic query, and
    send me that log? It will be easier to locate the problem without noise
    from other queries.

    Thanks,
    Skye


    On Wed, Apr 17, 2013 at 2:45 PM, Benjamin Kim <bbui...@gmail.com<javascript:>
    wrote:
    Skye,

    I attached the log file.

    Hope this helps.

    Thanks,
    Ben

    On Wednesday, April 17, 2013 12:47:09 PM UTC-7, Skye Wanderman-Milne
    wrote:
    These messages indicate that Impala ran into a problem parsing the Avro
    files. There should be more detailed information in the log -- if you send
    me your log files, or one of your Avro data files, I can take a look.

    Thanks,
    Skye

    On Wed, Apr 17, 2013 at 11:57 AM, Benjamin Kim wrote:

    Can someone explain to me what this error means?

    First error while processing: hdfs://nameservice1/amg/adr/**
    bi/event/events/1.0.0/ymdh/**2013/03/31/00/1364828298062-**
    965be4009adc11e2ade9782bcb2198**c4/000020_0.avro at offset: 269585170
    Format error in record or block header at end of file.
    First error while processing: hdfs://nameservice1/amg/adr/**
    bi/event/events/1.0.0/ymdh/**2013/03/31/00/1364828298062-**
    965be4009adc11e2ade9782bcb2198**c4/000012_0.avro at offset: 364234195
    Backend 5:Format error in record or block header at end of file.
    First error while processing: hdfs://nameservice1/amg/adr/**
    bi/event/events/1.0.0/ymdh/**2013/03/31/00/1364828298062-**
    965be4009adc11e2ade9782bcb2198**c4/000020_0.avro at offset: 269585170
    Backend 5:Format error in record or block header at end of file.
    First error while processing: hdfs://nameservice1/amg/adr/**
    bi/event/events/1.0.0/ymdh/**2013/03/31/00/1364828298062-**
    965be4009adc11e2ade9782bcb2198**c4/000020_0.avro at offset: 269585170
    Format error in record or block header at end of file.
    First error while processing: hdfs://nameservice1/amg/adr/**
    bi/event/events/1.0.0/ymdh/**2013/03/31/00/1364828298062-**
    965be4009adc11e2ade9782bcb2198**c4/000012_0.avro at offset: 364234195
    Backend 5:Format error in record or block header at end of file.
    First error while processing: hdfs://nameservice1/amg/adr/**
    bi/event/events/1.0.0/ymdh/**2013/03/31/00/1364828298062-**
    965be4009adc11e2ade9782bcb2198**c4/000020_0.avro at offset: 269585170
    Backend 5:Format error in record or block header at end of file.
    First error while processing: hdfs://nameservice1/amg/adr/**
    bi/event/events/1.0.0/ymdh/**2013/03/31/00/1364828298062-**
    965be4009adc11e2ade9782bcb2198**c4/000020_0.avro at offset: 269585170
    Format error in record or block header at end of file.
    First error while processing: hdfs://nameservice1/amg/adr/**
    bi/event/events/1.0.0/ymdh/**2013/03/31/00/1364828298062-**
    965be4009adc11e2ade9782bcb2198**c4/000012_0.avro at offset: 364234195
    Backend 5:Format error in record or block header at end of file.
    First error while processing: hdfs://nameservice1/amg/adr/**
    bi/event/events/1.0.0/ymdh/**2013/03/31/00/1364828298062-**
    965be4009adc11e2ade9782bcb2198**c4/000020_0.avro at offset: 269585170

    Thanks,
    Ben

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupimpala-user @
categorieshadoop
postedApr 17, '13 at 6:57p
activeApr 19, '13 at 4:04p
posts6
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase