FAQ
Is there any way to check whether
https://issues.cloudera.org/browse/IMPALA-177 is fixed in 1.0.1?

I'm getting an LZO error as described in it for a table that contains a
file larger than 2GB... Or have I stumbled upon a different bug?

Thanks!

Grisha

Search Discussions

  • Nong Li at Jun 11, 2013 at 4:40 am
    Can you share the exact error you are seeing?

    On Mon, Jun 10, 2013 at 9:13 PM, Grisha Trubetskoy wrote:


    Is there any way to check whether
    https://issues.cloudera.org/browse/IMPALA-177 is fixed in 1.0.1?

    I'm getting an LZO error as described in it for a table that contains a
    file larger than 2GB... Or have I stumbled upon a different bug?

    Thanks!

    Grisha
  • Nong Li at Jun 11, 2013 at 9:20 pm
    This looks like a different issue. Can you attach the .index file for one
    of the lzo files?

    On Mon, Jun 10, 2013 at 9:49 PM, Gregory Trubetskoy wrote:

    This is slightly modified to obscure our internal details:

    Backend 4:No block index for hdfs://<snip>/part-m-00000.lzo after offset:
    2684354560
    No block index for hdfs://i<snip>/part-m-00000.lzo after offset: 2684354560
    Backend 5:No block index for hdfs://<snip>/part-m-00000.lzo after offset:
    1879048192
    No block index for hdfs://<snip>/part-m-00000.lzo after offset: 1879048192
    Backend 6:No block index for hdfs://<snip>/part-m-00000.lzo after offset:
    2147483648
    No block index for hdfs://<snip>/part-m-00000.lzo after offset: 2147483648

    The query is a simple select count(*). It works in Hive. Deleting the
    *.index files fixes the Impala problem. The size of the part-m-00000.lzo is
    2882773633 bytes, 2GB appears to be what triggers it.


    On Tue, Jun 11, 2013 at 12:40 AM, Nong Li wrote:

    Can you share the exact error you are seeing?


    On Mon, Jun 10, 2013 at 9:13 PM, Grisha Trubetskoy <
    grisha.trubetskoy@hungrymachine.com> wrote:
    Is there any way to check whether
    https://issues.cloudera.org/browse/IMPALA-177 is fixed in 1.0.1?

    I'm getting an LZO error as described in it for a table that contains a
    file larger than 2GB... Or have I stumbled upon a different bug?

    Thanks!

    Grisha
  • Gregory Trubetskoy at Jun 11, 2013 at 10:54 pm
    Just sent it directly to your email, thx!

    On Tue, Jun 11, 2013 at 5:20 PM, Nong Li wrote:

    This looks like a different issue. Can you attach the .index file for one
    of the lzo files?


    On Mon, Jun 10, 2013 at 9:49 PM, Gregory Trubetskoy <
    grisha.trubetskoy@livingsocial.com> wrote:
    This is slightly modified to obscure our internal details:

    Backend 4:No block index for hdfs://<snip>/part-m-00000.lzo after offset:
    2684354560
    No block index for hdfs://i<snip>/part-m-00000.lzo after offset:
    2684354560
    Backend 5:No block index for hdfs://<snip>/part-m-00000.lzo after offset:
    1879048192
    No block index for hdfs://<snip>/part-m-00000.lzo after offset: 1879048192
    Backend 6:No block index for hdfs://<snip>/part-m-00000.lzo after offset:
    2147483648
    No block index for hdfs://<snip>/part-m-00000.lzo after offset:
    2147483648

    The query is a simple select count(*). It works in Hive. Deleting the
    *.index files fixes the Impala problem. The size of the part-m-00000.lzo is
    2882773633 bytes, 2GB appears to be what triggers it.


    On Tue, Jun 11, 2013 at 12:40 AM, Nong Li wrote:

    Can you share the exact error you are seeing?


    On Mon, Jun 10, 2013 at 9:13 PM, Grisha Trubetskoy <
    grisha.trubetskoy@hungrymachine.com> wrote:
    Is there any way to check whether
    https://issues.cloudera.org/browse/IMPALA-177 is fixed in 1.0.1?

    I'm getting an LZO error as described in it for a table that contains a
    file larger than 2GB... Or have I stumbled upon a different bug?

    Thanks!

    Grisha
  • Nong Li at Jun 11, 2013 at 10:58 pm
    Thanks. I've tracked down the issue. There was a place where we were
    running into an int overflow
    so the index file was not being read properly. This will be fixed in our
    upcoming 1.1 release.

    Nong

    On Tue, Jun 11, 2013 at 3:54 PM, Gregory Trubetskoy wrote:

    Just sent it directly to your email, thx!

    On Tue, Jun 11, 2013 at 5:20 PM, Nong Li wrote:

    This looks like a different issue. Can you attach the .index file for
    one of the lzo files?


    On Mon, Jun 10, 2013 at 9:49 PM, Gregory Trubetskoy <
    grisha.trubetskoy@livingsocial.com> wrote:
    This is slightly modified to obscure our internal details:

    Backend 4:No block index for hdfs://<snip>/part-m-00000.lzo after
    offset: 2684354560
    No block index for hdfs://i<snip>/part-m-00000.lzo after offset:
    2684354560
    Backend 5:No block index for hdfs://<snip>/part-m-00000.lzo after
    offset: 1879048192
    No block index for hdfs://<snip>/part-m-00000.lzo after offset:
    1879048192
    Backend 6:No block index for hdfs://<snip>/part-m-00000.lzo after
    offset: 2147483648
    No block index for hdfs://<snip>/part-m-00000.lzo after offset:
    2147483648

    The query is a simple select count(*). It works in Hive. Deleting the
    *.index files fixes the Impala problem. The size of the part-m-00000.lzo is
    2882773633 bytes, 2GB appears to be what triggers it.


    On Tue, Jun 11, 2013 at 12:40 AM, Nong Li wrote:

    Can you share the exact error you are seeing?


    On Mon, Jun 10, 2013 at 9:13 PM, Grisha Trubetskoy <
    grisha.trubetskoy@hungrymachine.com> wrote:
    Is there any way to check whether
    https://issues.cloudera.org/browse/IMPALA-177 is fixed in 1.0.1?

    I'm getting an LZO error as described in it for a table that contains
    a file larger than 2GB... Or have I stumbled upon a different bug?

    Thanks!

    Grisha

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupimpala-user @
categorieshadoop
postedJun 11, '13 at 4:13a
activeJun 11, '13 at 10:58p
posts5
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase