FAQ
I run the query "select count(*) from hbase_sdtst_m where month=5;" on our
impala cluster, when the progress is near 100%(about 98%), the progress
will not move forward anymore and the impala shell hangs forever.

Anyone can help on this?

Search Discussions

  • Ishaan Joshi at May 20, 2013 at 5:16 pm
    Zesheng,

       To help us further investigate the problem, could you attach the impala
    logs? They should be found under /var/log/impalad/impalad.INFO.
    Additionally, can you successfully run the query in hive?

    Thanks,

    -- Ishaan

    On Mon, May 20, 2013 at 2:33 AM, Zesheng Wu wrote:

    I run the query "select count(*) from hbase_sdtst_m where month=5;" on our
    impala cluster, when the progress is near 100%(about 98%), the progress
    will not move forward anymore and the impala shell hangs forever.

    Anyone can help on this?
  • Zesheng Wu at May 21, 2013 at 3:54 am
    I find this is a bug of impalad when it deal with sequence file without
    contents, I created a jira issue
    here: https://issues.cloudera.org/browse/IMPALA-362

    在 2013年5月21日星期二UTC+8上午9时15分45秒,Zesheng Wu写道:
    1. I found that one of the impalad hangs on GetBytes(), this is the
    stackstrace:
    (gdb) bt
    #0 0x0000003627a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
    /lib64/libpthread.so.0
    #1 0x0000000000cc8ef8 in boost::condition_variable::wait (this=0x4da8fe0,
    m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:20
    #2 0x0000000000ee0d7d in impala::ScannerContext::Stream::GetBytesInternal
    (this=0x4da8fc0, requested_len=4, out_buffer=0x7fefa6e134a8, peek=false,
    out_len=0x7fefa6e1341c,
    eos=0x7fefa6e1341b) at
    /home/wuzesheng/work/cloud/infra/impala/be/src/exec/scanner-context.cc:248
    #3 0x0000000000eae7ee in impala::ScannerContext::Stream::GetBytes
    (this=0x4da8fc0, requested_len=4, buffer=0x7fefa6e134a8,
    out_len=0x7fefa6e1341c, eos=0x7fefa6e1341b,
    status=0xdccaf18) at
    /home/wuzesheng/work/cloud/infra/impala/be/src/exec/scanner-context.inline.h:57
    #4 0x0000000000eae96a in impala::ScannerContext::Stream::ReadBytes
    (this=0x4da8fc0, length=4, buf=0x7fefa6e134a8, status=0xdccaf18)
    at
    /home/wuzesheng/work/cloud/infra/impala/be/src/exec/scanner-context.inline.h:68
    #5 0x0000000000eaecd3 in impala::ScannerContext::Stream::ReadInt
    (this=0x4da8fc0, val=0x7fefa6e1364c, status=0xdccaf18)
    at
    /home/wuzesheng/work/cloud/infra/impala/be/src/exec/scanner-context.inline.h:115
    #6 0x0000000000eb413a in impala::HdfsSequenceScanner::ReadFileHeader
    (this=0xdccaea0) at
    /home/wuzesheng/work/cloud/infra/impala/be/src/exec/hdfs-sequence-scanner.cc:460
    #7 0x0000000000ef55d8 in impala::BaseSequenceScanner::ProcessSplit
    (this=0xdccaea0, context=0x4e1b290)
    at
    /home/wuzesheng/work/cloud/infra/impala/be/src/exec/base-sequence-scanner.cc:92
    #8 0x0000000000e8ed8f in impala::HdfsScanNode::ScannerThread
    (this=0xd818700, scanner=0xdccaea0, context=0x4e1b290)
    at
    /home/wuzesheng/work/cloud/infra/impala/be/src/exec/hdfs-scan-node.cc:795
    #9 0x0000000000ea1804 in boost::_mfi::mf2<void, impala::HdfsScanNode,
    impala::HdfsScanner*, impala::ScannerContext*>::operator() (this=0x4d51730,
    p=0xd818700, a1=0xdccaea0,
    a2=0x4e1b290) at /usr/include/boost/bind/mem_fn_template.hpp:280
    #10 0x0000000000ea16ca in
    boost::_bi::list3<boost::_bi::value<impala::HdfsScanNode*>,
    boost::_bi::value<impala::HdfsScanner*>,
    boost::_bi::value<impala::ScannerContext*>
    ::operator()<boost::_mfi::mf2<void, impala::HdfsScanNode,
    impala::HdfsScanner*, impala::ScannerContext*>, boost::_bi::list0>
    (this=0x4d51740, f=..., a=...)
    at /usr/include/boost/bind/bind.hpp:392
    #11 0x0000000000ea15f1 in boost::_bi::bind_t<void, boost::_mfi::mf2<void,
    impala::HdfsScanNode, impala::HdfsScanner*, impala::ScannerContext*>,
    boost::_bi::list3<boost::_bi::value<impala::HdfsScanNode*>,
    boost::_bi::value<impala::HdfsScanner*>,
    boost::_bi::value<impala::ScannerContext*> > >::operator() (this=0x4d51730)
    at /usr/include/boost/bind/bind_template.hpp:20
    #12 0x0000000000ea11fe in
    boost::detail::thread_data<boost::_bi::bind_t<void, boost::_mfi::mf2<void,
    impala::HdfsScanNode, impala::HdfsScanner*, impala::ScannerContext*>,
    boost::_bi::list3<boost::_bi::value<impala::HdfsScanNode*>,
    boost::_bi::value<impala::HdfsScanner*>,
    boost::_bi::value<impala::ScannerContext*> > > >::run (this=0x4d51600)
    at /usr/include/boost/thread/detail/thread.hpp:56
    #13 0x00007fefdbd253fd in boost::(anonymous namespace)::thread_proxy
    (param=0x4d51600) at libs/thread/src/pthread/thread.cpp:120
    #14 0x0000003627a07851 in start_thread () from /lib64/libpthread.so.0
    #15 0x00000039a16e811d in clone () from /lib64/libc.so.6

    2. Yes, I run the query successfully on hive, this is the result:
    MapReduce Total cumulative CPU time: 14 minutes 0 seconds 600 msec
    Ended Job = job_1366614388548_3950
    MapReduce Jobs Launched:
    Job 0: Map: 21 Reduce: 1 Cumulative CPU: 840.6 sec HDFS Read:
    3848829397 HDFS Write: 10 SUCCESS
    Total MapReduce CPU Time Spent: 14 minutes 0 seconds 600 msec
    OK
    342887842
    Time taken: 110.618 seconds

    在 2013年5月21日星期二UTC+8上午1时16分16秒,Ishaan写道:
    Zesheng,

    To help us further investigate the problem, could you attach the impala
    logs? They should be found under /var/log/impalad/impalad.INFO.
    Additionally, can you successfully run the query in hive?

    Thanks,

    -- Ishaan

    On Mon, May 20, 2013 at 2:33 AM, Zesheng Wu wrote:

    I run the query "select count(*) from hbase_sdtst_m where month=5;" on
    our impala cluster, when the progress is near 100%(about 98%), the progress
    will not move forward anymore and the impala shell hangs forever.

    Anyone can help on this?
  • Ishaan Joshi at May 21, 2013 at 5:25 pm
    Thanks for filing this Zesheng, we're looking into it.

    -- Ishaan

    On Mon, May 20, 2013 at 8:54 PM, Zesheng Wu wrote:

    I find this is a bug of impalad when it deal with sequence file without
    contents, I created a jira issue here:
    https://issues.cloudera.org/browse/IMPALA-362

    在 2013年5月21日星期二UTC+8上午9时15分45秒,Zesheng Wu写道:
    1. I found that one of the impalad hangs on GetBytes(), this is the
    stackstrace:
    (gdb) bt
    #0 0x0000003627a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
    /lib64/libpthread.so.0
    #1 0x0000000000cc8ef8 in boost::condition_variable::**wait
    (this=0x4da8fe0, m=...) at /usr/include/boost/thread/**
    pthread/condition_variable.**hpp:20
    #2 0x0000000000ee0d7d in impala::ScannerContext::**Stream::GetBytesInternal
    (this=0x4da8fc0, requested_len=4, out_buffer=0x7fefa6e134a8, peek=false,
    out_len=0x7fefa6e1341c,
    eos=0x7fefa6e1341b) at /home/wuzesheng/work/cloud/**
    infra/impala/be/src/exec/**scanner-context.cc:248
    #3 0x0000000000eae7ee in impala::ScannerContext::**Stream::GetBytes
    (this=0x4da8fc0, requested_len=4, buffer=0x7fefa6e134a8,
    out_len=0x7fefa6e1341c, eos=0x7fefa6e1341b,
    status=0xdccaf18) at /home/wuzesheng/work/cloud/**
    infra/impala/be/src/exec/**scanner-context.inline.h:57
    #4 0x0000000000eae96a in impala::ScannerContext::**Stream::ReadBytes
    (this=0x4da8fc0, length=4, buf=0x7fefa6e134a8, status=0xdccaf18)
    at /home/wuzesheng/work/cloud/**infra/impala/be/src/exec/**
    scanner-context.inline.h:68
    #5 0x0000000000eaecd3 in impala::ScannerContext::**Stream::ReadInt
    (this=0x4da8fc0, val=0x7fefa6e1364c, status=0xdccaf18)
    at /home/wuzesheng/work/cloud/**infra/impala/be/src/exec/**
    scanner-context.inline.h:115
    #6 0x0000000000eb413a in impala::HdfsSequenceScanner::**ReadFileHeader
    (this=0xdccaea0) at /home/wuzesheng/work/cloud/**
    infra/impala/be/src/exec/hdfs-**sequence-scanner.cc:460
    #7 0x0000000000ef55d8 in impala::BaseSequenceScanner::**ProcessSplit
    (this=0xdccaea0, context=0x4e1b290)
    at /home/wuzesheng/work/cloud/**infra/impala/be/src/exec/base-**
    sequence-scanner.cc:92
    #8 0x0000000000e8ed8f in impala::HdfsScanNode::**ScannerThread
    (this=0xd818700, scanner=0xdccaea0, context=0x4e1b290)
    at /home/wuzesheng/work/cloud/**infra/impala/be/src/exec/hdfs-**
    scan-node.cc:795
    #9 0x0000000000ea1804 in boost::_mfi::mf2<void, impala::HdfsScanNode,
    impala::HdfsScanner*, impala::ScannerContext*>::**operator()
    (this=0x4d51730, p=0xd818700, a1=0xdccaea0,
    a2=0x4e1b290) at /usr/include/boost/bind/mem_**fn_template.hpp:280
    #10 0x0000000000ea16ca in boost::_bi::list3<boost::_bi::**value<impala::HdfsScanNode*>,
    boost::_bi::value<impala::**HdfsScanner*>, boost::_bi::value<impala::**ScannerContext*>
    ::operator()<boost::_mfi::**mf2<void, impala::HdfsScanNode,
    impala::HdfsScanner*, impala::ScannerContext*>, boost::_bi::list0>
    (this=0x4d51740, f=..., a=...)
    at /usr/include/boost/bind/bind.**hpp:392
    #11 0x0000000000ea15f1 in boost::_bi::bind_t<void, boost::_mfi::mf2<void,
    impala::HdfsScanNode, impala::HdfsScanner*, impala::ScannerContext*>,
    boost::_bi::list3<boost::_bi::**value<impala::HdfsScanNode*>,
    boost::_bi::value<impala::**HdfsScanner*>, boost::_bi::value<impala::**ScannerContext*>
    ::operator() (this=0x4d51730)
    at /usr/include/boost/bind/bind_**template.hpp:20
    #12 0x0000000000ea11fe in boost::detail::thread_data<**boost::_bi::bind_t<void,
    boost::_mfi::mf2<void, impala::HdfsScanNode, impala::HdfsScanner*,
    impala::ScannerContext*>, boost::_bi::list3<boost::_bi::**value<impala::HdfsScanNode*>,
    boost::_bi::value<impala::**HdfsScanner*>, boost::_bi::value<impala::**ScannerContext*>
    ::run (this=0x4d51600)
    at /usr/include/boost/thread/**detail/thread.hpp:56
    #13 0x00007fefdbd253fd in boost::(anonymous namespace)::thread_proxy
    (param=0x4d51600) at libs/thread/src/pthread/**thread.cpp:120
    #14 0x0000003627a07851 in start_thread () from /lib64/libpthread.so.0
    #15 0x00000039a16e811d in clone () from /lib64/libc.so.6

    2. Yes, I run the query successfully on hive, this is the result:
    MapReduce Total cumulative CPU time: 14 minutes 0 seconds 600 msec
    Ended Job = job_1366614388548_3950
    MapReduce Jobs Launched:
    Job 0: Map: 21 Reduce: 1 Cumulative CPU: 840.6 sec HDFS Read:
    3848829397 HDFS Write: 10 SUCCESS
    Total MapReduce CPU Time Spent: 14 minutes 0 seconds 600 msec
    OK
    342887842
    Time taken: 110.618 seconds

    在 2013年5月21日星期二UTC+8上午1时16分16秒,**Ishaan写道:
    Zesheng,

    To help us further investigate the problem, could you attach the
    impala logs? They should be found under /var/log/impalad/impalad.INFO.
    Additionally, can you successfully run the query in hive?

    Thanks,

    -- Ishaan

    On Mon, May 20, 2013 at 2:33 AM, Zesheng Wu wrote:

    I run the query "select count(*) from hbase_sdtst_m where month=5;" on
    our impala cluster, when the progress is near 100%(about 98%), the progress
    will not move forward anymore and the impala shell hangs forever.

    Anyone can help on this?

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupimpala-user @
categorieshadoop
postedMay 20, '13 at 9:33a
activeMay 21, '13 at 5:25p
posts4
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Ishaan Joshi: 2 posts Zesheng Wu: 2 posts

People

Translate

site design / logo © 2022 Grokbase