FAQ
Hi,

After I set "statestore_subscriber_timeout_seconds" to 60sec,
now it's working without reconnection messages.

But, there is still something wrong.
I query with the following parameter,

impala > set NUM_SCANNER_THREADS=8;

but, impala-server still scans files with one thread.
It used to work fine.
Is there still commucation problem between statestore and impala-server ?
Do you have any idea what is going on ?

Thanks,

Hiro

2014年1月6日月曜日 21時53分41秒 UTC+9 hiro:
Hi,

I am using Impala 1.2.3 on CDH4.
I installed impala before and I have run several queries successfully.
I just started Impala again after a while and
I get errors like the following when I query.

------------------
[impalad003:21000] > select * from part where p_retailprice <= 900.18;
Query: select * from part where p_retailprice <= 900.18
ERROR: Cancelled due to unreachable impalad(s): impalad006:22000,
impalad063:22000, impalad015:22000, impalad100:22000, ...
ERROR: Invalid query handle
------------------

There seems some connection problems.
I also checked impalad.INFO in each impalad node,
it keeps reconnecting to state-store over and over and never stops as
shown below.
As far as I know, it wasn't like this before, so it might be the cause.

------------------
I0106 21:46:07.333633 32072 state-store-subscriber.cc:202]
impalad002:22000: Connection with state-store lost, entering recovery mode
I0106 21:46:07.333675 32072 state-store-subscriber.cc:206] Trying to
re-register with state-store, attempt: 1
I0106 21:46:07.336510 32072 state-store-subscriber.cc:210] Reconnected to
state-store. Exiting recovery mode
I0106 21:46:17.338811 32072 state-store-subscriber.cc:202]
impalad002:22000: Connection with state-store lost, entering recovery mode
I0106 21:46:17.338851 32072 state-store-subscriber.cc:206] Trying to
re-register with state-store, attempt: 1
I0106 21:46:17.339157 32072 state-store-subscriber.cc:210] Reconnected to
state-store. Exiting recovery mode
W0106 21:46:24.055603 320 Configuration.java:2227]
org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@a1e5879:an attempt
to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
W0106 21:46:24.056337 320 Configuration.java:2227]
org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@a1e5879:an attempt
to override final parameter: mapreduce.job.end-notification.max.attempts;
Ignoring.
I0106 21:46:42.344254 32072 state-store-subscriber.cc:202]
impalad002:22000: Connection with state-store lost, entering recovery mode
I0106 21:46:42.344295 32072 state-store-subscriber.cc:206] Trying to
re-register with state-store, attempt: 1
I0106 21:46:42.344761 32072 state-store-subscriber.cc:210] Reconnected to
state-store. Exiting recovery mode
I0106 21:46:52.346424 32072 state-store-subscriber.cc:202]
impalad002:22000: Connection with state-store lost, entering recovery mode
I0106 21:46:52.346465 32072 state-store-subscriber.cc:206] Trying to
re-register with state-store, attempt: 1
I0106 21:46:52.347231 32072 state-store-subscriber.cc:210] Reconnected to
state-store. Exiting recovery mode
W0106 21:47:03.134521 325 Configuration.java:2227]
org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@6d347991:an attempt
to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
W0106 21:47:03.135227 325 Configuration.java:2227]
org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@6d347991:an attempt
to override final parameter: mapreduce.job.end-notification.max.attempts;
Ignoring.
... (never stops)
------------------

Does anyone have any idea about this ?
Thanks in advance.


Hiro

To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.

Search Discussions

  • Alan Choi at Jan 18, 2014 at 8:20 pm
    Hi Hiro,

    Normally, you probably don't need to mess around with
    "NUM_SCANNER_THREADS". I'm not sure how you determine that impala is
    scanning files with one thread, but you can look for
    "AverageScannerThreadConcurrency"
    in the query profile.

    Keep in mind that if you're querying only one block (or one parquet file
    which is one block), then Impala will use one scanner thread.

    Thanks,
    Alan


    On Sat, Jan 18, 2014 at 6:06 AM, hiro wrote:

    Hi,

    After I set "statestore_subscriber_timeout_seconds" to 60sec,
    now it's working without reconnection messages.

    But, there is still something wrong.
    I query with the following parameter,

    impala > set NUM_SCANNER_THREADS=8;

    but, impala-server still scans files with one thread.
    It used to work fine.
    Is there still commucation problem between statestore and impala-server ?
    Do you have any idea what is going on ?

    Thanks,

    Hiro

    2014年1月6日月曜日 21時53分41秒 UTC+9 hiro:
    Hi,

    I am using Impala 1.2.3 on CDH4.
    I installed impala before and I have run several queries successfully.
    I just started Impala again after a while and
    I get errors like the following when I query.

    ------------------
    [impalad003:21000] > select * from part where p_retailprice <= 900.18;
    Query: select * from part where p_retailprice <= 900.18
    ERROR: Cancelled due to unreachable impalad(s): impalad006:22000,
    impalad063:22000, impalad015:22000, impalad100:22000, ...
    ERROR: Invalid query handle
    ------------------

    There seems some connection problems.
    I also checked impalad.INFO in each impalad node,
    it keeps reconnecting to state-store over and over and never stops as
    shown below.
    As far as I know, it wasn't like this before, so it might be the cause.

    ------------------
    I0106 21:46:07.333633 32072 state-store-subscriber.cc:202]
    impalad002:22000: Connection with state-store lost, entering recovery mode
    I0106 21:46:07.333675 32072 state-store-subscriber.cc:206] Trying to
    re-register with state-store, attempt: 1
    I0106 21:46:07.336510 32072 state-store-subscriber.cc:210] Reconnected to
    state-store. Exiting recovery mode
    I0106 21:46:17.338811 32072 state-store-subscriber.cc:202]
    impalad002:22000: Connection with state-store lost, entering recovery mode
    I0106 21:46:17.338851 32072 state-store-subscriber.cc:206] Trying to
    re-register with state-store, attempt: 1
    I0106 21:46:17.339157 32072 state-store-subscriber.cc:210] Reconnected to
    state-store. Exiting recovery mode
    W0106 21:46:24.055603 320 Configuration.java:2227]
    org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@a1e5879:an
    attempt to override final parameter: mapreduce.job.end-
    notification.max.retry.interval; Ignoring.
    W0106 21:46:24.056337 320 Configuration.java:2227]
    org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@a1e5879:an
    attempt to override final parameter: mapreduce.job.end-notification.max.attempts;
    Ignoring.
    I0106 21:46:42.344254 32072 state-store-subscriber.cc:202]
    impalad002:22000: Connection with state-store lost, entering recovery mode
    I0106 21:46:42.344295 32072 state-store-subscriber.cc:206] Trying to
    re-register with state-store, attempt: 1
    I0106 21:46:42.344761 32072 state-store-subscriber.cc:210] Reconnected to
    state-store. Exiting recovery mode
    I0106 21:46:52.346424 32072 state-store-subscriber.cc:202]
    impalad002:22000: Connection with state-store lost, entering recovery mode
    I0106 21:46:52.346465 32072 state-store-subscriber.cc:206] Trying to
    re-register with state-store, attempt: 1
    I0106 21:46:52.347231 32072 state-store-subscriber.cc:210] Reconnected to
    state-store. Exiting recovery mode
    W0106 21:47:03.134521 325 Configuration.java:2227]
    org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@6d347991:an
    attempt to override final parameter: mapreduce.job.end-
    notification.max.retry.interval; Ignoring.
    W0106 21:47:03.135227 325 Configuration.java:2227]
    org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@6d347991:an
    attempt to override final parameter: mapreduce.job.end-notification.max.attempts;
    Ignoring.
    ... (never stops)
    ------------------

    Does anyone have any idea about this ?
    Thanks in advance.


    Hiro


    To unsubscribe from this group and stop receiving emails from it, send
    an email to impala-user+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.
  • Hiro at Jan 19, 2014 at 3:30 am
    Hi Alan,

    The server I am using is configured as RAID, so Impala treats it as just
    one block device and uses only one thread for scanning files.
    (I can tell it from the MB/s that impala-server achieves. When I set it as NUM_SCANNER_THREADS=8,
    impala achieves around 600MB/s, on the other hand, impala processes only
    around 80 MB/s without the configuration.)

    Anyways, (of cource) impala can't automatically detect hardware
    configurations, so I need to set it manually.
    The problem is that I can't do it. (I used to be able to do it, though.)
    What might be the reason ?

    Is there any way to configure NUM_SCANNER_THREADS except for setting
    through impala-shell ?

    Or, do I need to re-initialize impala configurations just to make sure
    everything is configured properly ?
    Is there any way for it ?

    Thanks,

    Hiro

    2014年1月19日日曜日 5時20分47秒 UTC+9 Alan:
    Hi Hiro,

    Normally, you probably don't need to mess around with "NUM_SCANNER_THREADS".
    I'm not sure how you determine that impala is scanning files with one
    thread, but you can look for "AverageScannerThreadConcurrency" in the
    query profile.

    Keep in mind that if you're querying only one block (or one parquet file
    which is one block), then Impala will use one scanner thread.

    Thanks,
    Alan



    On Sat, Jan 18, 2014 at 6:06 AM, hiro <mogw...@gmail.com <javascript:>>wrote:
    Hi,

    After I set "statestore_subscriber_timeout_seconds" to 60sec,
    now it's working without reconnection messages.

    But, there is still something wrong.
    I query with the following parameter,

    impala > set NUM_SCANNER_THREADS=8;

    but, impala-server still scans files with one thread.
    It used to work fine.
    Is there still commucation problem between statestore and impala-server ?
    Do you have any idea what is going on ?

    Thanks,

    Hiro

    2014年1月6日月曜日 21時53分41秒 UTC+9 hiro:
    Hi,

    I am using Impala 1.2.3 on CDH4.
    I installed impala before and I have run several queries successfully.
    I just started Impala again after a while and
    I get errors like the following when I query.

    ------------------
    [impalad003:21000] > select * from part where p_retailprice <= 900.18;
    Query: select * from part where p_retailprice <= 900.18
    ERROR: Cancelled due to unreachable impalad(s): impalad006:22000,
    impalad063:22000, impalad015:22000, impalad100:22000, ...
    ERROR: Invalid query handle
    ------------------

    There seems some connection problems.
    I also checked impalad.INFO in each impalad node,
    it keeps reconnecting to state-store over and over and never stops as
    shown below.
    As far as I know, it wasn't like this before, so it might be the cause.

    ------------------
    I0106 21:46:07.333633 32072 state-store-subscriber.cc:202]
    impalad002:22000: Connection with state-store lost, entering recovery mode
    I0106 21:46:07.333675 32072 state-store-subscriber.cc:206] Trying to
    re-register with state-store, attempt: 1
    I0106 21:46:07.336510 32072 state-store-subscriber.cc:210] Reconnected
    to state-store. Exiting recovery mode
    I0106 21:46:17.338811 32072 state-store-subscriber.cc:202]
    impalad002:22000: Connection with state-store lost, entering recovery mode
    I0106 21:46:17.338851 32072 state-store-subscriber.cc:206] Trying to
    re-register with state-store, attempt: 1
    I0106 21:46:17.339157 32072 state-store-subscriber.cc:210] Reconnected
    to state-store. Exiting recovery mode
    W0106 21:46:24.055603 320 Configuration.java:2227]
    org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@a1e5879:an
    attempt to override final parameter: mapreduce.job.end-
    notification.max.retry.interval; Ignoring.
    W0106 21:46:24.056337 320 Configuration.java:2227]
    org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@a1e5879:an
    attempt to override final parameter: mapreduce.job.end-notification.max.attempts;
    Ignoring.
    I0106 21:46:42.344254 32072 state-store-subscriber.cc:202]
    impalad002:22000: Connection with state-store lost, entering recovery mode
    I0106 21:46:42.344295 32072 state-store-subscriber.cc:206] Trying to
    re-register with state-store, attempt: 1
    I0106 21:46:42.344761 32072 state-store-subscriber.cc:210] Reconnected
    to state-store. Exiting recovery mode
    I0106 21:46:52.346424 32072 state-store-subscriber.cc:202]
    impalad002:22000: Connection with state-store lost, entering recovery mode
    I0106 21:46:52.346465 32072 state-store-subscriber.cc:206] Trying to
    re-register with state-store, attempt: 1
    I0106 21:46:52.347231 32072 state-store-subscriber.cc:210] Reconnected
    to state-store. Exiting recovery mode
    W0106 21:47:03.134521 325 Configuration.java:2227]
    org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@6d347991:an
    attempt to override final parameter: mapreduce.job.end-
    notification.max.retry.interval; Ignoring.
    W0106 21:47:03.135227 325 Configuration.java:2227]
    org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@6d347991:an
    attempt to override final parameter: mapreduce.job.end-notification.max.attempts;
    Ignoring.
    ... (never stops)
    ------------------

    Does anyone have any idea about this ?
    Thanks in advance.


    Hiro


    To unsubscribe from this group and stop receiving emails from it, send
    an email to impala-user...@cloudera.org <javascript:>.
    To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupimpala-user @
categorieshadoop
postedJan 18, '14 at 2:06p
activeJan 19, '14 at 3:30a
posts3
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Hiro: 2 posts Alan Choi: 1 post

People

Translate

site design / logo © 2022 Grokbase