FAQ
Hi Alan,

The server I am using is configured as RAID, so Impala treats it as just
one block device and uses only one thread for scanning files.
(I can tell it from the MB/s that impala-server achieves. When I set it as NUM_SCANNER_THREADS=8,
impala achieves around 600MB/s, on the other hand, impala processes only
around 80 MB/s without the configuration.)

Anyways, (of cource) impala can't automatically detect hardware
configurations, so I need to set it manually.
The problem is that I can't do it. (I used to be able to do it, though.)
What might be the reason ?

Is there any way to configure NUM_SCANNER_THREADS except for setting
through impala-shell ?

Or, do I need to re-initialize impala configurations just to make sure
everything is configured properly ?
Is there any way for it ?

Thanks,

Hiro

2014年1月19日日曜日 5時20分47秒 UTC+9 Alan:
Hi Hiro,

Normally, you probably don't need to mess around with "NUM_SCANNER_THREADS".
I'm not sure how you determine that impala is scanning files with one
thread, but you can look for "AverageScannerThreadConcurrency" in the
query profile.

Keep in mind that if you're querying only one block (or one parquet file
which is one block), then Impala will use one scanner thread.

Thanks,
Alan



On Sat, Jan 18, 2014 at 6:06 AM, hiro <mogw...@gmail.com <javascript:>>wrote:
Hi,

After I set "statestore_subscriber_timeout_seconds" to 60sec,
now it's working without reconnection messages.

But, there is still something wrong.
I query with the following parameter,

impala > set NUM_SCANNER_THREADS=8;

but, impala-server still scans files with one thread.
It used to work fine.
Is there still commucation problem between statestore and impala-server ?
Do you have any idea what is going on ?

Thanks,

Hiro

2014年1月6日月曜日 21時53分41秒 UTC+9 hiro:
Hi,

I am using Impala 1.2.3 on CDH4.
I installed impala before and I have run several queries successfully.
I just started Impala again after a while and
I get errors like the following when I query.

------------------
[impalad003:21000] > select * from part where p_retailprice <= 900.18;
Query: select * from part where p_retailprice <= 900.18
ERROR: Cancelled due to unreachable impalad(s): impalad006:22000,
impalad063:22000, impalad015:22000, impalad100:22000, ...
ERROR: Invalid query handle
------------------

There seems some connection problems.
I also checked impalad.INFO in each impalad node,
it keeps reconnecting to state-store over and over and never stops as
shown below.
As far as I know, it wasn't like this before, so it might be the cause.

------------------
I0106 21:46:07.333633 32072 state-store-subscriber.cc:202]
impalad002:22000: Connection with state-store lost, entering recovery mode
I0106 21:46:07.333675 32072 state-store-subscriber.cc:206] Trying to
re-register with state-store, attempt: 1
I0106 21:46:07.336510 32072 state-store-subscriber.cc:210] Reconnected
to state-store. Exiting recovery mode
I0106 21:46:17.338811 32072 state-store-subscriber.cc:202]
impalad002:22000: Connection with state-store lost, entering recovery mode
I0106 21:46:17.338851 32072 state-store-subscriber.cc:206] Trying to
re-register with state-store, attempt: 1
I0106 21:46:17.339157 32072 state-store-subscriber.cc:210] Reconnected
to state-store. Exiting recovery mode
W0106 21:46:24.055603 320 Configuration.java:2227]
org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@a1e5879:an
attempt to override final parameter: mapreduce.job.end-
notification.max.retry.interval; Ignoring.
W0106 21:46:24.056337 320 Configuration.java:2227]
org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@a1e5879:an
attempt to override final parameter: mapreduce.job.end-notification.max.attempts;
Ignoring.
I0106 21:46:42.344254 32072 state-store-subscriber.cc:202]
impalad002:22000: Connection with state-store lost, entering recovery mode
I0106 21:46:42.344295 32072 state-store-subscriber.cc:206] Trying to
re-register with state-store, attempt: 1
I0106 21:46:42.344761 32072 state-store-subscriber.cc:210] Reconnected
to state-store. Exiting recovery mode
I0106 21:46:52.346424 32072 state-store-subscriber.cc:202]
impalad002:22000: Connection with state-store lost, entering recovery mode
I0106 21:46:52.346465 32072 state-store-subscriber.cc:206] Trying to
re-register with state-store, attempt: 1
I0106 21:46:52.347231 32072 state-store-subscriber.cc:210] Reconnected
to state-store. Exiting recovery mode
W0106 21:47:03.134521 325 Configuration.java:2227]
org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@6d347991:an
attempt to override final parameter: mapreduce.job.end-
notification.max.retry.interval; Ignoring.
W0106 21:47:03.135227 325 Configuration.java:2227]
org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@6d347991:an
attempt to override final parameter: mapreduce.job.end-notification.max.attempts;
Ignoring.
... (never stops)
------------------

Does anyone have any idea about this ?
Thanks in advance.


Hiro


To unsubscribe from this group and stop receiving emails from it, send
an email to impala-user...@cloudera.org <javascript:>.
To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.

Search Discussions

Discussion Posts

Previous

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 3 of 3 | next ›
Discussion Overview
groupimpala-user @
categorieshadoop
postedJan 18, '14 at 2:06p
activeJan 19, '14 at 3:30a
posts3
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Hiro: 2 posts Alan Choi: 1 post

People

Translate

site design / logo © 2022 Grokbase