FAQ
Hi Sammy,

The huge CPU usage is more likely to be caused by a large number of files
or blocks. We're aware of the performance problem when the catalog/metadata
is huge. We're working on it. Stay tuned!

Thanks,
Alan


On Mon, Jan 13, 2014 at 5:29 PM, Sammy Yu wrote:

Hi,
I also see that catalogd is not behaving very nicely after this table
was found. It's eating up a lot of CPU, this is the output from top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND


4894 impala 20 0 3008m 1.2g 23m S 83.1 16.8 17:37.46 catalogd



Is there a limit to the maximum number of partitions a table can handle in
impala? If there is, could one create a tables per partition? Would
having 30,000+ tables causing problems in impala?

Thanks,
Sammy
On Friday, January 10, 2014 7:39:07 PM UTC-8, Sammy Yu wrote:

Hi,
I'm running impala 1.2.3 on with a rcfile table with 38687 partitions
that was created from hive. Afterwards, I did a refresh metadata and
compared the select count(1) results and noticed that the result differed
(impala results was significantly smaller than hive). I did further
investigation and determined that impala was not considering some of my
later partitions.

The hive show partition results came back as expected. I tried using the
show table stats command in impala, but I'm getting an error:
[ip-10-124-195-6.ec2.internal:21000] > SHOW TABLE STATS rcfile_3p;
Query: show TABLE STATS rcfile_3p
ERROR: IllegalArgumentException: Comparison method violates its general
contract!

Thanks for your help.

Best,
Sammy
To unsubscribe from this group and stop receiving emails from it, send an
email to impala-user+unsubscribe@cloudera.org.
To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 3 of 7 | next ›
Discussion Overview
groupimpala-user @
categorieshadoop
postedJan 11, '14 at 3:39a
activeJan 15, '14 at 6:15p
posts7
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Alan Choi: 4 posts Sammy Yu: 3 posts

People

Translate

site design / logo © 2022 Grokbase