FAQ
Moving this over to impala-user.

---------- Forwarded message ----------
From: <priyadarshini.verma@gmail.com>
Date: Sun, Jan 20, 2013 at 10:48 PM
Subject: Re: CDH Impala giving error randomly
To: cdh-dev@cloudera.org
Cc: priyadarshini.verma@gmail.com


This appears to be ThriftServer issue. I'm still trying to find the
root cause... but it appears to be an issue related to the following
error msg -

TThreadedServer client died: No more data to read.


Any help would be highy appreciated!


On Friday, 18 January 2013 20:01:39 UTC+5:30, priyadars...@gmail.com wrote:

Hi,

I'm trying to gauge Impala's performance. I have a 3 node CDH4 setup with 3 impalads running. My test setup basically is firing sequential queries (basic ones on pre created Hive tables - Im using Hive metastore) to these 3 impala ds in a round robin fashion.

It worked pretty well upto 10 parallel executions of these sequential queries.

But when I increased the concurrency to 15 - I started getting all sorts of weird exceptions - one of them is pasted below -
(note: these errors occur on a random basis - for some of my threads - I;m able to get results)
----
NINFO0118 14:20:01.174000 Thread-6 org.apache.hadoop.hive.metastore.HiveMetaStore] 6: get_partitions : db=default tbl=ds_internal_100
INFO0118 14:20:01.174000 Thread-3 org.apache.hadoop.hive.metastore.HiveMetaStore] 1: get_partitions : db=default tbl=ds_internal_100
INFO0118 14:20:01.175000 Thread-6 org.apache.hadoop.hive.metastore.HiveMetaStore.audit] ugi=impala ip=unknown-ip-addr cmd=get_partitions : db=default tbl=ds_internal_100
INFO0118 14:20:01.174000 Thread-4 DataNucleus.Datastore] The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
INFO0118 14:20:01.175000 Thread-3 org.apache.hadoop.hive.metastore.HiveMetaStore.audit] ugi=impala ip=unknown-ip-addr cmd=get_partitions : db=default tbl=ds_internal_100
INFO0118 14:20:01.175000 Thread-4 DataNucleus.Persistence] Managing Persistence of Class : org.apache.hadoop.hive.metastore.model.MPartition [Table : `PARTITIONS`, InheritanceStrategy : new-table]
INFO0118 14:20:01.175000 Thread-4 DataNucleus.Persistence] Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MPartition.parameters [Table : `PARTITION_PARAMS`]
INFO0118 14:20:01.176000 Thread-4 DataNucleus.Persistence] Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MPartition.values [Table : `PARTITION_KEY_VALS`]
INFO0118 14:20:01.176000 Thread-3 com.cloudera.impala.service.Frontend] Analysis exception (in select count(*) from default.ds_internal_100)
I0118 14:20:01.204556 8541 status.cc:36] com.cloudera.impala.common.AnalysisException: Analysis exception (in select count(*) from default.ds_internal_100)
at com.cloudera.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:133)
at com.cloudera.impala.service.Frontend.createExecRequest(Frontend.java:216)
at com.cloudera.impala.service.JniFrontend.createExecRequest(JniFrontend.java:86)
Caused by: com.cloudera.impala.common.AnalysisException: Failed to load metadata for table: ds_internal_100
at com.cloudera.impala.analysis.Analyzer.registerBaseTableRef(Analyzer.java:174)
at com.cloudera.impala.analysis.BaseTableRef.analyze(BaseTableRef.java:51)
at com.cloudera.impala.analysis.SelectStmt.analyze(SelectStmt.java:115)
at com.cloudera.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:130)
... 2 more
Caused by: com.cloudera.impala.catalog.Db$TableLoadingException: Failed to load metadata for table: ds_internal_100
at com.cloudera.impala.catalog.HdfsTable.load(HdfsTable.java:289)
at com.cloudera.impala.catalog.Table.load(Table.java:114)
at com.cloudera.impala.catalog.Db$LazyTableMap.get(Db.java:151)
at com.cloudera.impala.catalog.Db.getTable(Db.java:217)
at com.cloudera.impala.analysis.Analyzer.registerBaseTableRef(Analyzer.java:172)
... 5 more
Caused by: java.lang.IllegalStateException: Table object has not been been initialised : `PARTITIONS`
at org.datanucleus.store.rdbms.table.AbstractTable.assertIsInitialized(AbstractTable.java:625)
at org.datanucleus.store.rdbms.table.ClassTable.getMemberMapping(ClassTable.java:2996)
at org.datanucleus.store.mapped.expression.LogicSetExpression.newFieldExpression(LogicSetExpression.java:195)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileIdentifier(JDOQLQueryCompiler.java:953)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compilePrimary(JDOQLQueryCompiler.java:774)
at org.datanucleus.store.rdbms.query.legacy.QueryCompiler.compileUnaryExpressionNotPlusMinus(QueryCompiler.java:1333)
at org.datanucleus.store.rdbms.query.legacy.QueryCompiler.compileUnaryExpression(QueryCompiler.java:1314)
at org.datanucleus.store.rdbms.query.legacy.QueryCompiler.compileMultiplicativeExpression(QueryCompiler.java:1267)
at org.datanucleus.store.rdbms.query.legacy.QueryCompiler.compileAdditiveExpression(QueryCompiler.java:1244)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileRelationalExpression(JDOQLQueryCompiler.java:687)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileEqualityExpression(JDOQLQueryCompiler.java:659)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileAndExpression(JDOQLQueryCompiler.java:647)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileExclusiveOrExpression(JDOQLQueryCompiler.java:635)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileInclusiveOrExpression(JDOQLQueryCompiler.java:623)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileConditionalAndExpression(JDOQLQueryCompiler.java:611)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileConditionalOrExpression(JDOQLQueryCompiler.java:593)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileExpression(JDOQLQueryCompiler.java:570)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileExpressionFromString(JDOQLQueryCompiler.java:550)
at org.datanucleus.store.rdbms.query.legacy.QueryCompiler.compileFilter(QueryCompiler.java:852)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.performCompile(JDOQLQueryCompiler.java:261)
at org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile(QueryCompiler.java:374)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(JDOQLQueryCompiler.java:225)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(JDOQLQuery.java:175)
at org.datanucleus.store.query.Query.executeQuery(Query.java:1628)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245)
at org.datanucleus.store.query.Query.executeWithArray(Query.java:1499)
at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:266)
at org.apache.hadoop.hive.metastore.ObjectStore.listMPartitions(ObjectStore.java:1516)
at org.apache.hadoop.hive.metastore.ObjectStore.getPartitions(ObjectStore.java:1256)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:111)
at $Proxy11.getPartitions(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions(HiveMetaStore.java:1725)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions(HiveMetaStore.java:1725)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitions(HiveMetaStoreClient.java:645)
at com.cloudera.impala.catalog.HdfsTable.load(HdfsTable.java:285)
... 9 more

@ 0x767901 (unknown)
@ 0x638274 (unknown)
@ 0x640690 (unknown)
@ 0x640e24 (unknown)
@ 0x641480 (unknown)
@ 0x7c0dbf (unknown)
@ 0x7b9d04 (unknown)
@ 0x7adbfb (unknown)
@ 0x77ac49 (unknown)
@ 0x76bd4f (unknown)
@ 0x76dcc2 (unknown)
@ 0x768822 (unknown)
@ 0x3b47e07851 (unknown)
@ 0x3b47ae811d (unknown)
INFO0118 14:20:01.249000 Thread-4 DataNucleus.Datastore.Schema] Validating 4 index(es) for table `PARTITIONS`
INFO0118 14:20:01.250000 Thread-4 DataNucleus.Datastore.Schema] Validating 2 foreign key(s) for table `PARTITIONS`
INFO0118 14:20:01.252000 Thread-4 DataNucleus.Datastore.Schema] Validating 2 unique key(s) for table `PARTITIONS`
INFO0118 14:20:01.253000 Thread-4 DataNucleus.Datastore.Schema] Validating 2 index(es) for table `PARTITION_KEY_VALS`
INFO0118 14:20:01.254000 Thread-4 DataNucleus.Datastore.Schema] Validating 1 foreign key(s) for table `PARTITION_KEY_VALS`
INFO0118 14:20:01.256000 Thread-4 DataNucleus.Datastore.Schema] Validating 1 unique key(s) for table `PARTITION_KEY_VALS`
INFO0118 14:20:01.257000 Thread-4 DataNucleus.Datastore.Schema] Validating 2 index(es) for table `PARTITION_PARAMS`
INFO0118 14:20:01.258000 Thread-4 DataNucleus.Datastore.Schema] Validating 1 foreign key(s) for table `PARTITION_PARAMS`
INFO0118 14:20:01.259000 Thread-4 DataNucleus.Datastore.Schema] Validating 1 unique key(s) for table `PARTITION_PARAMS`
-----
--



--

--

Search Discussions

  • Marcel Kornacker at Jan 25, 2013 at 8:40 pm
    I filed a bug ticket: https://issues.cloudera.org/browse/IMPALA-30
    On Wed, Jan 23, 2013 at 12:32 PM, Marcel Kornacker wrote:
    Moving this over to impala-user.

    ---------- Forwarded message ----------
    From: <priyadarshini.verma@gmail.com>
    Date: Sun, Jan 20, 2013 at 10:48 PM
    Subject: Re: CDH Impala giving error randomly
    To: cdh-dev@cloudera.org
    Cc: priyadarshini.verma@gmail.com


    This appears to be ThriftServer issue. I'm still trying to find the
    root cause... but it appears to be an issue related to the following
    error msg -

    TThreadedServer client died: No more data to read.


    Any help would be highy appreciated!


    On Friday, 18 January 2013 20:01:39 UTC+5:30, priyadars...@gmail.com wrote:

    Hi,

    I'm trying to gauge Impala's performance. I have a 3 node CDH4 setup with 3 impalads running. My test setup basically is firing sequential queries (basic ones on pre created Hive tables - Im using Hive metastore) to these 3 impala ds in a round robin fashion.

    It worked pretty well upto 10 parallel executions of these sequential queries.

    But when I increased the concurrency to 15 - I started getting all sorts of weird exceptions - one of them is pasted below -
    (note: these errors occur on a random basis - for some of my threads - I;m able to get results)
    ----
    NINFO0118 14:20:01.174000 Thread-6 org.apache.hadoop.hive.metastore.HiveMetaStore] 6: get_partitions : db=default tbl=ds_internal_100
    INFO0118 14:20:01.174000 Thread-3 org.apache.hadoop.hive.metastore.HiveMetaStore] 1: get_partitions : db=default tbl=ds_internal_100
    INFO0118 14:20:01.175000 Thread-6 org.apache.hadoop.hive.metastore.HiveMetaStore.audit] ugi=impala ip=unknown-ip-addr cmd=get_partitions : db=default tbl=ds_internal_100
    INFO0118 14:20:01.174000 Thread-4 DataNucleus.Datastore] The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
    INFO0118 14:20:01.175000 Thread-3 org.apache.hadoop.hive.metastore.HiveMetaStore.audit] ugi=impala ip=unknown-ip-addr cmd=get_partitions : db=default tbl=ds_internal_100
    INFO0118 14:20:01.175000 Thread-4 DataNucleus.Persistence] Managing Persistence of Class : org.apache.hadoop.hive.metastore.model.MPartition [Table : `PARTITIONS`, InheritanceStrategy : new-table]
    INFO0118 14:20:01.175000 Thread-4 DataNucleus.Persistence] Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MPartition.parameters [Table : `PARTITION_PARAMS`]
    INFO0118 14:20:01.176000 Thread-4 DataNucleus.Persistence] Managing Persistence of Field : org.apache.hadoop.hive.metastore.model.MPartition.values [Table : `PARTITION_KEY_VALS`]
    INFO0118 14:20:01.176000 Thread-3 com.cloudera.impala.service.Frontend] Analysis exception (in select count(*) from default.ds_internal_100)
    I0118 14:20:01.204556 8541 status.cc:36] com.cloudera.impala.common.AnalysisException: Analysis exception (in select count(*) from default.ds_internal_100)
    at com.cloudera.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:133)
    at com.cloudera.impala.service.Frontend.createExecRequest(Frontend.java:216)
    at com.cloudera.impala.service.JniFrontend.createExecRequest(JniFrontend.java:86)
    Caused by: com.cloudera.impala.common.AnalysisException: Failed to load metadata for table: ds_internal_100
    at com.cloudera.impala.analysis.Analyzer.registerBaseTableRef(Analyzer.java:174)
    at com.cloudera.impala.analysis.BaseTableRef.analyze(BaseTableRef.java:51)
    at com.cloudera.impala.analysis.SelectStmt.analyze(SelectStmt.java:115)
    at com.cloudera.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:130)
    ... 2 more
    Caused by: com.cloudera.impala.catalog.Db$TableLoadingException: Failed to load metadata for table: ds_internal_100
    at com.cloudera.impala.catalog.HdfsTable.load(HdfsTable.java:289)
    at com.cloudera.impala.catalog.Table.load(Table.java:114)
    at com.cloudera.impala.catalog.Db$LazyTableMap.get(Db.java:151)
    at com.cloudera.impala.catalog.Db.getTable(Db.java:217)
    at com.cloudera.impala.analysis.Analyzer.registerBaseTableRef(Analyzer.java:172)
    ... 5 more
    Caused by: java.lang.IllegalStateException: Table object has not been been initialised : `PARTITIONS`
    at org.datanucleus.store.rdbms.table.AbstractTable.assertIsInitialized(AbstractTable.java:625)
    at org.datanucleus.store.rdbms.table.ClassTable.getMemberMapping(ClassTable.java:2996)
    at org.datanucleus.store.mapped.expression.LogicSetExpression.newFieldExpression(LogicSetExpression.java:195)
    at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileIdentifier(JDOQLQueryCompiler.java:953)
    at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compilePrimary(JDOQLQueryCompiler.java:774)
    at org.datanucleus.store.rdbms.query.legacy.QueryCompiler.compileUnaryExpressionNotPlusMinus(QueryCompiler.java:1333)
    at org.datanucleus.store.rdbms.query.legacy.QueryCompiler.compileUnaryExpression(QueryCompiler.java:1314)
    at org.datanucleus.store.rdbms.query.legacy.QueryCompiler.compileMultiplicativeExpression(QueryCompiler.java:1267)
    at org.datanucleus.store.rdbms.query.legacy.QueryCompiler.compileAdditiveExpression(QueryCompiler.java:1244)
    at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileRelationalExpression(JDOQLQueryCompiler.java:687)
    at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileEqualityExpression(JDOQLQueryCompiler.java:659)
    at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileAndExpression(JDOQLQueryCompiler.java:647)
    at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileExclusiveOrExpression(JDOQLQueryCompiler.java:635)
    at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileInclusiveOrExpression(JDOQLQueryCompiler.java:623)
    at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileConditionalAndExpression(JDOQLQueryCompiler.java:611)
    at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileConditionalOrExpression(JDOQLQueryCompiler.java:593)
    at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileExpression(JDOQLQueryCompiler.java:570)
    at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileExpressionFromString(JDOQLQueryCompiler.java:550)
    at org.datanucleus.store.rdbms.query.legacy.QueryCompiler.compileFilter(QueryCompiler.java:852)
    at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.performCompile(JDOQLQueryCompiler.java:261)
    at org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile(QueryCompiler.java:374)
    at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(JDOQLQueryCompiler.java:225)
    at org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(JDOQLQuery.java:175)
    at org.datanucleus.store.query.Query.executeQuery(Query.java:1628)
    at org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245)
    at org.datanucleus.store.query.Query.executeWithArray(Query.java:1499)
    at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:266)
    at org.apache.hadoop.hive.metastore.ObjectStore.listMPartitions(ObjectStore.java:1516)
    at org.apache.hadoop.hive.metastore.ObjectStore.getPartitions(ObjectStore.java:1256)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:111)
    at $Proxy11.getPartitions(Unknown Source)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions(HiveMetaStore.java:1725)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions(HiveMetaStore.java:1725)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitions(HiveMetaStoreClient.java:645)
    at com.cloudera.impala.catalog.HdfsTable.load(HdfsTable.java:285)
    ... 9 more

    @ 0x767901 (unknown)
    @ 0x638274 (unknown)
    @ 0x640690 (unknown)
    @ 0x640e24 (unknown)
    @ 0x641480 (unknown)
    @ 0x7c0dbf (unknown)
    @ 0x7b9d04 (unknown)
    @ 0x7adbfb (unknown)
    @ 0x77ac49 (unknown)
    @ 0x76bd4f (unknown)
    @ 0x76dcc2 (unknown)
    @ 0x768822 (unknown)
    @ 0x3b47e07851 (unknown)
    @ 0x3b47ae811d (unknown)
    INFO0118 14:20:01.249000 Thread-4 DataNucleus.Datastore.Schema] Validating 4 index(es) for table `PARTITIONS`
    INFO0118 14:20:01.250000 Thread-4 DataNucleus.Datastore.Schema] Validating 2 foreign key(s) for table `PARTITIONS`
    INFO0118 14:20:01.252000 Thread-4 DataNucleus.Datastore.Schema] Validating 2 unique key(s) for table `PARTITIONS`
    INFO0118 14:20:01.253000 Thread-4 DataNucleus.Datastore.Schema] Validating 2 index(es) for table `PARTITION_KEY_VALS`
    INFO0118 14:20:01.254000 Thread-4 DataNucleus.Datastore.Schema] Validating 1 foreign key(s) for table `PARTITION_KEY_VALS`
    INFO0118 14:20:01.256000 Thread-4 DataNucleus.Datastore.Schema] Validating 1 unique key(s) for table `PARTITION_KEY_VALS`
    INFO0118 14:20:01.257000 Thread-4 DataNucleus.Datastore.Schema] Validating 2 index(es) for table `PARTITION_PARAMS`
    INFO0118 14:20:01.258000 Thread-4 DataNucleus.Datastore.Schema] Validating 1 foreign key(s) for table `PARTITION_PARAMS`
    INFO0118 14:20:01.259000 Thread-4 DataNucleus.Datastore.Schema] Validating 1 unique key(s) for table `PARTITION_PARAMS`
    -----
    --



    --
    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupimpala-user @
categorieshadoop
postedJan 23, '13 at 8:32p
activeJan 25, '13 at 8:40p
posts2
users1
websitecloudera.com
irc#hadoop

1 user in discussion

Marcel Kornacker: 2 posts

People

Translate

site design / logo © 2022 Grokbase