Grokbase › Groups › Hadoop › impala-user › March 2014
FAQ
Hi Percy,

Have you changed your hbase config recently? If so, please make sure that
you've copied the latest config to Impala.

Thanks,
Alan

On Tue, Mar 11, 2014 at 4:46 PM, Marcel Kornacker wrote:

Please provide the impalad and catalogd log files.
On Tue, Mar 11, 2014 at 4:45 PM, Percy Zhao wrote:
Any update on this? Also I can see from netstat that catalogd is
connecting
to the right region server.

thanks,
percy

On Monday, March 10, 2014 11:31:57 AM UTC-7, Percy Zhao wrote:

I restarted impalad and catalogd, then catalogd log file keep throwing
error message like this over and over again

W0311 02:14:39.846324 10258 HConnectionManager.java:1144] Encountered
problems when prefetch hbase:meta table:
Java exception follows:
java.io.IOException: HRegionInfo was null or empty in Meta for ssp,
row=ssp,,99999999999999
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:152)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1141)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1201)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1093)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1050)
at
org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:330)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:314)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:186)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:154)
at
com.cloudera.impala.catalog.HBaseTable.load(HBaseTable.java:235)
at com.cloudera.impala.catalog.Table.load(Table.java:176)
at com.cloudera.impala.catalog.Db.loadTable(Db.java:95)
at com.cloudera.impala.catalog.Db.access$000(Db.java:52)
at com.cloudera.impala.catalog.Db$1.load(Db.java:69)
at com.cloudera.impala.catalog.Db$1.load(Db.java:65)
at
com.cloudera.impala.catalog.CatalogObjectCache$1.call(CatalogObjectCache.java:229)
at
com.cloudera.impala.catalog.CatalogObjectCache$1.call(CatalogObjectCache.java:226)
at
com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
at
com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
at
com.cloudera.impala.catalog.CatalogObjectCache.get(CatalogObjectCache.java:226)
at com.cloudera.impala.catalog.Db.getTable(Db.java:189)
at com.cloudera.impala.catalog.Catalog$2.run(Catalog.java:513)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
W0311 02:14:39.854532 10261 HConnectionManager.java:1144] Encountered
problems when prefetch hbase:meta table:
Java exception follows:
java.io.IOException: HRegionInfo was null or empty in Meta for
hbasealltypessmall, row=hbasealltypessmall,,99999999999999
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:152)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1141)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1201)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1093)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1050)
at
org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:330)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:314)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:186)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:154)
at
com.cloudera.impala.catalog.HBaseTable.load(HBaseTable.java:235)
at com.cloudera.impala.catalog.Table.load(Table.java:176)
at com.cloudera.impala.catalog.Db.loadTable(Db.java:95)
at com.cloudera.impala.catalog.Db.access$000(Db.java:52)
at com.cloudera.impala.catalog.Db$1.load(Db.java:69)
at com.cloudera.impala.catalog.Db$1.load(Db.java:65)
at
com.cloudera.impala.catalog.CatalogObjectCache$1.call(CatalogObjectCache.java:229)
at
com.cloudera.impala.catalog.CatalogObjectCache$1.call(CatalogObjectCache.java:226)
at
com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
at
com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
at
com.cloudera.impala.catalog.CatalogObjectCache.get(CatalogObjectCache.java:226)
at com.cloudera.impala.catalog.Db.getTable(Db.java:189)
at com.cloudera.impala.catalog.Catalog$2.run(Catalog.java:513)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)


and here's the table scan of hbase meta table:

hbase(main):009:0* scan 'hbase:meta'
ROW
COLUMN+CELL
hbase:namespace,,1387263658257.16638622f2b9f25a825aab6867baaab5.
column=info:regioninfo, timestamp=1387263658870, value={ENCODED =>
16638622f2b9f25a825aab6867baaab5, NAME =>
'hbase:namespace,,1387263658257.16638622f2b9f25a825aab6867baaab5.',
STARTKEY
=> '
',
ENDKEY => ''}
hbase:namespace,,1387263658257.16638622f2b9f25a825aab6867baaab5.
column=info:seqnumDuringOpen, timestamp=1394470137408,
value=\x00\x00\x00\x00\x00\x00\x00\x05
hbase:namespace,,1387263658257.16638622f2b9f25a825aab6867baaab5.
column=info:server, timestamp=1394470137408,
value=dumbo010-cdh5.dumbo.allyes.com:60020
hbase:namespace,,1387263658257.16638622f2b9f25a825aab6867baaab5.
column=info:serverstartcode, timestamp=1394470137408,
value=1394470128013
hbasealltypessmall,,1394217597873.badd2977a76007d2b086a4ebc6cd85
column=info:regioninfo, timestamp=1394217598196, value={ENCODED =>
badd2977a76007d2b086a4ebc6cd858a, NAME =>
'hbasealltypessmall,,1394217597873.badd2977a76007d2b086a4ebc6cd858a.',
STARTKEY =
8a. > '',
ENDKEY => ''}
hbasealltypessmall,,1394217597873.badd2977a76007d2b086a4ebc6cd85
column=info:seqnumDuringOpen, timestamp=1394470137309,
value=\x00\x00\x00\x00\x00\x00\x00\x01
8a.
hbasealltypessmall,,1394217597873.badd2977a76007d2b086a4ebc6cd85
column=info:server, timestamp=1394470137309,
value=dumbo003-cdh5.dumbo.allyes.com:60020
8a.
hbasealltypessmall,,1394217597873.badd2977a76007d2b086a4ebc6cd85
column=info:serverstartcode, timestamp=1394470137309,
value=1394470127965
8a.
job_info,,1392968629874.25d3984c294f7cacb9d43b209d71e693.
column=info:regioninfo, timestamp=1392968630664, value={ENCODED =>
25d3984c294f7cacb9d43b209d71e693, NAME =>
'job_info,,1392968629874.25d3984c294f7cacb9d43b209d71e693.', STARTKEY
=> '',
ENDK
EY =>
''}
job_info,,1392968629874.25d3984c294f7cacb9d43b209d71e693.
column=info:seqnumDuringOpen, timestamp=1394470438617,
value=\x00\x00\x00\x00\x00\x00\x00;
job_info,,1392968629874.25d3984c294f7cacb9d43b209d71e693.
column=info:server, timestamp=1394470438617,
value=dumbo006-cdh5.dumbo.allyes.com:60020
job_info,,1392968629874.25d3984c294f7cacb9d43b209d71e693.
column=info:serverstartcode, timestamp=1394470438617,
value=1394470120098
job_info_record,,1392968665194.770fde6b862cf1adb82d4da2ed2ba37e.
column=info:regioninfo, timestamp=1392968665575, value={ENCODED =>
770fde6b862cf1adb82d4da2ed2ba37e, NAME =>
'job_info_record,,1392968665194.770fde6b862cf1adb82d4da2ed2ba37e.',
STARTKEY
=> '
',
ENDKEY => ''}
job_info_record,,1392968665194.770fde6b862cf1adb82d4da2ed2ba37e.
column=info:seqnumDuringOpen, timestamp=1394470137269,
value=\x00\x00\x00\x00\x00\x00\x00\x01
job_info_record,,1392968665194.770fde6b862cf1adb82d4da2ed2ba37e.
column=info:server, timestamp=1394470137269,
value=dumbo009-cdh5.dumbo.allyes.com:60020
job_info_record,,1392968665194.770fde6b862cf1adb82d4da2ed2ba37e.
column=info:serverstartcode, timestamp=1394470137269,
value=1394470127884
job_info_report,,1392968686718.33ba7ad3617955c8afa64cc48b1ace35.
column=info:regioninfo, timestamp=1392968687098, value={ENCODED =>
33ba7ad3617955c8afa64cc48b1ace35, NAME =>
'job_info_report,,1392968686718.33ba7ad3617955c8afa64cc48b1ace35.',
STARTKEY
=> '
',
ENDKEY => ''}
job_info_report,,1392968686718.33ba7ad3617955c8afa64cc48b1ace35.
column=info:seqnumDuringOpen, timestamp=1394470137295,
value=\x00\x00\x00\x00\x00\x00\x00\x01
job_info_report,,1392968686718.33ba7ad3617955c8afa64cc48b1ace35.
column=info:server, timestamp=1394470137295,
value=dumbo008-cdh5.dumbo.allyes.com:60020
job_info_report,,1392968686718.33ba7ad3617955c8afa64cc48b1ace35.
column=info:serverstartcode, timestamp=1394470137295,
value=1394470128046
ssp,,1394422999709.3b4a0a51b92d219ab3e42be693f1d001.
column=info:regioninfo, timestamp=1394422999998, value={ENCODED =>
3b4a0a51b92d219ab3e42be693f1d001, NAME =>
'ssp,,1394422999709.3b4a0a51b92d219ab3e42be693f1d001.', STARTKEY => '',
ENDKEY =>
''}
ssp,,1394422999709.3b4a0a51b92d219ab3e42be693f1d001.
column=info:seqnumDuringOpen, timestamp=1394470137489,
value=\x00\x00\x00\x00\x00\x00\x00E
ssp,,1394422999709.3b4a0a51b92d219ab3e42be693f1d001.
column=info:server, timestamp=1394470137489,
value=dumbo005-cdh5.dumbo.allyes.com:60020
ssp,,1394422999709.3b4a0a51b92d219ab3e42be693f1d001.
column=info:serverstartcode, timestamp=1394470137489,
value=1394470127865
6 row(s) in 0.1070 seconds


I also read somewhere else on a similar error message, and it was
suggesting to create the hbase table with region splits. Hope this
might be
helpful.


Thanks again for your help!
Percy



On Monday, March 10, 2014 10:09:04 AM UTC-7, lskuff wrote:

Please provide the impalad and catalogd log files.

Thanks,
Lenni

On Mon, Mar 10, 2014 at 10:00 AM, Percy Zhao wrote:

Hi Lenni,
Thanks much for the help. Yes I'm able to query this hbase table from
hive. I checked hbase logs, there are a few zk connection related
error and
I assume hbase recovered from them. Please let me know if there is
any other
information I can provide to help you troubleshoot.

Thanks,
Percy

On Monday, March 10, 2014 9:12:31 AM UTC-7, lskuff wrote:

Hi Percy,
This looks like it might be due an HBase issue. Can you check the
HBase
logs to verify HBase is running properly? It also would be good to
verify
whether you are able to query the table (not just describe) via Hive.

If you see errors in your HBase log you may want to move the
discussion
to cdh-...@cloudera.com.

Thanks,
Lenni
On Fri, Mar 7, 2014 at 2:16 PM, Percy Zhao wrote:

Environment:
Hadoop CDH5 Beta 2
Hive: 0.12.0-cdh5.0.0-beta-2
HBase: 0.96.1.1-cdh5.0.0-beta-2
Impala : v1.2.3-cdh5.0.0-beta-2 (8e266e0)

Follow the instruction on
http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_impala_hbase.html
:
1. create hbase table successfully
hbase(main):003:0> describe 'hbasealltypessmall'
'hbasealltypessmall', {NAME => 'bools', DATA_BLOCK_ENCODING =>
'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS =>
'1',
COMPRESSION => 'NONE',
MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS =>
'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
'true'},
{NAME => 'floats', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER =>
'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION =>
'NONE',
MIN_VERSIONS => '0',
TTL => '2147483647',KEEP_DELETED_CELLS => 'false', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME =>
'ints',
DATA_BLOCK_ENCODING => 'NONE',
BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1',
COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647',
KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY =>
'false',
BLOCKCACHE => 'true'}, {NAME => 'strings', DATA_BLOCK_ENCODING =>
'NONE',
BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1',
COMPRESSION
=> NONE', MIN_VERSIONS => '0', TTL => '2147483647',
KEEP_DELETED_CELLS =>
'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
'true'}
2. create hive table successfully
hive> desc extended hbasealltypessmall;
OK
id int from deserializer
bool_col boolean from deserializer
tinyint_col tinyint from deserializer
smallint_col smallint from deserializer
int_col int from deserializer
bigint_col bigint from deserializer
float_col float from deserializer
double_col double from deserializer
date_string_col string from deserializer
string_col string from deserializer
timestamp_col timestamp from deserializer

Detailed Table Information Table(tableName:hbasealltypessmall,
dbName:default, owner:hadoop, createTime:1394217917,
lastAccessTime:0,
retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:id,
type:int,
comment:null), FieldSchema(name:bool_col, type:boolean,
comment:null),
FieldSchema(name:tinyint_col, type:tinyint, comment:null),
FieldSchema(name:smallint_col, type:smallint, comment:null),
FieldSchema(name:int_col, type:int, comment:null),
FieldSchema(name:bigint_col, type:bigint, comment:null),
FieldSchema(name:float_col, type:float, comment:null),
FieldSchema(name:double_col, type:double, comment:null),
FieldSchema(name:date_string_col, type:string, comment:null),
FieldSchema(name:string_col, type:string, comment:null),
FieldSchema(name:timestamp_col, type:timestamp, comment:null)],
location:hdfs://cdh5/user/hive/warehouse/hbasealltypessmall,
inputFormat:org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat,
outputFormat:org.apache.hadoop.hive.ql.io.HivePassThroughOutputFormat,
compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null,
serializationLib:org.apache.hadoop.hive.hbase.HBaseSerDe,
parameters:{serialization.format=1,
hbase.columns.mapping=:key,bools:bool_col,ints:tinyint_col,ints:smallint_col,ints:int_col,ints:bigint_col,floats:float_col,floats:double_col,strings:date_string_col,strings:string_col,strings:timestamp_col}),
bucketCols:[], sortCols:[], parameters:{},
skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
skewedColValueLocationMaps:{}), storedAsSubDirectories:false),
partitionKeys:[], parameters:{EXTERNAL=TRUE,
hbase.table.name=hbasealltypessmall,
transient_lastDdlTime=1394217917,
storage_handler=org.apache.hadoop.hive.hbase.HBaseStorageHandler},
viewOriginalText:null, viewExpandedText:null,
tableType:EXTERNAL_TABLE)
Time taken: 5.197 seconds, Fetched: 13 row(s)

And running select statement against this table was successful as
well.

3. run impala shell, invalidate metadata. I can see the table but
when
I try to access this table, i got this following error. A select
statement
against this table generates the same error:
[dumbo002-cdh5:21000] > desc hbasealltypessmall;
Query: describe hbasealltypessmall
ERROR: AnalysisException: Failed to load metadata for table:
default.hbasealltypessmall
CAUSED BY: TableLoadingException: TableLoadingException: Failed to
load metadata for HBase table: hbasealltypessmall
CAUSED BY: IOException: HRegionInfo was null or empty in hbase:meta,
row=keyvalues=NONE


BTW, I was able to perform this integration in CDH5-beta1.

Thanks,
Percy

To unsubscribe from this group and stop receiving emails from it,
send
an email to impala-user...@cloudera.org.
To unsubscribe from this group and stop receiving emails from it, send
an email to impala-user...@cloudera.org.
To unsubscribe from this group and stop receiving emails from it, send an
email to impala-user+unsubscribe@cloudera.org.
To unsubscribe from this group and stop receiving emails from it, send an
email to impala-user+unsubscribe@cloudera.org.
To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 6 of 7 | next ›
Discussion Overview
groupimpala-user @
categorieshadoop
postedMar 10, '14 at 4:12p
activeMar 13, '14 at 6:26p
posts7
users4
websitecloudera.com
irc#hadoop

4 users in discussion

Percy Zhao: 3 posts Lenni Kuff: 2 posts Alan Choi: 1 post Marcel Kornacker: 1 post

Content

People

Support

Translate

site design / logo © 2022 Grokbase