FAQ
Hi

We are having a issue while running the job as it hangs forever at 99%
(reducer) inserting into the table , hbase hbck -details finds
inconsistency in the table I am trying to insert data as :--



Chain of regions in table NAM_CLUSTERKEYS3 is broken; edges does not
contain
00000000|00000000000000000USA|0000240b|000000000000000LAKE,|002ff027|000
0000000CONSULTING|006e8f90|000000000000000000NY:0

I tried hbase hbck -fix but does not help.



Tried scanning the '.META' table



scan
'.META.',{STARTROW=>'NAM_CLUSTERKEYS3,00000000|00000000000000000USA|000
240b|000000000000000LAKE',LIMIT=>1}



Getting this error:--





ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying
to contact region server doop6. for region .META.,,1, row
'NAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|000000000000000
LAKE', but failed after 7 attempts.

Exceptions:

java.io.IOException: java.io.IOException:
java.lang.IllegalArgumentException: No 44 in
<WKNAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|0000000000000
00LAK????????>, length=58, offset=27

at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE
(HRegionServer.java:992)

at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE
(HRegionServer.java:981)

at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionSe
rver.java:1783)

at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)

at java.lang.reflect.Method.invoke(Method.java:616)

at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)

at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)

Caused by: java.lang.IllegalArgumentException: No 44 in
<WKNAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|0000000000000
00LAK????????>, length=58, offset=27

at
org.apache.hadoop.hbase.KeyValue.getRequiredDelimiterInReverse(KeyValue.
java:1281)

at
org.apache.hadoop.hbase.KeyValue$MetaKeyComparator.compareRows(KeyValue.
java:1827)

at
org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:186
6)

at
org.apache.hadoop.hbase.util.Bytes.binarySearch(Bytes.java:1159)

at
org.apache.hadoop.hbase.io.hfile.HFile$BlockIndex.blockContainingKey(HFi
le.java:1618)

at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.blockContainingKey(HFile.j
ava:918)

at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:
1296)

at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(Stor
eFileScanner.java:136)

at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScan
ner.java:96)

at
org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.ja
va:77)

at
org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1341)

at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegio
n.java:2269)

at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(
HRegion.java:1126)

at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:111
8)

at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:110
2)

at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionSe
rver.java:1781)

... 5 more



java.io.IOException: java.io.IOException:
java.lang.IllegalArgumentException: No 44 in
<WKNAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|0000000000000
00LAK????????>, length=58, offset=27

at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE
(HRegionServer.java:992)

at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE
(HRegionServer.java:981)

at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionSe
rver.java:1783)

at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)

at java.lang.reflect.Method.invoke(Method.java:616)

at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)

at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)

Caused by: java.lang.IllegalArgumentException: No 44 in
<WKNAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|0000000000000
00LAK????????>, length=58, offset=27

at
org.apache.hadoop.hbase.KeyValue.getRequiredDelimiterInReverse(KeyValue.
java:1281)

at
org.apache.hadoop.hbase.KeyValue$MetaKeyComparator.compareRows(KeyValue.
java:1827)

at
org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:186
6)

at
org.apache.hadoop.hbase.util.Bytes.binarySearch(Bytes.java:1159)

at
org.apache.hadoop.hbase.io.hfile.HFile$BlockIndex.blockContainingKey(HFi
le.java:1618)

at org.apache.hadoop.hbase.io.hfile.HFile$Reader.blockContaini



How can I fix the problem in the table . Any help would be appreciated.

Search Discussions

  • Stack at Aug 13, 2011 at 5:47 am

    On Fri, Aug 12, 2011 at 10:48 AM, Rohit Nigam wrote:
    Tried scanning  the '.META' table
    scan
    '.META.',{STARTROW=>'NAM_CLUSTERKEYS3,00000000|00000000000000000USA|000
    240b|000000000000000LAKE',LIMIT=>1}



    Getting this error:--
    I think you need to add two commas to the end of the key when scanning
    .META. to avoid the IllegalArgumentException below.

    How can I fix the problem in the table . Any help would be appreciated.
    See if you can scan the region as you are doing above. What error do
    you get back or are you able to scan?

    Next see if an entry for it in .META. (Look at content of meta by
    doing '> echo "scan '.META.'" | ./bin/hbase shell &> /tmp/meta.txt'
    then review /tmp/meta.txt.

    Get this regions encoded name from the above /tmp/meta.txt. See if
    its in the filesystem. Grep its encoded name in the master. See if
    you can tell story on what happened with this region.

    St.Ack

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshbase, hadoop
postedAug 12, '11 at 10:51p
activeAug 13, '11 at 5:47a
posts2
users2
websitehbase.apache.org

2 users in discussion

Rohit Nigam: 1 post Stack: 1 post

People

Translate

site design / logo © 2022 Grokbase