Hi!
We are running a small hadoop/hbase cluster on two machines, master and
slave. Versions are 0.19. Both machines are running hadoop datanodes and
hbase region servers. The master machine is running the hbase master
node and the hadoop name node. We have seven tables in hbase.
Now, to see how fail safe this setup is I tried rebooting the slave
machine. After the reboot the region server on the slave machine was
started manually by running bin/start-hbase.sh on the master.
It started successfully but when I list tables in the hbase shell all
seven tables are gone.
$ bin/hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Version: 0.19.0, r735381, Sun Jan 18 14:29:34 PST 2009
hbase(main):001:0> list
0 row(s) in 0.1380 seconds
This may be relevant parts of the log on the slave region server right
after it has been restarted:
2009-02-27 09:48:02,978 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
2009-02-27 09:48:02,979 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 6 on 60020: starting
2009-02-27 09:48:02,981 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
Files in hadoop seem to be intact. I have tried restarting hadoop and
hbase, but the tables still don't magically reappear as I was hoping
for. What can I do to debug this and hopefully recover my data?
There is a historian directory under /hbase/.META. that contains a data
and an index file, can I try to recover my tables using these somehow?
Thanks for any help,
Ludvig Omholt
