FAQ
Hi!

We are running a small hadoop/hbase cluster on two machines, master and
slave. Versions are 0.19. Both machines are running hadoop datanodes and
hbase region servers. The master machine is running the hbase master
node and the hadoop name node. We have seven tables in hbase.

Now, to see how fail safe this setup is I tried rebooting the slave
machine. After the reboot the region server on the slave machine was
started manually by running bin/start-hbase.sh on the master.

It started successfully but when I list tables in the hbase shell all
seven tables are gone.

$ bin/hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Version: 0.19.0, r735381, Sun Jan 18 14:29:34 PST 2009
hbase(main):001:0> list
0 row(s) in 0.1380 seconds

This may be relevant parts of the log on the slave region server right
after it has been restarted:

2009-02-27 09:48:02,978 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
2009-02-27 09:48:02,979 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 6 on 60020: starting
2009-02-27 09:48:02,981 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)


Files in hadoop seem to be intact. I have tried restarting hadoop and
hbase, but the tables still don't magically reappear as I was hoping
for. What can I do to debug this and hopefully recover my data?

There is a historian directory under /hbase/.META. that contains a data
and an index file, can I try to recover my tables using these somehow?

Thanks for any help,
Ludvig Omholt

Search Discussions

  • Stack at Feb 27, 2009 at 3:44 pm
    What did you have configured as the datanode data directory? By default it
    writes /tmp. On reboot /tmp is cleared.

    Files in hadoop seem to be intact. I have tried restarting hadoop and
    hbase, but the tables still don't magically reappear as I was hoping
    for. What can I do to debug this and hopefully recover my data?
    Can you list the content?

    ./bin/hadoop -fs lsr /hbase

    Or, did you create the tables in quick succession and before the catalog
    tables had a chance to persist -- dump the inmemory edits to the filesystem
    -- you restarted? HBase can lose data because there is no working flush in
    hdfs currently. Because of this, the commit-log mechanism is ineffective
    (Hopefully addressed in hdfs 0.20.0).

    St.Ack
  • Ludvig Omholt at Feb 27, 2009 at 3:58 pm

    stack wrote:
    What did you have configured as the datanode data directory? By default it
    writes /tmp. On reboot /tmp is cleared.
    Datanodes have two data directories (on separate disks):

    $ grep -B1 -A2 data.dir conf/hadoop-site.xml
    <property>
    <name>dfs.data.dir</name>
    <value>/var/lib/hadoop/data1,/var/lib/hadoop/data2</value>
    </property>
    Files in hadoop seem to be intact. I have tried restarting hadoop and
    hbase, but the tables still don't magically reappear as I was hoping
    for. What can I do to debug this and hopefully recover my data?
    Can you list the content?

    ./bin/hadoop -fs lsr /hbase
    Yes. Data from all seven tables seem to still be there.

    $ bin/hadoop fs -dus /hbase
    hdfs://bright:9000/hbase 2389548908

    $ bin/hadoop fs -lsr /hbase/.META. /hbase/-ROOT-
    drwxr-xr-x - hbase supergroup 0 2009-02-27 10:17 /hbase/.META./1028785192
    drwxr-xr-x - hbase supergroup 0 2009-02-27 10:17 /hbase/.META./1028785192/historian
    drwxr-xr-x - hbase supergroup 0 2009-02-27 10:17 /hbase/.META./1028785192/historian/info
    -rw-r--r-- 2 hbase supergroup 9 2009-02-27 10:17 /hbase/.META./1028785192/historian/info/550003861967926036
    drwxr-xr-x - hbase supergroup 0 2009-02-27 10:17 /hbase/.META./1028785192/historian/mapfiles
    drwxr-xr-x - hbase supergroup 0 2009-02-27 10:17 /hbase/.META./1028785192/historian/mapfiles/550003861967926036
    -rw-r--r-- 2 hbase supergroup 3420 2009-02-27 10:17 /hbase/.META./1028785192/historian/mapfiles/550003861967926036/data
    -rw-r--r-- 2 hbase supergroup 251 2009-02-27 10:17 /hbase/.META./1028785192/historian/mapfiles/550003861967926036/index
    drwxr-xr-x - hbase supergroup 0 2009-02-27 10:17 /hbase/.META./1028785192/info
    drwxr-xr-x - hbase supergroup 0 2009-02-27 10:17 /hbase/.META./1028785192/info/info
    -rw-r--r-- 2 hbase supergroup 9 2009-02-27 10:17 /hbase/.META./1028785192/info/info/2113823753877423194
    drwxr-xr-x - hbase supergroup 0 2009-02-27 10:17 /hbase/.META./1028785192/info/mapfiles
    drwxr-xr-x - hbase supergroup 0 2009-02-27 10:17 /hbase/.META./1028785192/info/mapfiles/2113823753877423194
    -rw-r--r-- 2 hbase supergroup 3046 2009-02-27 10:17 /hbase/.META./1028785192/info/mapfiles/2113823753877423194/data
    -rw-r--r-- 2 hbase supergroup 242 2009-02-27 10:17 /hbase/.META./1028785192/info/mapfiles/2113823753877423194/index
    drwxr-xr-x - hbase supergroup 0 2009-02-27 11:16 /hbase/-ROOT-/70236052
    drwxr-xr-x - hbase supergroup 0 2009-02-27 11:16 /hbase/-ROOT-/70236052/info
    drwxr-xr-x - hbase supergroup 0 2009-02-27 15:12 /hbase/-ROOT-/70236052/info/info
    -rw-r--r-- 2 hbase supergroup 9 2009-02-27 15:12 /hbase/-ROOT-/70236052/info/info/293531972055088648
    -rw-r--r-- 2 hbase supergroup 11 2009-02-27 10:25 /hbase/-ROOT-/70236052/info/info/5005305484623244108
    drwxr-xr-x - hbase supergroup 0 2009-02-27 15:12 /hbase/-ROOT-/70236052/info/mapfiles
    drwxr-xr-x - hbase supergroup 0 2009-02-27 15:12 /hbase/-ROOT-/70236052/info/mapfiles/293531972055088648
    -rw-r--r-- 2 hbase supergroup 350 2009-02-27 15:12 /hbase/-ROOT-/70236052/info/mapfiles/293531972055088648/data
    -rw-r--r-- 2 hbase supergroup 230 2009-02-27 15:12 /hbase/-ROOT-/70236052/info/mapfiles/293531972055088648/index
    drwxr-xr-x - hbase supergroup 0 2009-02-27 10:25 /hbase/-ROOT-/70236052/info/mapfiles/5005305484623244108
    -rw-r--r-- 2 hbase supergroup 937 2009-02-27 10:25 /hbase/-ROOT-/70236052/info/mapfiles/5005305484623244108/data
    -rw-r--r-- 2 hbase supergroup 232 2009-02-27 10:25 /hbase/-ROOT-/70236052/info/mapfiles/5005305484623244108/index
    drwxr-xr-x - hbase supergroup 0 2009-02-27 15:12 /hbase/-ROOT-/compaction.dir
    Or, did you create the tables in quick succession and before the catalog
    tables had a chance to persist -- dump the inmemory edits to the filesystem
    -- you restarted? HBase can lose data because there is no working flush in
    hdfs currently. Because of this, the commit-log mechanism is ineffective
    (Hopefully addressed in hdfs 0.20.0).
    I don't think this is the case, data has been written over a couple of
    days, so there has been lots of time for persisting it.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshbase, hadoop
postedFeb 27, '09 at 2:12p
activeFeb 27, '09 at 3:58p
posts3
users2
websitehbase.apache.org

2 users in discussion

Ludvig Omholt: 2 posts Stack: 1 post

People

Translate

site design / logo © 2022 Grokbase