You can affect the current situation with a little development
effort, as we (privately) have at my employer. Comments inline
From: Cosmin Lehene
My answers inline...
On 1/7/09 12:05 PM, "Genady" wrote:
Could somebody explain an expected HBase 0.18.1 nodes
behavior in case Hadoop cluster failover for a following
- HBase master region server fails
You need to manually set a different machine as master and
redistribute the configuration files on all other region
servers and restart the cluster.
Maybe someone in the development team could explain if this
will change with Zookeeper integration.
First, my understanding is that the ZK integration will handle
master role reassignment without requiring a cluster restart.
J-D could say more (or deny).
What we (my employer) do currently is run the Hadoop and HBase
daemons as child processes of custom monitoring daemons that
use a private DHT which supports TTLs on cells to write
heartbeats into the cloud. This same mechanism also supports
service discovery. All HBase processes in particular can be
automatically restarted should the location of the master shift.
(The location of the master may shift if a node has a hard
failure.) We write Hadoop and HBase configuration files on the
fly as necessary.
This all took me only a few days to implement and a few more
Relocation of the Hadoop name node is trickier. I believe it
is possible to have it write the fs image out to a NFS share
such that a service relocation to another host with the same
NFS mount will pick up the latest edits seamlessly. However I
do not trust NFS under a number of failure conditions so will
not try this myself. There might be other better strategies
for replication of the fs image.
- HBase slave region server fails
This is handled transparently. Regions served by the failed
region server are reassigned to the rest of the region
- Hadoop master server fails
I suppose you mean the HDFS namenode. Currently, the
namenode is a single point of failure in HDFS and needs
manual intervention to configure a new namenode. A
secondary namenode can be configured, however this one only
keeps a metadata replica and does not act as failover node.http://wiki.apache.org/hadoop/NameNode
See my comments above.
- Hadoop slave server fails
If a HDFS datanode fails it's files are already replicated
on 2 other datanodes. Eventually the replication will be
fixed by the namenode - creating third replica on one of
You want to make sure your client is requesting the default
replication. The stock Hadoop config does allow DFS clients
to specify a replication factor of 1 only. However the HBase
DFS client will always request the default so this is not
an issue for HBase.
- Hadoop master and HBase master are fail (
in case they're installed on the same computer and for
instance disk has failover)
These servers run independently so you can see above what
Don't run them on the same node regardless. The Hadoop name
node can become very busy given a lot of DFS file system
activity. Let it have its own dedicated node to avoid problems
e.g. replication stalls.
- HBase slave region server is failed but
HBase data could be recovered and copied to other node
and the new node is added instead of failed one.
Hbase region servers don't actually hold the data. Data
is stored in HDFS. Region servers just serve regions and
when the region server fails the regions are reassigned