FAQ

[HBase-user] How to make HBase redundant?

Andrey Timerbaev
Sep 26, 2010 at 9:22 am
Dear All,

In my Hadoop 0.20.1/HBase 0.20.3 cluster I’ve got 2 region servers. On each
region server, Hadoop DataNode and HBase RegionServer processes are running.
Hadoop is set up redundantly (it writes each data to both DataNodes, i.e. writes
data with replication factor 2). But as per information from the HBase Web UI,
HBase creates on the region servers unique regions. E.g. tables A, B are created
on region server 1, and tables C, D are created on the region server 2. Next, I
noticed, when region server 1 is not available, the tables A and B are not
available as well (cannot be read/written to).

Could you kindly suggest, how to make HBase redundant, so that all tables are
available even if one of region servers fails/dies?
Have I missed a Hadoop/HBase setting to do so?

Thank you,
Andrey Timerbaev
reply

Search Discussions

10 responses

  • Stanislaw Kogut at Sep 26, 2010 at 9:53 am

    On Sun, Sep 26, 2010 at 12:21 PM, Andrey Timerbaev wrote:
    Dear All,

    In my Hadoop 0.20.1/HBase 0.20.3 cluster I’ve got 2 region servers. On each
    region server, Hadoop DataNode and HBase RegionServer processes are running.
    Hadoop is set up redundantly (it writes each data to both DataNodes, i.e. writes
    data with replication factor 2). But as per information from the HBase Web UI,
    HBase creates on the region servers unique regions. E.g. tables A, B are created
    on region server 1, and tables C, D are created on the region server 2. Next, I
    noticed, when region server 1 is not available, the tables A and B are not
    available as well (cannot be read/written to).

    Could you kindly suggest, how to make HBase redundant, so that all tables are
    available even if one of region servers fails/dies?
    Have I missed a Hadoop/HBase setting to do so?

    Thank you,
    Andrey Timerbaev

    In case when one of regionservers fail, HBase Master will reassign all
    regions, so, any regions served by failed regionserver will be
    assigned to operating regionservers. It is possible because of
    replication on datanodes, regionserver heartbeats and
    Write-Ahead-Logs. HBase master can be made redundant too.

    --
    Regards,
    Stanislaw Kogut
    Sistyma LLC
  • Andrey Timerbaev at Sep 27, 2010 at 6:24 am

    Stanislaw Kogut writes:

    In case when one of regionservers fail, HBase Master will reassign all
    regions, so, any regions served by failed regionserver will be
    assigned to operating regionservers. It is possible because of
    replication on datanodes, regionserver heartbeats and
    Write-Ahead-Logs. HBase master can be made redundant too.
    Hello Stanislaw,

    This is a good news. Is a particular setting needed to do HBase behave in the
    redundant manner, or is it an out-of-box behavior? (In my installation I didn't
    notice namely, that HBase reassigns the regions.)

    Thank you,
    Andrey
  • Matthew LeMieux at Sep 27, 2010 at 6:53 am
    It is the default behavior. However, in my installation it has never succeeded. So, the regions never actually get re-assigned. The result is a cluster in a funky state that needs to be completely brought down and then started up again in order to recover.

    I'm running on a version of HBase that is several weeks old, but I've never seen the cluster recover from a region server going on previous versions either. I'm hoping this will be fixed soon.

    You should see the Master making the attempt at the same time the region server goes down in the logs. In my case, it complains about this: "java.io.IOException: cannot get log writer" caused by this "java.io.FileNotFoundException: Parent path is not a directory".

    FYI,

    Matthew

    On Sep 26, 2010, at 11:23 PM, Andrey Timerbaev wrote:

    Stanislaw Kogut <skogut@...> writes:
    In case when one of regionservers fail, HBase Master will reassign all
    regions, so, any regions served by failed regionserver will be
    assigned to operating regionservers. It is possible because of
    replication on datanodes, regionserver heartbeats and
    Write-Ahead-Logs. HBase master can be made redundant too.
    Hello Stanislaw,

    This is a good news. Is a particular setting needed to do HBase behave in the
    redundant manner, or is it an out-of-box behavior? (In my installation I didn't
    notice namely, that HBase reassigns the regions.)

    Thank you,
    Andrey


  • Andrey Timerbaev at Sep 27, 2010 at 9:15 am

    Matthew LeMieux writes:

    It is the default behavior. However, in my installation it has never
    succeeded. So, the regions never
    actually get re-assigned. The result is a cluster in a funky state that
    needs to be completely brought down
    and then started up again in order to recover.

    I'm running on a version of HBase that is several weeks old, but I've never
    seen the cluster recover from a region server going on previous versions
    either.
    That's funny. My understanding was, region servers were redundant inherently. If
    they are "semiredundant", there should be a root cause like some wrong settings
    or a bug.

    Could someone from HBase experts comment on this?
  • Jean-Daniel Cryans at Sep 27, 2010 at 6:40 pm

    That's funny. My understanding was, region servers were redundant inherently. If
    they are "semiredundant", there should be a root cause like some wrong settings
    or a bug.

    Could someone from HBase experts comment on this?
    0.89 is a developer release, it should be treated as such (eg do
    expect bugs) and this is the version used by Matthew. A newer release
    candidate was posted here:
    http://people.apache.org/~jdcryans/hbase-0.89.20100924-candidate-1/
    and this is the version we're using in production (and on a few other
    clusters) at StumbleUpon. We can kill -9 region servers as much as we
    want, and the cluster does recover. Previously there was an issue with
    empty log files that's now fixed, also 0.89.2010830 introduced a
    changed that's incompatible with the old way of recovering edits from
    a failed region server, so if leftovers from a previous split are
    present this can prevent the master from splitting logs at all (which
    is another issue Matthew got in a separate thread).

    J-D
  • Andrey Timerbaev at Sep 28, 2010 at 6:30 am

    Jean-Daniel Cryans writes:

    0.89 is a developer release, it should be treated as such (eg do
    expect bugs) and this is the version used by Matthew. A newer release
    candidate was posted here:
    http://people.apache.org/~jdcryans/hbase-0.89.20100924-candidate-1/
    and this is the version we're using in production (and on a few other
    clusters) at StumbleUpon. We can kill -9 region servers as much as we
    want, and the cluster does recover.
    Hello Jean-Daniel,

    Does the 0.20.6 provide the redundancy of region servers as well as the
    0.89.20100924?

    Thank you,
    Andrey
  • Jean-Daniel Cryans at Sep 28, 2010 at 4:29 pm

    Does the 0.20.6 provide the redundancy of region servers as well as the
    0.89.20100924?
    It generally does, but since HDFS 0.20 didn't support fsSync then
    testing region server recovery is pretty hard (since you never know
    how much data you lost). But for 0.90, which the 0.89 releases are
    snapshots of, the goal is no data loss and 100% recovery. Bugs that
    are found in that regard always have high priority.

    J-D
  • Andrey Timerbaev at Sep 28, 2010 at 2:22 pm

    Jean-Daniel Cryans writes:

    We can kill -9 region servers as much as we
    want, and the cluster does recover.

    J-D
    Just FYI: I've just tested the redundancy of region servers on a Hadoop
    0.20.2/HBase 0.20.6, with 2 region servers. The results are: it works fine, only
    if the region to be killed doesn't contain ROOT and META regions. In the
    opposite case, i.e. if the killed region contained the ROOT, cluster is not
    operable after killing it
    (org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0). Neither after
    the killed region server has been brought up again. Both reads and writes failed.

    As result, per my tests, the region servers are not redundant in 0.20.6.

    Andrey
  • Matthew LeMieux at Sep 29, 2010 at 10:49 am
    The problem I referred to was also being addressed in a separate thread, thanks to the contributors to the mailing list and mostly to J-D and Stack.

    I have recently upgraded to the 0.89.20100924 version and after more than 24 hours am very happy with the results. I think I must have missed the previous announcement of this particular version being available. (which list should I join to get such announcements?).

    In the end, I would like to thank them for their timely, relevant and ingenious efforts in response to the posts on the mailing list.

    -Matthew


    O n Sep 27, 2010, at 11:40 AM, Jean-Daniel Cryans wrote:
    That's funny. My understanding was, region servers were redundant inherently. If
    they are "semiredundant", there should be a root cause like some wrong settings
    or a bug.

    Could someone from HBase experts comment on this?
    0.89 is a developer release, it should be treated as such (eg do
    expect bugs) and this is the version used by Matthew. A newer release
    candidate was posted here:
    http://people.apache.org/~jdcryans/hbase-0.89.20100924-candidate-1/
    and this is the version we're using in production (and on a few other
    clusters) at StumbleUpon. We can kill -9 region servers as much as we
    want, and the cluster does recover. Previously there was an issue with
    empty log files that's now fixed, also 0.89.2010830 introduced a
    changed that's incompatible with the old way of recovering edits from
    a failed region server, so if leftovers from a previous split are
    present this can prevent the master from splitting logs at all (which
    is another issue Matthew got in a separate thread).

    J-D
  • Jean-Daniel Cryans at Sep 29, 2010 at 4:16 pm

    On Wed, Sep 29, 2010 at 3:49 AM, Matthew LeMieux wrote:
    The problem I referred to was also being addressed in a separate thread, thanks to the contributors to the mailing list and mostly to J-D and Stack.

    I have recently upgraded to the 0.89.20100924 version  and after more than 24 hours am very happy with the results.  I think I must have missed the previous announcement of this particular version being available.   (which list should I join to get such announcements?).
    The announcements for the release candidates are made on dev@, once
    the vote passes it will be announced on user@
    In the end, I would like to thank them for their timely, relevant and ingenious efforts in response to the posts on the mailing list.
    Trying our best :)

    J-D

Related Discussions

Discussion Navigation
viewthread | post