FAQ

[HBase-issues] [jira] Commented: (HBASE-2342) Consider adding a watchdog node next to region server

ryan rawson (JIRA)
Mar 17, 2010 at 11:44 pm
[ https://issues.apache.org/jira/browse/HBASE-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846676#action_12846676 ]

ryan rawson commented on HBASE-2342:
------------------------------------

Run it as a parent process, thus ensuring via the OS level primitives that you get notification of a process death. You can also aggressively set the ZK timeout low (since it will be running in a low-GC pause process) and you can tune the detection of a Juliet pause differently. Maybe you would accept a 30 second GC pause, it might be better than killing the processing, ensuring data loss (pre HDFS-200) or otherwise causing unnecessary cluster churn.
Consider adding a watchdog node next to region server
-----------------------------------------------------

Key: HBASE-2342
URL: https://issues.apache.org/jira/browse/HBASE-2342
Project: Hadoop HBase
Issue Type: New Feature
Components: regionserver
Reporter: Todd Lipcon

This idea has been bandied about a fair amount. The concept is to add a second java process that runs next to each region server to act as a watchdog. Several possible purposes:
- monitor the RS for liveness - if it exhibits Juliet syndrome ("appears dead") then we kill it agressively to prevent it from coming back to life
- restart RS automatically in failure cases
- potentially move the entire ZK session to the watchdog to decouple node liveness from the particular JVM liveness
Let's discuss in this JIRA.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
reply

Search Discussions

6 responses

  • Todd Lipcon (JIRA) at Mar 17, 2010 at 11:50 pm
    [ https://issues.apache.org/jira/browse/HBASE-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846681#action_12846681 ]

    Todd Lipcon commented on HBASE-2342:
    ------------------------------------

    One question is whether _both_ nodes would be ZK clients, or just the watchdog? If only the watchdog, we'd have to communicate back and forth between them about any ZK stuff, which would be a big pain in my opinion.

    Another thought worth considering here is whether we could proactively do "rolling restarts" of region servers to avoid heap fragmentation in the first place. It's a bit of a pain since you'd end up with a cold cache, but if we could detect when the heap was getting fragmented and do a very fast RS restart, it's worth thinking about.
    Consider adding a watchdog node next to region server
    -----------------------------------------------------

    Key: HBASE-2342
    URL: https://issues.apache.org/jira/browse/HBASE-2342
    Project: Hadoop HBase
    Issue Type: New Feature
    Components: regionserver
    Reporter: Todd Lipcon

    This idea has been bandied about a fair amount. The concept is to add a second java process that runs next to each region server to act as a watchdog. Several possible purposes:
    - monitor the RS for liveness - if it exhibits Juliet syndrome ("appears dead") then we kill it agressively to prevent it from coming back to life
    - restart RS automatically in failure cases
    - potentially move the entire ZK session to the watchdog to decouple node liveness from the particular JVM liveness
    Let's discuss in this JIRA.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Todd Lipcon (JIRA) at Mar 18, 2010 at 6:42 am
    [ https://issues.apache.org/jira/browse/HBASE-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846772#action_12846772 ]

    Todd Lipcon commented on HBASE-2342:
    ------------------------------------

    Here's a crazy idea: JVM-TI (the JVM tool interface) allows you to hook events like GC start and GC stop. This could potentially be useful while watching the region server to understand whether the RS is "just sleeping" in GC or actually hosed in some way. It's also a pain in the butt because it requires native code, but worth thinking about.
    Consider adding a watchdog node next to region server
    -----------------------------------------------------

    Key: HBASE-2342
    URL: https://issues.apache.org/jira/browse/HBASE-2342
    Project: Hadoop HBase
    Issue Type: New Feature
    Components: regionserver
    Reporter: Todd Lipcon

    This idea has been bandied about a fair amount. The concept is to add a second java process that runs next to each region server to act as a watchdog. Several possible purposes:
    - monitor the RS for liveness - if it exhibits Juliet syndrome ("appears dead") then we kill it agressively to prevent it from coming back to life
    - restart RS automatically in failure cases
    - potentially move the entire ZK session to the watchdog to decouple node liveness from the particular JVM liveness
    Let's discuss in this JIRA.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Mar 20, 2010 at 3:35 am
    [ https://issues.apache.org/jira/browse/HBASE-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847704#action_12847704 ]

    stack commented on HBASE-2342:
    ------------------------------

    I like the ideas in here, in particular the interception of GC start/stop and proactive rolling restart.

    I agree that catching zk events in the babysitter then relaying them to the RS would be a horrorshow, especially where we are talking of all RS transitions going via zk.

    I wrote the hotspot gc list to see if there is a way of getting forewarning a full serial GC is imminent or if we can trigger it at preordained time. Lets see if I get anything back orther than consult with your vendor.

    What about the idea that was discussed a good while back of a lightweight zk heartbeater that lived outside of the RS in native space done up in native code? It'd be immune to GC pause. It would keep up the session with zk (only). It'd kill the RS if it lost its zk session -- perhaps its parent process on the RS -- and whereas the RS with its heavy-weight zk connection, once it'd revived, it'd just renew its session or, just have timeouts that were hard to expire and carry on from where it'd left off.

    While the above might run into long Juliet pauses, we'd log them. GC tuning would help mitigate, at least some (though it seems occasional full serial GC is unavoidlable).

    Consider adding a watchdog node next to region server
    -----------------------------------------------------

    Key: HBASE-2342
    URL: https://issues.apache.org/jira/browse/HBASE-2342
    Project: Hadoop HBase
    Issue Type: New Feature
    Components: regionserver
    Reporter: Todd Lipcon

    This idea has been bandied about a fair amount. The concept is to add a second java process that runs next to each region server to act as a watchdog. Several possible purposes:
    - monitor the RS for liveness - if it exhibits Juliet syndrome ("appears dead") then we kill it agressively to prevent it from coming back to life
    - restart RS automatically in failure cases
    - potentially move the entire ZK session to the watchdog to decouple node liveness from the particular JVM liveness
    Let's discuss in this JIRA.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Mar 20, 2010 at 3:45 am
    [ https://issues.apache.org/jira/browse/HBASE-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847705#action_12847705 ]

    stack commented on HBASE-2342:
    ------------------------------

    On the possibility of scheduling an explicit defrag of the tenured generation:

    {code}
    from Jon Masamitsu <Jon.Masamitsu@sun.com>
    cc hotspot-gc-use@openjdk.java.net
    date Fri, Mar 19, 2010 at 8:43 PM
    subject Re: Using CMS, any chance of forewarning a serial full GC is imminent?

    System.gc() will cause CMS to do a full collection. Does that
    help? As long as you have not turned off explicit GC's nor
    set flags so that CMS does System.gc() concurrently, you can
    compact the tenured (CMS) gen that way.
    {code}
    Consider adding a watchdog node next to region server
    -----------------------------------------------------

    Key: HBASE-2342
    URL: https://issues.apache.org/jira/browse/HBASE-2342
    Project: Hadoop HBase
    Issue Type: New Feature
    Components: regionserver
    Reporter: Todd Lipcon

    This idea has been bandied about a fair amount. The concept is to add a second java process that runs next to each region server to act as a watchdog. Several possible purposes:
    - monitor the RS for liveness - if it exhibits Juliet syndrome ("appears dead") then we kill it agressively to prevent it from coming back to life
    - restart RS automatically in failure cases
    - potentially move the entire ZK session to the watchdog to decouple node liveness from the particular JVM liveness
    Let's discuss in this JIRA.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Mar 20, 2010 at 7:13 pm
    [ https://issues.apache.org/jira/browse/HBASE-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847772#action_12847772 ]

    stack commented on HBASE-2342:
    ------------------------------

    More:

    {code}
    Jon Masamitsu <Jon.Masamitsu@sun.com>
    date Fri, Mar 19, 2010 at 10:11 PM
    subject Re: Using CMS, any chance of forewarning a serial full GC is imminent?

    So it's a fragmentation issue. If you haven't seen

    http://blogs.sun.com/jonthecollector/entry/what_the_heck_s_a

    it might help. Also try CMS in 6u20. It has some improvements for
    reducing fragmentation.
    {code}


    Consider adding a watchdog node next to region server
    -----------------------------------------------------

    Key: HBASE-2342
    URL: https://issues.apache.org/jira/browse/HBASE-2342
    Project: Hadoop HBase
    Issue Type: New Feature
    Components: regionserver
    Reporter: Todd Lipcon

    This idea has been bandied about a fair amount. The concept is to add a second java process that runs next to each region server to act as a watchdog. Several possible purposes:
    - monitor the RS for liveness - if it exhibits Juliet syndrome ("appears dead") then we kill it agressively to prevent it from coming back to life
    - restart RS automatically in failure cases
    - potentially move the entire ZK session to the watchdog to decouple node liveness from the particular JVM liveness
    Let's discuss in this JIRA.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Mar 23, 2010 at 6:14 am
    [ https://issues.apache.org/jira/browse/HBASE-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848559#action_12848559 ]

    stack commented on HBASE-2342:
    ------------------------------

    Another reponse on the thread talks of difficulty signalling imminent stop-the-world full GC: http://permalink.gmane.org/gmane.comp.java.openjdk.hotspot.gc.devel/1680
    Consider adding a watchdog node next to region server
    -----------------------------------------------------

    Key: HBASE-2342
    URL: https://issues.apache.org/jira/browse/HBASE-2342
    Project: Hadoop HBase
    Issue Type: New Feature
    Components: regionserver
    Reporter: Todd Lipcon

    This idea has been bandied about a fair amount. The concept is to add a second java process that runs next to each region server to act as a watchdog. Several possible purposes:
    - monitor the RS for liveness - if it exhibits Juliet syndrome ("appears dead") then we kill it agressively to prevent it from coming back to life
    - restart RS automatically in failure cases
    - potentially move the entire ZK session to the watchdog to decouple node liveness from the particular JVM liveness
    Let's discuss in this JIRA.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post

1 user in discussion

stack (JIRA): 7 posts