FAQ
Hello,

I have an HBase 0.90.3 fleet with more than 100 hosts distributed
across 3 datacenters. Yesterday, we experienced some network issues
for several minutes and the region servers from one data center lost
the connectivity with the namenode. They started the shutdown sequence
but about 20 hosts were unable to complete it successfully. This is
bad for us because we have to restart them manually.

I checked the logs and it looks like the embedded Jetty does not close
all the threads. Here is a sample from the logs.

[FATAL] (IPC Server handler 5 on 60020)
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
server serverName=pa-hbase-datanode-na-7031,60020,1316498497389,
load=(requests=19902, regions=32, usedHeap=582, maxHeap=1820): File
System not available { java.io.IOException: File system is not
available …..
[INFO] (IPC Server handler 5 on 60020)
org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60020:
exiting
[WARN] (regionserver60020) org.mortbay.log: 1 threads could not be stopped
[INFO] (regionserver60020)
org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020
exiting

This is a sample from the stack trace dump on that host. There is one
Jetty thread in runnable state.

Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode):
"1692369893@qtp-1688716382-1 - Acceptor0
SelectChannelConnector@0.0.0.0:60030" prio=10 tid=0x00002aaab064e000
nid=0x2841 runnable [0x0000000042cf8000]
java.lang.Thread.State: RUNNABLE
at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:724)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Is my assumption correct? Should I open a JIRA?


Regards,
Bogdan

Search Discussions

  • Bogdan Ghidireac at Sep 21, 2011 at 8:13 am
    Hello,

    I have an HBase 0.90.3 fleet with more than 100 hosts distributed
    across 3 datacenters. Yesterday, we experienced some network issues
    for several minutes and the region servers from one data center lost
    the connectivity with the namenode. They started the shutdown sequence
    but about 20 hosts were unable to complete it successfully. This is
    bad for us because we have to restart them manually.

    I checked the logs and it looks like the embedded Jetty does not close
    all the threads. Here is a sample from the logs.

    [FATAL] (IPC Server handler 5 on 60020)
    org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
    server serverName=pa-hbase-datanode-na-7031,60020,1316498497389,
    load=(requests=19902, regions=32, usedHeap=582, maxHeap=1820): File
    System not available { java.io.IOException: File system is not
    available …..
    [INFO] (IPC Server handler 5 on 60020)
    org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60020:
    exiting
    [WARN] (regionserver60020) org.mortbay.log: 1 threads could not be stopped
    [INFO] (regionserver60020)
    org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020
    exiting

    This is a sample from the stack trace dump on that host. There is one
    Jetty thread in runnable state.

    Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed
    mode): "1692369893@qtp-1688716382-1 - Acceptor0
    SelectChannelConnector@0.0.0.0:60030" prio=10 tid=0x00002aaab064e000
    nid=0x2841 runnable [0x0000000042cf8000]
    java.lang.Thread.State: RUNNABLE
    at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:724)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

    Is my assumption correct? Should I open a JIRA?


    Regards,
    Bogdan
  • Stack at Sep 21, 2011 at 4:06 pm

    On Wed, Sep 21, 2011 at 1:07 AM, Bogdan Ghidireac wrote:
    I have an HBase 0.90.3 fleet with more than 100 hosts distributed
    across 3 datacenters.
    Yesterday, we experienced some network issues
    for several minutes and the region servers from one data center lost
    the connectivity with the namenode. They started the shutdown sequence
    but about 20 hosts were unable to complete it successfully. This is
    bad for us because we have to restart them manually.
    HBase and HDFS are not made to span datacenters.
    Is my assumption correct? Should I open a JIRA?
    You could paste the thread dump but before doing anything, I'd suggest
    you change your layout so cluster runs inside a single datacenter.

    St.Ack
  • Bogdan Ghidireac at Sep 21, 2011 at 5:51 pm
    HBase and HDFS are not made to span datacenters.
    Yes, I know. I think the correct terminology is 3 availability zones
    within the same region (ping between them is about 1ms)
    You could paste the thread dump
    This is the full stacktrace:

    Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode):

    "LruBlockCache.EvictionThread" daemon prio=10 tid=0x0000000052cc8000
    nid=0x2859 in Object.wait() [0x0000000044510000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x0000000781b79cf0> (a
    org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
    at java.lang.Object.wait(Object.java:485)
    at org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:519)
    - locked <0x0000000781b79cf0> (a
    org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)

    "1692369893@qtp-1688716382-1 - Acceptor0
    SelectChannelConnector@0.0.0.0:60030" prio=10 tid=0x00002aaab064e000
    nid=0x2841 runnable [0x0000000042cf8000]
    java.lang.Thread.State: RUNNABLE
    at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:724)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

    "DestroyJavaVM" prio=10 tid=0x00000000520e2000 nid=0x2808 waiting on
    condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

    "Low Memory Detector" daemon prio=10 tid=0x000000005216f800 nid=0x2811
    runnable [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

    "C2 CompilerThread1" daemon prio=10 tid=0x000000005216d800 nid=0x2810
    waiting on condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

    "C2 CompilerThread0" daemon prio=10 tid=0x0000000052167800 nid=0x280f
    waiting on condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

    "Signal Dispatcher" daemon prio=10 tid=0x0000000052165800 nid=0x280e
    waiting on condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

    "Finalizer" daemon prio=10 tid=0x0000000052145000 nid=0x280d in
    Object.wait() [0x0000000041de9000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007800a43e8> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <0x00000007800a43e8> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

    "Reference Handler" daemon prio=10 tid=0x0000000052143000 nid=0x280c
    in Object.wait() [0x0000000041ce8000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007800a3618> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:485)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    - locked <0x00000007800a3618> (a java.lang.ref.Reference$Lock)

    "VM Thread" prio=10 tid=0x000000005213c000 nid=0x280b runnable

    "GC task thread#0 (ParallelGC)" prio=10 tid=0x00000000520f5000
    nid=0x2809 runnable

    "GC task thread#1 (ParallelGC)" prio=10 tid=0x00000000520f7000
    nid=0x280a runnable

    "VM Periodic Task Thread" prio=10 tid=0x000000005217a000 nid=0x2812
    waiting on condition

    JNI global references: 974

    Heap
    PSYoungGen total 560640K, used 480899K [0x00000007d5560000,
    0x0000000800000000, 0x0000000800000000)
    eden space 490368K, 83% used
    [0x00000007d5560000,0x00000007ee66ca20,0x00000007f3440000)
    from space 70272K, 99% used
    [0x00000007f3440000,0x00000007f78d4590,0x00000007f78e0000)
    to space 110464K, 0% used
    [0x00000007f9420000,0x00000007f9420000,0x0000000800000000)
    PSOldGen total 266368K, used 155632K [0x0000000780000000,
    0x0000000790420000, 0x00000007d5560000)
    object space 266368K, 58% used
    [0x0000000780000000,0x00000007897fc3c0,0x0000000790420000)
    PSPermGen total 33216K, used 18585K [0x000000077ae00000,
    0x000000077ce70000, 0x0000000780000000)
    object space 33216K, 55% used
    [0x000000077ae00000,0x000000077c0267c0,0x000000077ce70000)

    Regards,
    Bogdan

    On Wed, Sep 21, 2011 at 7:06 PM, Stack wrote:
    On Wed, Sep 21, 2011 at 1:07 AM, Bogdan Ghidireac wrote:
    I have an HBase 0.90.3 fleet with more than 100 hosts distributed
    across 3 datacenters.
    Yesterday, we experienced some network issues
    for several minutes and the region servers from one data center lost
    the connectivity with the namenode. They started the shutdown sequence
    but about 20 hosts were unable to complete it successfully. This is
    bad for us because we have to restart them manually.
    HBase and HDFS are not made to span datacenters.
    Is my assumption correct? Should I open a JIRA?
    You could paste the thread dump but before doing anything, I'd suggest
    you change your layout so cluster runs inside a single datacenter.

    St.Ack
  • Daniel Washburn at Sep 22, 2011 at 1:22 am

    On Sep 21, 2011, at 1:51 PM, Bogdan Ghidireac wrote:

    HBase and HDFS are not made to span datacenters.
    Yes, I know. I think the correct terminology is 3 availability zones
    within the same region (ping between them is about 1ms)
    [snip]
    Is my assumption correct? Should I open a JIRA?
    You could paste the thread dump but before doing anything, I'd suggest
    you change your layout so cluster runs inside a single datacenter.

    St.Ack
    Echoing St.ack's advice, but not wanting to sound like I'm scolding, you really should consider moving your AMIs into a single Availability Zone. EC2 puts up some impressive availability numbers as a platform, but when you stack several layers of software onto it, you start to push the limits of the availability zone SLA, and certainly the availability requirements of the NameNode.

    --Dan

    --

    Daniel Washburn
    Systems Engineer
    Explorys, Inc.
    daniel.washburn@explorys.com
  • Stack at Sep 22, 2011 at 1:41 am
    The bit about 1ms ping times sounds a bit good to me but Dan below
    says is better.
    St.Ack

    On Wed, Sep 21, 2011 at 6:21 PM, Daniel Washburn
    wrote:
    On Sep 21, 2011, at 1:51 PM, Bogdan Ghidireac wrote:

    HBase and HDFS are not made to span datacenters.
    Yes, I know. I think the correct terminology is 3 availability zones
    within the same region (ping between them is about 1ms)
    [snip]
    Is my assumption correct? Should I open a JIRA?
    You could paste the thread dump but before doing anything, I'd suggest
    you change your layout so cluster runs inside a single datacenter.

    St.Ack
    Echoing St.ack's advice, but not wanting to sound like I'm scolding, you really should consider moving your AMIs into a single Availability Zone.  EC2 puts up some impressive availability numbers as a platform, but when you stack several layers of software onto it, you start to push the limits of the availability zone SLA, and certainly the availability requirements of the NameNode.

    --Dan

    --

    Daniel Washburn
    Systems Engineer
    Explorys, Inc.
    daniel.washburn@explorys.com
  • Bogdan Ghidireac at Sep 22, 2011 at 9:25 am

    On Thu, Sep 22, 2011 at 4:41 AM, Stack wrote:
    The bit about 1ms ping times sounds a bit good to me but Dan below
    says is better.
    St.Ack
    I am not using EC2 but it is a good analogy :)

    pa-hbase-master-na-7001.iad7$ ping pa-hbase-datanode-na-1024.vdc
    PING pa-hbase-datanode-na-1024.vdc (10.119.133.76) 56(84) bytes of data.
    64 bytes from pa-hbase-datanode-na-1024.vdc (10.119.133.76):
    icmp_seq=1 ttl=58 time=0.812 ms
    64 bytes from pa-hbase-datanode-na-1024.vdc (10.119.133.76):
    icmp_seq=2 ttl=58 time=0.789 ms
    64 bytes from pa-hbase-datanode-na-1024.vdc (10.119.133.76):
    icmp_seq=3 ttl=58 time=0.787 ms

    pa-hbase-master-na-7001.iad7$ ping pa-hbase-datanode-na-6004.iad6
    PING pa-hbase-datanode-na-6004.iad6 (10.194.179.68) 56(84) bytes of data.
    64 bytes from pa-hbase-datanode-na-6004.iad6 (10.194.179.68):
    icmp_seq=1 ttl=58 time=1.54 ms
    64 bytes from pa-hbase-datanode-na-6004.iad6 (10.194.179.68):
    icmp_seq=2 ttl=58 time=1.54 ms
    64 bytes from pa-hbase-datanode-na-6004.iad6 (10.194.179.68):
    icmp_seq=3 ttl=58 time=1.58 ms


    What about the RS shutdown problem? I don't see the connection between
    the ping time and the fact that same threads will not close.

    Regards,
    Bogdan
    On Wed, Sep 21, 2011 at 6:21 PM, Daniel Washburn
    wrote:
    On Sep 21, 2011, at 1:51 PM, Bogdan Ghidireac wrote:

    HBase and HDFS are not made to span datacenters.
    Yes, I know. I think the correct terminology is 3 availability zones
    within the same region (ping between them is about 1ms)
    [snip]
    Is my assumption correct? Should I open a JIRA?
    You could paste the thread dump but before doing anything, I'd suggest
    you change your layout so cluster runs inside a single datacenter.

    St.Ack
    Echoing St.ack's advice, but not wanting to sound like I'm scolding, you really should consider moving your AMIs into a single Availability Zone.  EC2 puts up some impressive availability numbers as a platform, but when you stack several layers of software onto it, you start to push the limits of the availability zone SLA, and certainly the availability requirements of the NameNode.

    --Dan

    --

    Daniel Washburn
    Systems Engineer
    Explorys, Inc.
    daniel.washburn@explorys.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshbase, hadoop
postedSep 21, '11 at 8:07a
activeSep 22, '11 at 9:25a
posts7
users3
websitehbase.apache.org

People

Translate

site design / logo © 2022 Grokbase