Grokbase Groups HBase dev March 2010
FAQ
deadlock between put and cacheflusher in 0.20 branch
----------------------------------------------------

Key: HBASE-2322
URL: https://issues.apache.org/jira/browse/HBASE-2322
Project: Hadoop HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
Fix For: 0.20.4, 0.21.0


{code}
Found one Java-level deadlock:
=============================
"IPC Server handler 59 on 60020":
waiting for ownable synchronizer 0x00007fec9eb050f8, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by "IPC Server handler 54 on 60020"
"IPC Server handler 54 on 60020":
waiting to lock monitor 0x000000004190e950 (object 0x00007fec64f25258, a java.util.HashSet),
which is held by "regionserver/10.20.20.186:60020.cacheFlusher"
"regionserver/10.20.20.186:60020.cacheFlusher":
waiting for ownable synchronizer 0x00007fec651df998, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "IPC Server handler 19 on 60020"
"IPC Server handler 19 on 60020":
waiting for ownable synchronizer 0x00007fec9eb050f8, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by "IPC Server handler 54 on 60020"

Java stack information for the threads listed above:
===================================================
"IPC Server handler 59 on 60020":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007fec9eb050f8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1299)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1281)
at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1789)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
"IPC Server handler 54 on 60020":
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.request(MemStoreFlusher.java:172)
- waiting to lock <0x00007fec64f25258> (a java.util.HashSet)
at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1549)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1534)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1281)
at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1789)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
"regionserver/10.20.20.186:60020.cacheFlusher":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007fec651df998> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:235)
- locked <0x00007fec64f25258> (a java.util.HashSet)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:149)
"IPC Server handler 19 on 60020":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007fec9eb050f8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:980)
at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:873)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:241)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushSomeRegions(MemStoreFlusher.java:352)
- locked <0x00007fec64ed96f0> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java:321)
- locked <0x00007fec64ed96f0> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1783)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)

Found 1 deadlock.
{code}

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • stack (JIRA) at Apr 16, 2010 at 1:03 am
    [ https://issues.apache.org/jira/browse/HBASE-2322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    stack resolved HBASE-2322.
    --------------------------

    Resolution: Fixed

    Fixed by hbase-2248.
    deadlock between put and cacheflusher in 0.20 branch
    ----------------------------------------------------

    Key: HBASE-2322
    URL: https://issues.apache.org/jira/browse/HBASE-2322
    Project: Hadoop HBase
    Issue Type: Bug
    Reporter: stack
    Assignee: stack
    Priority: Blocker
    Fix For: 0.20.4, 0.21.0

    Attachments: hbase-2322.png


    {code}
    Found one Java-level deadlock:
    =============================
    "IPC Server handler 59 on 60020":
    waiting for ownable synchronizer 0x00007fec9eb050f8, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
    which is held by "IPC Server handler 54 on 60020"
    "IPC Server handler 54 on 60020":
    waiting to lock monitor 0x000000004190e950 (object 0x00007fec64f25258, a java.util.HashSet),
    which is held by "regionserver/10.20.20.186:60020.cacheFlusher"
    "regionserver/10.20.20.186:60020.cacheFlusher":
    waiting for ownable synchronizer 0x00007fec651df998, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
    which is held by "IPC Server handler 19 on 60020"
    "IPC Server handler 19 on 60020":
    waiting for ownable synchronizer 0x00007fec9eb050f8, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
    which is held by "IPC Server handler 54 on 60020"
    Java stack information for the threads listed above:
    ===================================================
    "IPC Server handler 59 on 60020":
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x00007fec9eb050f8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
    at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
    at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1299)
    at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1281)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1789)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
    "IPC Server handler 54 on 60020":
    at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.request(MemStoreFlusher.java:172)
    - waiting to lock <0x00007fec64f25258> (a java.util.HashSet)
    at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1549)
    at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1534)
    at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
    at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1281)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1789)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
    "regionserver/10.20.20.186:60020.cacheFlusher":
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x00007fec651df998> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
    at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
    at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
    at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:235)
    - locked <0x00007fec64f25258> (a java.util.HashSet)
    at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:149)
    "IPC Server handler 19 on 60020":
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x00007fec9eb050f8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
    at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
    at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:980)
    at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:873)
    at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:241)
    at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushSomeRegions(MemStoreFlusher.java:352)
    - locked <0x00007fec64ed96f0> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
    at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java:321)
    - locked <0x00007fec64ed96f0> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1783)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
    Found 1 deadlock.
    {code}
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categorieshbase, hadoop
postedMar 14, '10 at 1:09a
activeApr 16, '10 at 1:03a
posts2
users1
websitehbase.apache.org

1 user in discussion

stack (JIRA): 2 posts

People

Translate

site design / logo © 2022 Grokbase