FAQ
Hi
Datanode goes down. and then looks like ReplicationMonitor tries to
even-out the replication
However while doing so,
it holds the lock on FsNameSystem
With this lock held, other threads wait on this lock to respond
As a result, the namenode does not list the dirs/ Web-UI does not respond

I would appreciate any pointers for this problem ?

(Hadoop .18.1)

-Sagar


Namenode freeze stackdump :


2009-01-14 00:57:02
Full thread dump Java HotSpot(TM) 64-Bit Server VM (10.0-b23 mixed mode):

"SocketListener0-4" prio=10 tid=0x00002aac54008000 nid=0x644d in
Object.wait() [0x000000004535a000..0x000000004535aa80]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aab6cb1dba0> (a
org.mortbay.util.ThreadPool$PoolThread)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
- locked <0x00002aab6cb1dba0> (a org.mortbay.util.ThreadPool$PoolThread)

Locked ownable synchronizers:
- None

"SocketListener0-5" prio=10 tid=0x00002aac54008c00 nid=0x63f1 in
Object.wait() [0x000000004545b000..0x000000004545bb00]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aab6c2ea1a8> (a
org.mortbay.util.ThreadPool$PoolThread)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
- locked <0x00002aab6c2ea1a8> (a org.mortbay.util.ThreadPool$PoolThread)

Locked ownable synchronizers:
- None

"Trash Emptier" daemon prio=10 tid=0x00000000511ca400 nid=0x1fd waiting
on condition [0x0000000045259000..0x0000000045259a00]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.fs.Trash$Emptier.run(Trash.java:219)
at java.lang.Thread.run(Thread.java:619)

Locked ownable synchronizers:
- None

"org.apache.hadoop.dfs.DFSClient$LeaseChecker@767a9224" daemon prio=10
tid=0x0000000051384400 nid=0x1fc
sleeping[0x0000000045158000..0x0000000045158a80]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:792)
at java.lang.Thread.run(Thread.java:619)

Locked ownable synchronizers:
- None

"IPC Server handler 44 on 54310" daemon prio=10 tid=0x00002aac40183c00
nid=0x1f4 waiting for monitor entry [0x0000000044f56000..0x0000000044f56d80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.blockReportProcessed(FSNamesystem.java:1880)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.handleHeartbeat(FSNamesystem.java:2127)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:602)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 43 on 54310" daemon prio=10 tid=0x00002aac40182400
nid=0x1f3 waiting for monitor entry [0x0000000044e55000..0x0000000044e55a00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:922)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:903)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:284)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 42 on 54310" daemon prio=10 tid=0x00002aac40181000
nid=0x1f2 waiting for monitor entry [0x0000000044d54000..0x0000000044d54a80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.blockReportProcessed(FSNamesystem.java:1880)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.handleHeartbeat(FSNamesystem.java:2127)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:602)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 41 on 54310" daemon prio=10 tid=0x00002aac4017fc00
nid=0x1f1 waiting for monitor entry [0x0000000044c53000..0x0000000044c53b00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:741)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocations(FSNamesystem.java:727)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocations(FSNamesystem.java:703)
at org.apache.hadoop.dfs.NameNode.getBlockLocations(NameNode.java:257)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 40 on 54310" daemon prio=10 tid=0x00002aac4017e400
nid=0x1f0 waiting for monitor entry [0x0000000044b52000..0x0000000044b52b80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:2652)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:613)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 39 on 54310" daemon prio=10 tid=0x00002aac4017d000
nid=0x1ef waiting for monitor entry [0x0000000044a51000..0x0000000044a51c00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:922)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:903)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:284)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 38 on 54310" daemon prio=10 tid=0x00002aac4017bc00
nid=0x1ee waiting for monitor entry [0x0000000044950000..0x0000000044950c80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:922)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:903)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:284)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 37 on 54310" daemon prio=10 tid=0x00002aac4017a400
nid=0x1ed waiting for monitor entry [0x000000004484f000..0x000000004484fd00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:741)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocations(FSNamesystem.java:727)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocations(FSNamesystem.java:703)
at org.apache.hadoop.dfs.NameNode.getBlockLocations(NameNode.java:257)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 36 on 54310" daemon prio=10 tid=0x00002aac40179000
nid=0x1ec waiting for monitor entry [0x000000004474e000..0x000000004474ed80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:741)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocations(FSNamesystem.java:727)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocations(FSNamesystem.java:703)
at org.apache.hadoop.dfs.NameNode.getBlockLocations(NameNode.java:257)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 35 on 54310" daemon prio=10 tid=0x00002aac40177c00
nid=0x1eb waiting for monitor entry [0x000000004464d000..0x000000004464da00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:922)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:903)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:284)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 34 on 54310" daemon prio=10 tid=0x00002aac40176400
nid=0x1ea waiting for monitor entry [0x000000004454c000..0x000000004454ca80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:922)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:903)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:284)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 33 on 54310" daemon prio=10 tid=0x00002aac40175000
nid=0x1e9 waiting for monitor entry [0x000000004444b000..0x000000004444bb00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.blockReportProcessed(FSNamesystem.java:1880)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.handleHeartbeat(FSNamesystem.java:2127)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:602)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 32 on 54310" daemon prio=10 tid=0x00002aac40173800
nid=0x1e8 waiting for monitor entry [0x000000004434a000..0x000000004434ab80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.blockReportProcessed(FSNamesystem.java:1880)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.handleHeartbeat(FSNamesystem.java:2127)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:602)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 31 on 54310" daemon prio=10 tid=0x00002aac40172400
nid=0x1e7 waiting for monitor entry [0x0000000044249000..0x0000000044249c00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.blockReportProcessed(FSNamesystem.java:1880)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.handleHeartbeat(FSNamesystem.java:2127)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:602)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 30 on 54310" daemon prio=10 tid=0x00002aac40171000
nid=0x1e6 waiting for monitor entry [0x0000000044148000..0x0000000044148c80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:922)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:903)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:284)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 29 on 54310" daemon prio=10 tid=0x00002aac4016f800
nid=0x1e5 waiting for monitor entry [0x0000000044047000..0x0000000044047d00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:922)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:903)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:284)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 28 on 54310" daemon prio=10 tid=0x00002aac4016e400
nid=0x1e4 waiting for monitor entry [0x0000000043f46000..0x0000000043f46d80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:922)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:903)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:284)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 27 on 54310" daemon prio=10 tid=0x00002aac4016d000
nid=0x1e3 waiting for monitor entry [0x0000000043e45000..0x0000000043e45a00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:922)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:903)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:284)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 26 on 54310" daemon prio=10 tid=0x00002aac4016b800
nid=0x1e2 waiting for monitor entry [0x0000000043d44000..0x0000000043d44a80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:741)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocations(FSNamesystem.java:727)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocations(FSNamesystem.java:703)
at org.apache.hadoop.dfs.NameNode.getBlockLocations(NameNode.java:257)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 25 on 54310" daemon prio=10 tid=0x00002aac4016a400
nid=0x1e1 waiting for monitor entry [0x0000000043c43000..0x0000000043c43b00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.blockReportProcessed(FSNamesystem.java:1880)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.handleHeartbeat(FSNamesystem.java:2127)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:602)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 24 on 54310" daemon prio=10 tid=0x00002aac40168c00
nid=0x1e0 waiting for monitor entry [0x0000000043b42000..0x0000000043b42b80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.blockReportProcessed(FSNamesystem.java:1880)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.handleHeartbeat(FSNamesystem.java:2127)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:602)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 23 on 54310" daemon prio=10 tid=0x00002aac40167800
nid=0x1df waiting for monitor entry [0x0000000043a41000..0x0000000043a41c00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:922)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:903)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:284)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 22 on 54310" daemon prio=10 tid=0x00002aac40166400
nid=0x1de waiting for monitor entry [0x0000000043940000..0x0000000043940c80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.blockReportProcessed(FSNamesystem.java:1880)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.handleHeartbeat(FSNamesystem.java:2127)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:602)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 21 on 54310" daemon prio=10 tid=0x00002aac40164c00
nid=0x1dd waiting for monitor entry [0x000000004383f000..0x000000004383fd00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:741)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocations(FSNamesystem.java:727)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocations(FSNamesystem.java:703)
at org.apache.hadoop.dfs.NameNode.getBlockLocations(NameNode.java:257)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 20 on 54310" daemon prio=10 tid=0x00002aac40163800
nid=0x1dc waiting for monitor entry [0x000000004373e000..0x000000004373ed80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:741)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocations(FSNamesystem.java:727)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocations(FSNamesystem.java:703)
at org.apache.hadoop.dfs.NameNode.getBlockLocations(NameNode.java:257)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 19 on 54310" daemon prio=10 tid=0x00002aac40162400
nid=0x1db waiting for monitor entry [0x000000004363d000..0x000000004363da00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.blockReportProcessed(FSNamesystem.java:1880)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.handleHeartbeat(FSNamesystem.java:2127)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:602)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 18 on 54310" daemon prio=10 tid=0x00002aac40160c00
nid=0x1da waiting for monitor entry [0x000000004353c000..0x000000004353ca80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:922)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:903)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:284)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 17 on 54310" daemon prio=10 tid=0x00002aac4015f800
nid=0x1d9 waiting for monitor entry [0x000000004343b000..0x000000004343bb00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:922)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:903)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:284)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 16 on 54310" daemon prio=10 tid=0x00002aac4015e400
nid=0x1d8 waiting for monitor entry [0x000000004333a000..0x000000004333ab80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.datanodeReport(FSNamesystem.java:3262)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.NameNode.getDatanodeReport(NameNode.java:491)
at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 15 on 54310" daemon prio=10 tid=0x00002aac4015cc00
nid=0x1d7 waiting for monitor entry [0x0000000043239000..0x0000000043239c00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:922)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:903)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:284)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 14 on 54310" daemon prio=10 tid=0x00002aac4015b800
nid=0x1d6 waiting for monitor entry [0x0000000043138000..0x0000000043138c80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.markBlockAsCorrupt(FSNamesystem.java:1351)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.NameNode.reportBadBlocks(NameNode.java:374)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 13 on 54310" daemon prio=10 tid=0x00002aac4015a400
nid=0x1d5 waiting for monitor entry [0x0000000043037000..0x0000000043037d00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.blockReportProcessed(FSNamesystem.java:1880)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.handleHeartbeat(FSNamesystem.java:2127)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:602)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 12 on 54310" daemon prio=10 tid=0x00002aac40158c00
nid=0x1d4 waiting for monitor entry [0x0000000042f36000..0x0000000042f36d80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.blockReportProcessed(FSNamesystem.java:1880)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.handleHeartbeat(FSNamesystem.java:2127)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:602)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 11 on 54310" daemon prio=10 tid=0x00002aac40157800
nid=0x1d3 waiting for monitor entry [0x0000000042e35000..0x0000000042e35a00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:922)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:903)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:284)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 10 on 54310" daemon prio=10 tid=0x00002aac40156000
nid=0x1d2 waiting for monitor entry [0x0000000042d34000..0x0000000042d34a80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:741)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocations(FSNamesystem.java:727)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocations(FSNamesystem.java:703)
at org.apache.hadoop.dfs.NameNode.getBlockLocations(NameNode.java:257)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 9 on 54310" daemon prio=10 tid=0x00002aac40154c00
nid=0x1d1 waiting for monitor entry [0x0000000042c33000..0x0000000042c33b00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.markBlockAsCorrupt(FSNamesystem.java:1351)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.NameNode.reportBadBlocks(NameNode.java:374)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 8 on 54310" daemon prio=10 tid=0x00002aac40153800
nid=0x1d0 waiting for monitor entry [0x0000000042b32000..0x0000000042b32b80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:922)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:903)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:284)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 7 on 54310" daemon prio=10 tid=0x00002aac40152400
nid=0x1cf waiting for monitor entry [0x0000000042a31000..0x0000000042a31c00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:922)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:903)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:284)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 6 on 54310" daemon prio=10 tid=0x00002aac4014b000
nid=0x1ce waiting for monitor entry [0x0000000042930000..0x0000000042930c80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.blockReportProcessed(FSNamesystem.java:1880)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.handleHeartbeat(FSNamesystem.java:2127)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:602)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 5 on 54310" daemon prio=10 tid=0x00002aac40149800
nid=0x1cd waiting for monitor entry [0x000000004282f000..0x000000004282fd00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:922)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:903)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:284)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 4 on 54310" daemon prio=10 tid=0x00002aac40148800
nid=0x1cc waiting for monitor entry [0x000000004272e000..0x000000004272ed80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.blockReportProcessed(FSNamesystem.java:1880)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.handleHeartbeat(FSNamesystem.java:2127)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:602)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 3 on 54310" daemon prio=10 tid=0x00002aac4008ec00
nid=0x1cb waiting for monitor entry [0x000000004262d000..0x000000004262da00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.blockReportProcessed(FSNamesystem.java:1880)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.handleHeartbeat(FSNamesystem.java:2127)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:602)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 2 on 54310" daemon prio=10 tid=0x00002aac4008a000
nid=0x1ca waiting for monitor entry [0x000000004252c000..0x000000004252ca80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.blockReportProcessed(FSNamesystem.java:1880)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.handleHeartbeat(FSNamesystem.java:2127)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:602)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 1 on 54310" daemon prio=10 tid=0x00002aac40088c00
nid=0x1c9 waiting for monitor entry [0x000000004242b000..0x000000004242bb00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:741)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocations(FSNamesystem.java:727)
at
org.apache.hadoop.dfs.FSNamesystem.getBlockLocations(FSNamesystem.java:703)
at org.apache.hadoop.dfs.NameNode.getBlockLocations(NameNode.java:257)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server handler 0 on 54310" daemon prio=10 tid=0x00002aac400ac000
nid=0x1c8 waiting for monitor entry [0x000000004232a000..0x000000004232ab80]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.dfs.FSNamesystem.blockReportProcessed(FSNamesystem.java:1880)
- waiting to lock <0x00002aaab423a530> (a
org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.handleHeartbeat(FSNamesystem.java:2127)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:602)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Locked ownable synchronizers:
- None

"IPC Server listener on 54310" daemon prio=10 tid=0x00002aac400c7800
nid=0x1c7 runnable [0x0000000042229000..0x0000000042229c00]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00002aaab41bff70> (a sun.nio.ch.Util$1)
- locked <0x00002aaab41bff58> (a java.util.Collections$UnmodifiableSet)
- locked <0x00002aaab41bfd90> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at org.apache.hadoop.ipc.Server$Listener.run(Server.java:296)

Locked ownable synchronizers:
- None

"IPC Server Responder" daemon prio=10 tid=0x00002aac40086800 nid=0x1c6
runnable [0x0000000042128000..0x0000000042128c80]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00002aaab41c09c8> (a sun.nio.ch.Util$1)
- locked <0x00002aaab41c09b0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00002aaab41c0650> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.apache.hadoop.ipc.Server$Responder.run(Server.java:455)

Locked ownable synchronizers:
- None

"Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=50070]"
prio=10 tid=0x00002aac400adc00 nid=0x1c3 runnable
[0x0000000041e25000..0x0000000041e25a00]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked <0x00002aaae69a9990> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:432)
at org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:631)

Locked ownable synchronizers:
- None

"SessionScavenger" daemon prio=10 tid=0x00002aac400af000 nid=0x1c2
waiting on condition [0x0000000041d24000..0x0000000041d24a80]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)

Locked ownable synchronizers:
- None

"org.apache.hadoop.dfs.FSNamesystem$DecommissionedMonitor@35ecfe07"
daemon prio=10 tid=0x00002aac40083800 nid=0x1c1 waiting on condition
[0x0000000041c23000..0x0000000041c23b00]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.dfs.FSNamesystem$DecommissionedMonitor.run(FSNamesystem.java:3602)
at java.lang.Thread.run(Thread.java:619)

Locked ownable synchronizers:
- None

"org.apache.hadoop.dfs.FSNamesystem$ResolutionMonitor@49f4bcf7" daemon
prio=10 tid=0x00002aac40070000 nid=0x1c0 waiting on condition
[0x0000000041b22000..0x0000000041b22b80]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab40fec38> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
org.apache.hadoop.dfs.FSNamesystem$ResolutionMonitor.run(FSNamesystem.java:1840)
at java.lang.Thread.run(Thread.java:619)

Locked ownable synchronizers:
- None

"org.apache.hadoop.dfs.FSNamesystem$ReplicationMonitor@53077fc9" daemon
prio=10 tid=0x00002aac4006ec00 nid=0x1bf runnable
[0x0000000041a21000..0x0000000041a21c00]
java.lang.Thread.State: RUNNABLE
at
org.apache.hadoop.dfs.FSNamesystem.invalidateWorkForOneNode(FSNamesystem.java:2469)
- locked <0x00002aaab423a530> (a org.apache.hadoop.dfs.FSNamesystem)
at
org.apache.hadoop.dfs.FSNamesystem.computeInvalidateWork(FSNamesystem.java:2247)
at
org.apache.hadoop.dfs.FSNamesystem.computeDatanodeWork(FSNamesystem.java:2240)
at
org.apache.hadoop.dfs.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2189)
at java.lang.Thread.run(Thread.java:619)

Locked ownable synchronizers:
- None

"org.apache.hadoop.dfs.FSNamesystem$HeartbeatMonitor@21cc5069" daemon
prio=10 tid=0x00002aac4006a000 nid=0x1bd waiting on condition
[0x000000004181f000..0x000000004181fd00]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.dfs.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2173)
at java.lang.Thread.run(Thread.java:619)

Locked ownable synchronizers:
- None

"org.apache.hadoop.dfs.PendingReplicationBlocks$PendingReplicationMonitor@5e29c58e"
daemon prio=10 tid=0x00002aac40069800 nid=0x1bc waiting on condition
[0x000000004171e000..0x000000004171ed80]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.dfs.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:185)
at java.lang.Thread.run(Thread.java:619)

Locked ownable synchronizers:
- None

"Attach Listener" daemon prio=10 tid=0x000000005143e800 nid=0x7fe9
waiting on condition [0x0000000000000000..0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"Timer thread for monitoring dfs" daemon prio=10 tid=0x00002aac4000cc00
nid=0x7ea4 in Object.wait() [0x000000004151c000..0x000000004151cc80]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab41a5cc0> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x00002aaab41a5cc0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)

Locked ownable synchronizers:
- None

"Timer thread for monitoring jvm" daemon prio=10 tid=0x00002aac4000b800
nid=0x7ea3 in Object.wait() [0x000000004141b000..0x000000004141bd00]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab41aa850> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x00002aaab41aa850> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)

Locked ownable synchronizers:
- None

"RMI TCP Accept-0" daemon prio=10 tid=0x00002aac3c146c00 nid=0x7ea0
runnable [0x0000000041219000..0x0000000041219a00]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked <0x00002aaab41a5278> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:619)

Locked ownable synchronizers:
- None

"Low Memory Detector" daemon prio=10 tid=0x0000000051114c00 nid=0x7e9f
runnable [0x0000000000000000..0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"CompilerThread1" daemon prio=10 tid=0x0000000051112800 nid=0x7e9e
waiting on condition [0x0000000000000000..0x00000000410163a0]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"CompilerThread0" daemon prio=10 tid=0x000000005110f000 nid=0x7e9d
waiting on condition [0x0000000000000000..0x0000000040f15430]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"Signal Dispatcher" daemon prio=10 tid=0x000000005110d800 nid=0x7e9c
runnable [0x0000000000000000..0x0000000040e15890]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"Finalizer" daemon prio=10 tid=0x00000000510e2400 nid=0x7e9b in
Object.wait() [0x0000000040d14000..0x0000000040d14c80]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab41257e0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x00002aaab41257e0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

Locked ownable synchronizers:
- None

"Reference Handler" daemon prio=10 tid=0x00000000510e0c00 nid=0x7e9a in
Object.wait() [0x0000000040c13000..0x0000000040c13d00]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab41af668> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00002aaab41af668> (a java.lang.ref.Reference$Lock)

Locked ownable synchronizers:
- None

"main" prio=10 tid=0x0000000051074000 nid=0x7e90 in Object.wait()
[0x0000000040209000..0x0000000040209ec0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab41a9ee8> (a org.apache.hadoop.ipc.RPC$Server)
at java.lang.Object.wait(Object.java:485)
at org.apache.hadoop.ipc.Server.join(Server.java:1021)
- locked <0x00002aaab41a9ee8> (a org.apache.hadoop.ipc.RPC$Server)
at org.apache.hadoop.dfs.NameNode.join(NameNode.java:206)
at org.apache.hadoop.dfs.NameNode.main(NameNode.java:841)

Locked ownable synchronizers:
- None

"VM Thread" prio=10 tid=0x00000000510dbc00 nid=0x7e99 runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x000000005107e400
nid=0x7e91 runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x000000005107f800
nid=0x7e92 runnable

"GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000051080c00
nid=0x7e93 runnable

"GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000051082000
nid=0x7e94 runnable

"GC task thread#4 (ParallelGC)" prio=10 tid=0x0000000051083400
nid=0x7e95 runnable

"GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000051084800
nid=0x7e96 runnable

"GC task thread#6 (ParallelGC)" prio=10 tid=0x0000000051085c00
nid=0x7e97 runnable

"GC task thread#7 (ParallelGC)" prio=10 tid=0x0000000051087000
nid=0x7e98 runnable

"VM Periodic Task Thread" prio=10 tid=0x00002aac3c14ec00 nid=0x7ea1
waiting on condition

JNI global references: 1125

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJan 14, '09 at 9:06a
activeJan 14, '09 at 9:06a
posts1
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Sagar Naik: 1 post

People

Translate

site design / logo © 2022 Grokbase