Have anyone experienced this?
When the cluster is busy sometimes one map task of a job gets stuck.
An attempt is then made to kill the task, but this fails.
This shows up in the web interface:
task_0028_m_000534_0 node17.herd1 RUNNING 0.00% 10-Nov-2006 12:21:12
10-Nov-2006 12:22:19 (1mins, 6sec)
Task failed to report status for 604 seconds. Killing.
And that's the way it stays. Only way to get rid of it / rescue the job is
to restart the tasktracker.
This is what I found in the log a few times.
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(
SocketChannelImpl.java:125)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294)
at org.apache.hadoop.ipc.SocketChannelOutputStream.flushBuffer(
SocketChannelOutputStream.java:108)
at org.apache.hadoop.ipc.SocketChannelOutputStream.write(
SocketChannelOutputStream.java:89)
at java.io.BufferedOutputStream.flushBuffer(
BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:532)
I've filed an issue about this at:
http://issues.apache.org/jira/browse/HADOOP-707
The open file limit is at 65000, so shouldn't be a problem.
All nodes are on the same network switch.
Full thread dump:
2006-11-17 15:08:14,707 INFO org.apache.hadoop.mapred.TaskTracker:
task_0133_m_000948_0: Task failed to report status for 608 seconds. Killing.
2006-11-17 15:08:14,722 INFO org.apache.hadoop.mapred.TaskTracker: Process
Thread Dump: lost task
65 active threads
Thread 46827 (IPC Client connection to master1.herd1/10.0.5.10:9001):
State: WAITING
Blocked count: 1
Waited count: 2
Waiting on org.apache.hadoop.ipc.Client$[email protected]
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:195)
org.apache.hadoop.ipc.Client$Connection.run(Client.java:234)
Thread 46826 (btpool0-893):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46825 (btpool0-892):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46822 (btpool0-891):
State: RUNNABLE
Blocked count: 0
Waited count: 1
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46811 (btpool0-890):
State: RUNNABLE
Blocked count: 3
Waited count: 1
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46810 (btpool0-889):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46803 (btpool0-888):
State: RUNNABLE
Blocked count: 0
Waited count: 8
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46408 (btpool0-887):
State: RUNNABLE
Blocked count: 0
Waited count: 41
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46402 (btpool0-886):
State: RUNNABLE
Blocked count: 0
Waited count: 26
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46396 (btpool0-885):
State: RUNNABLE
Blocked count: 4
Waited count: 48
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.read(
LocalFileSystem.java:113)
org.apache.hadoop.fs.FSDataInputStream$Checker.read(
FSDataInputStream.java:91)
org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(
FSDataInputStream.java:189)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
java.io.DataInputStream.read(DataInputStream.java:80)
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(
TaskTracker.java:1411)
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:356)
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)
org.mortbay.jetty.handler.ContextHandlerCollection.handle(
ContextHandlerCollection.java:149)
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:141)
org.mortbay.jetty.Server.handle(Server.java:269)
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:430)
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(
HttpConnection.java:687)
Thread 46345 (btpool0-883):
State: TIMED_WAITING
Blocked count: 0
Waited count: 26
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:469)
Thread 46333 (btpool0-881):
State: RUNNABLE
Blocked count: 0
Waited count: 42
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46305 (btpool0-880):
State: RUNNABLE
Blocked count: 0
Waited count: 43
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46304 (btpool0-879):
State: RUNNABLE
Blocked count: 0
Waited count: 37
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46295 (btpool0-878):
State: RUNNABLE
Blocked count: 1
Waited count: 36
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46253 (btpool0-877):
State: RUNNABLE
Blocked count: 1
Waited count: 48
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46252 (btpool0-876):
State: RUNNABLE
Blocked count: 0
Waited count: 46
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46251 (btpool0-875):
State: RUNNABLE
Blocked count: 0
Waited count: 51
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46242 (btpool0-874):
State: RUNNABLE
Blocked count: 0
Waited count: 43
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 45039 (Thread-28307):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 45038 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 45031 (Thread-28302):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 45030 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 45026 (Thread-28299):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 45025 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 44998 (Thread-28283):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 44997 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 44987 (Thread-28277):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 44986 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 44905 (btpool0-872):
State: TIMED_WAITING
Blocked count: 2
Waited count: 87
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:469)
Thread 44903 (btpool0-870):
State: TIMED_WAITING
Blocked count: 1
Waited count: 103
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:469)
Thread 44891 (btpool0-865):
State: RUNNABLE
Blocked count: 5
Waited count: 55
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 44888 (btpool0-863):
State: RUNNABLE
Blocked count: 7
Waited count: 135
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 44887 (btpool0-862):
State: RUNNABLE
Blocked count: 4
Waited count: 131
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 44882 (btpool0-857):
State: RUNNABLE
Blocked count: 9
Waited count: 138
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 44881 (btpool0-856):
State: RUNNABLE
Blocked count: 3
Waited count: 127
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 44841 (btpool0-851):
State: RUNNABLE
Blocked count: 6
Waited count: 127
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 44752 (btpool0-849):
State: RUNNABLE
Blocked count: 7
Waited count: 134
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 43631 (Thread-27401):
State: RUNNABLE
Blocked count: 114
Waited count: 1730
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43610 (Thread-27384):
State: RUNNABLE
Blocked count: 116
Waited count: 1687
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43591 (Thread-27369):
State: RUNNABLE
Blocked count: 79
Waited count: 1690
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43583 (Thread-27361):
State: RUNNABLE
Blocked count: 109
Waited count: 1690
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43575 (Thread-27353):
State: RUNNABLE
Blocked count: 128
Waited count: 1726
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43052 (btpool0-844):
State: RUNNABLE
Blocked count: 6
Waited count: 217
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 43049 (btpool0-842):
State: RUNNABLE
Blocked count: 4
Waited count: 213
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 43042 (btpool0-835):
State: RUNNABLE
Blocked count: 12
Waited count: 229
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.read(
LocalFileSystem.java:113)
org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(
FSDataInputStream.java:189)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
java.io.DataInputStream.readFully(DataInputStream.java:176)
java.io.DataInputStream.readFully(DataInputStream.java:152)
org.apache.hadoop.fs.FSDataInputStream$Checker.<init>(
FSDataInputStream.java:60)
org.apache.hadoop.fs.FSDataInputStream.<init>(FSDataInputStream.java
:279)
org.apache.hadoop.fs.FileSystem.open(FileSystem.java:156)
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(
TaskTracker.java:1409)
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:356)
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)
org.mortbay.jetty.handler.ContextHandlerCollection.handle(
ContextHandlerCollection.java:149)
Thread 40895 (btpool0-803):
State: RUNNABLE
Blocked count: 3
Waited count: 361
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 40380 (btpool0-790):
State: RUNNABLE
Blocked count: 7
Waited count: 391
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.read(
LocalFileSystem.java:113)
org.apache.hadoop.fs.FSDataInputStream$Checker.read(
FSDataInputStream.java:91)
org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(
FSDataInputStream.java:189)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
java.io.DataInputStream.read(DataInputStream.java:80)
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(
TaskTracker.java:1411)
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:356)
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)
org.mortbay.jetty.handler.ContextHandlerCollection.handle(
ContextHandlerCollection.java:149)
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:141)
org.mortbay.jetty.Server.handle(Server.java:269)
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:430)
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(
HttpConnection.java:687)
Thread 39121 (btpool0-784):
State: RUNNABLE
Blocked count: 16
Waited count: 439
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 39112 (btpool0-775):
State: RUNNABLE
Blocked count: 13
Waited count: 396
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 28 (org.apache.hadoop.dfs.DFSClient$[email protected]):
State: TIMED_WAITING
Blocked count: 0
Waited count: 0
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:462)
java.lang.Thread.run(Thread.java:595)
Thread 19 (org.apache.hadoop.io.ObjectWritable Connection Culler):
State: TIMED_WAITING
Blocked count: 154
Waited count: 0
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.ipc.Client$ConnectionCuller.run(Client.java:388)
Thread 18 (IPC Server handler 4 on 50050):
State: TIMED_WAITING
Blocked count: 1769
Waited count: 51161
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 17 (IPC Server handler 3 on 50050):
State: TIMED_WAITING
Blocked count: 1673
Waited count: 51202
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 16 (IPC Server handler 2 on 50050):
State: TIMED_WAITING
Blocked count: 1685
Waited count: 51188
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 15 (IPC Server handler 1 on 50050):
State: TIMED_WAITING
Blocked count: 1694
Waited count: 51183
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 14 (IPC Server handler 0 on 50050):
State: TIMED_WAITING
Blocked count: 1913
Waited count: 51137
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 13 (IPC Server listener on 50050):
State: RUNNABLE
Blocked count: 634
Waited count: 0
Stack:
sun.nio.ch.PollArrayWrapper.poll0(Native Method)
sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:224)
Thread 12 (btpool0-1 - Acceptor0 SocketConnector @ 0.0.0.0:50060):
State: RUNNABLE
Blocked count: 85
Waited count: 1517
Stack:
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
java.net.ServerSocket.implAccept(ServerSocket.java:450)
java.net.ServerSocket.accept(ServerSocket.java:421)
org.mortbay.jetty.bio.SocketConnector.accept(SocketConnector.java:97)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java
:630)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 11 (btpool0-0 - Invalidator - /):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(
AbstractSessionManager.java:933)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 9 (taskCleanup):
State: WAITING
Blocked count: 457
Waited count: 118
Waiting on [email protected]
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
org.apache.hadoop.mapred.TaskTracker$BlockingQueue.take(TaskTracker.java
:783)
org.apache.hadoop.mapred.TaskTracker$1.run(TaskTracker.java:143)
java.lang.Thread.run(Thread.java:595)
Thread 4 (Signal Dispatcher):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
Thread 3 (Finalizer):
State: WAITING
Blocked count: 99047
Waited count: 39648
Waiting on java.lang.ref.ReferenceQueue$[email protected]
Stack:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread 2 (Reference Handler):
State: WAITING
Blocked count: 102986
Waited count: 3131
Waiting on java.lang.ref.Reference$[email protected]
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread 1 (main):
State: RUNNABLE
Blocked count: 1538
Waited count: 107465
Stack:
sun.management.ThreadImpl.getThreadInfo0(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:144)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:120)
org.apache.hadoop.util.ReflectionUtils.printThreadInfo(
ReflectionUtils.java:93)
org.apache.hadoop.util.ReflectionUtils.logThreadInfo(
ReflectionUtils.java:141)
org.apache.hadoop.mapred.TaskTracker.markUnresponsiveTasks(
TaskTracker.java:553)
org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:449)
org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:720)
org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1374)
/Johan
