FAQ
Have anyone experienced this?

When the cluster is busy sometimes one map task of a job gets stuck.
An attempt is then made to kill the task, but this fails.

This shows up in the web interface:
task_0028_m_000534_0 node17.herd1 RUNNING 0.00% 10-Nov-2006 12:21:12
10-Nov-2006 12:22:19 (1mins, 6sec)
Task failed to report status for 604 seconds. Killing.

And that's the way it stays. Only way to get rid of it / rescue the job is
to restart the tasktracker.

This is what I found in the log a few times.

java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(
SocketChannelImpl.java:125)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294)
at org.apache.hadoop.ipc.SocketChannelOutputStream.flushBuffer(
SocketChannelOutputStream.java:108)
at org.apache.hadoop.ipc.SocketChannelOutputStream.write(
SocketChannelOutputStream.java:89)
at java.io.BufferedOutputStream.flushBuffer(
BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)

at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:532)

I've filed an issue about this at:
http://issues.apache.org/jira/browse/HADOOP-707

The open file limit is at 65000, so shouldn't be a problem.
All nodes are on the same network switch.

Full thread dump:

2006-11-17 15:08:14,707 INFO org.apache.hadoop.mapred.TaskTracker:
task_0133_m_000948_0: Task failed to report status for 608 seconds. Killing.
2006-11-17 15:08:14,722 INFO org.apache.hadoop.mapred.TaskTracker: Process
Thread Dump: lost task
65 active threads
Thread 46827 (IPC Client connection to master1.herd1/10.0.5.10:9001):
State: WAITING
Blocked count: 1
Waited count: 2
Waiting on org.apache.hadoop.ipc.Client$Connection@74e65f
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:195)
org.apache.hadoop.ipc.Client$Connection.run(Client.java:234)
Thread 46826 (btpool0-893):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46825 (btpool0-892):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46822 (btpool0-891):
State: RUNNABLE
Blocked count: 0
Waited count: 1
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46811 (btpool0-890):
State: RUNNABLE
Blocked count: 3
Waited count: 1
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46810 (btpool0-889):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46803 (btpool0-888):
State: RUNNABLE
Blocked count: 0
Waited count: 8
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46408 (btpool0-887):
State: RUNNABLE
Blocked count: 0
Waited count: 41
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46402 (btpool0-886):
State: RUNNABLE
Blocked count: 0
Waited count: 26
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46396 (btpool0-885):
State: RUNNABLE
Blocked count: 4
Waited count: 48
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.read(
LocalFileSystem.java:113)
org.apache.hadoop.fs.FSDataInputStream$Checker.read(
FSDataInputStream.java:91)
org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(
FSDataInputStream.java:189)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
java.io.DataInputStream.read(DataInputStream.java:80)
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(
TaskTracker.java:1411)
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:356)
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)
org.mortbay.jetty.handler.ContextHandlerCollection.handle(
ContextHandlerCollection.java:149)
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:141)
org.mortbay.jetty.Server.handle(Server.java:269)
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:430)
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(
HttpConnection.java:687)
Thread 46345 (btpool0-883):
State: TIMED_WAITING
Blocked count: 0
Waited count: 26
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:469)
Thread 46333 (btpool0-881):
State: RUNNABLE
Blocked count: 0
Waited count: 42
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46305 (btpool0-880):
State: RUNNABLE
Blocked count: 0
Waited count: 43
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46304 (btpool0-879):
State: RUNNABLE
Blocked count: 0
Waited count: 37
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46295 (btpool0-878):
State: RUNNABLE
Blocked count: 1
Waited count: 36
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46253 (btpool0-877):
State: RUNNABLE
Blocked count: 1
Waited count: 48
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46252 (btpool0-876):
State: RUNNABLE
Blocked count: 0
Waited count: 46
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46251 (btpool0-875):
State: RUNNABLE
Blocked count: 0
Waited count: 51
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 46242 (btpool0-874):
State: RUNNABLE
Blocked count: 0
Waited count: 43
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 45039 (Thread-28307):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 45038 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 45031 (Thread-28302):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 45030 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 45026 (Thread-28299):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 45025 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 44998 (Thread-28283):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 44997 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 44987 (Thread-28277):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 44986 (process reaper):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 44905 (btpool0-872):
State: TIMED_WAITING
Blocked count: 2
Waited count: 87
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:469)
Thread 44903 (btpool0-870):
State: TIMED_WAITING
Blocked count: 1
Waited count: 103
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:469)
Thread 44891 (btpool0-865):
State: RUNNABLE
Blocked count: 5
Waited count: 55
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 44888 (btpool0-863):
State: RUNNABLE
Blocked count: 7
Waited count: 135
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 44887 (btpool0-862):
State: RUNNABLE
Blocked count: 4
Waited count: 131
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 44882 (btpool0-857):
State: RUNNABLE
Blocked count: 9
Waited count: 138
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 44881 (btpool0-856):
State: RUNNABLE
Blocked count: 3
Waited count: 127
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 44841 (btpool0-851):
State: RUNNABLE
Blocked count: 6
Waited count: 127
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 44752 (btpool0-849):
State: RUNNABLE
Blocked count: 7
Waited count: 134
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 43631 (Thread-27401):
State: RUNNABLE
Blocked count: 114
Waited count: 1730
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43610 (Thread-27384):
State: RUNNABLE
Blocked count: 116
Waited count: 1687
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43591 (Thread-27369):
State: RUNNABLE
Blocked count: 79
Waited count: 1690
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43583 (Thread-27361):
State: RUNNABLE
Blocked count: 109
Waited count: 1690
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43575 (Thread-27353):
State: RUNNABLE
Blocked count: 128
Waited count: 1726
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43052 (btpool0-844):
State: RUNNABLE
Blocked count: 6
Waited count: 217
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 43049 (btpool0-842):
State: RUNNABLE
Blocked count: 4
Waited count: 213
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 43042 (btpool0-835):
State: RUNNABLE
Blocked count: 12
Waited count: 229
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.read(
LocalFileSystem.java:113)
org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(
FSDataInputStream.java:189)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
java.io.DataInputStream.readFully(DataInputStream.java:176)
java.io.DataInputStream.readFully(DataInputStream.java:152)
org.apache.hadoop.fs.FSDataInputStream$Checker.<init>(
FSDataInputStream.java:60)
org.apache.hadoop.fs.FSDataInputStream.<init>(FSDataInputStream.java
:279)
org.apache.hadoop.fs.FileSystem.open(FileSystem.java:156)
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(
TaskTracker.java:1409)
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:356)
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)
org.mortbay.jetty.handler.ContextHandlerCollection.handle(
ContextHandlerCollection.java:149)
Thread 40895 (btpool0-803):
State: RUNNABLE
Blocked count: 3
Waited count: 361
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 40380 (btpool0-790):
State: RUNNABLE
Blocked count: 7
Waited count: 391
Stack:
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.read(
LocalFileSystem.java:113)
org.apache.hadoop.fs.FSDataInputStream$Checker.read(
FSDataInputStream.java:91)
org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(
FSDataInputStream.java:189)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
java.io.DataInputStream.read(DataInputStream.java:80)
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(
TaskTracker.java:1411)
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:356)
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)
org.mortbay.jetty.handler.ContextHandlerCollection.handle(
ContextHandlerCollection.java:149)
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:141)
org.mortbay.jetty.Server.handle(Server.java:269)
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:430)
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(
HttpConnection.java:687)
Thread 39121 (btpool0-784):
State: RUNNABLE
Blocked count: 16
Waited count: 439
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 39112 (btpool0-775):
State: RUNNABLE
Blocked count: 13
Waited count: 396
Stack:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
org.mortbay.jetty.bio.SocketConnector$Connection.fill(
SocketConnector.java:181)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
org.mortbay.jetty.bio.SocketConnector$Connection.run(
SocketConnector.java:208)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 28 (org.apache.hadoop.dfs.DFSClient$LeaseChecker@1a7ddcf):
State: TIMED_WAITING
Blocked count: 0
Waited count: 0
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:462)
java.lang.Thread.run(Thread.java:595)
Thread 19 (org.apache.hadoop.io.ObjectWritable Connection Culler):
State: TIMED_WAITING
Blocked count: 154
Waited count: 0
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.ipc.Client$ConnectionCuller.run(Client.java:388)
Thread 18 (IPC Server handler 4 on 50050):
State: TIMED_WAITING
Blocked count: 1769
Waited count: 51161
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 17 (IPC Server handler 3 on 50050):
State: TIMED_WAITING
Blocked count: 1673
Waited count: 51202
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 16 (IPC Server handler 2 on 50050):
State: TIMED_WAITING
Blocked count: 1685
Waited count: 51188
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 15 (IPC Server handler 1 on 50050):
State: TIMED_WAITING
Blocked count: 1694
Waited count: 51183
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 14 (IPC Server handler 0 on 50050):
State: TIMED_WAITING
Blocked count: 1913
Waited count: 51137
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 13 (IPC Server listener on 50050):
State: RUNNABLE
Blocked count: 634
Waited count: 0
Stack:
sun.nio.ch.PollArrayWrapper.poll0(Native Method)
sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:224)
Thread 12 (btpool0-1 - Acceptor0 SocketConnector @ 0.0.0.0:50060):
State: RUNNABLE
Blocked count: 85
Waited count: 1517
Stack:
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
java.net.ServerSocket.implAccept(ServerSocket.java:450)
java.net.ServerSocket.accept(ServerSocket.java:421)
org.mortbay.jetty.bio.SocketConnector.accept(SocketConnector.java:97)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java
:630)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 11 (btpool0-0 - Invalidator - /):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(
AbstractSessionManager.java:933)
org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:475)
Thread 9 (taskCleanup):
State: WAITING
Blocked count: 457
Waited count: 118
Waiting on java.util.ArrayList@15192a
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
org.apache.hadoop.mapred.TaskTracker$BlockingQueue.take(TaskTracker.java
:783)
org.apache.hadoop.mapred.TaskTracker$1.run(TaskTracker.java:143)
java.lang.Thread.run(Thread.java:595)
Thread 4 (Signal Dispatcher):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
Thread 3 (Finalizer):
State: WAITING
Blocked count: 99047
Waited count: 39648
Waiting on java.lang.ref.ReferenceQueue$Lock@ac6214
Stack:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread 2 (Reference Handler):
State: WAITING
Blocked count: 102986
Waited count: 3131
Waiting on java.lang.ref.Reference$Lock@15b40a1
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread 1 (main):
State: RUNNABLE
Blocked count: 1538
Waited count: 107465
Stack:
sun.management.ThreadImpl.getThreadInfo0(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:144)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:120)
org.apache.hadoop.util.ReflectionUtils.printThreadInfo(
ReflectionUtils.java:93)
org.apache.hadoop.util.ReflectionUtils.logThreadInfo(
ReflectionUtils.java:141)
org.apache.hadoop.mapred.TaskTracker.markUnresponsiveTasks(
TaskTracker.java:553)
org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:449)
org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:720)
org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1374)


/Johan

Search Discussions

  • Albert Chern at Nov 17, 2006 at 7:58 pm
    Just wanted to chime in and say that I have experienced this often
    too. The task eventually completes, but after an inordinate amount of
    time. The last lone task gets stuck in the initialiazation stage.
    It's usually faster to restart the job.
    On 11/17/06, Johan Oskarsson wrote:
    Have anyone experienced this?

    When the cluster is busy sometimes one map task of a job gets stuck.
    An attempt is then made to kill the task, but this fails.

    This shows up in the web interface:
    task_0028_m_000534_0 node17.herd1 RUNNING 0.00% 10-Nov-2006 12:21:12
    10-Nov-2006 12:22:19 (1mins, 6sec)
    Task failed to report status for 604 seconds. Killing.

    And that's the way it stays. Only way to get rid of it / rescue the job is
    to restart the tasktracker.

    This is what I found in the log a few times.

    java.nio.channels.ClosedChannelException
    at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(
    SocketChannelImpl.java:125)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294)
    at org.apache.hadoop.ipc.SocketChannelOutputStream.flushBuffer(
    SocketChannelOutputStream.java:108)
    at org.apache.hadoop.ipc.SocketChannelOutputStream.write(
    SocketChannelOutputStream.java:89)
    at java.io.BufferedOutputStream.flushBuffer(
    BufferedOutputStream.java:65)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)

    at java.io.DataOutputStream.flush(DataOutputStream.java:106)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:532)

    I've filed an issue about this at:
    http://issues.apache.org/jira/browse/HADOOP-707

    The open file limit is at 65000, so shouldn't be a problem.
    All nodes are on the same network switch.

    Full thread dump:

    2006-11-17 15:08:14,707 INFO org.apache.hadoop.mapred.TaskTracker:
    task_0133_m_000948_0: Task failed to report status for 608 seconds. Killing.
    2006-11-17 15:08:14,722 INFO org.apache.hadoop.mapred.TaskTracker: Process
    Thread Dump: lost task
    65 active threads
    Thread 46827 (IPC Client connection to master1.herd1/10.0.5.10:9001):
    State: WAITING
    Blocked count: 1
    Waited count: 2
    Waiting on org.apache.hadoop.ipc.Client$Connection@74e65f
    Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:474)
    org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:195)
    org.apache.hadoop.ipc.Client$Connection.run(Client.java:234)
    Thread 46826 (btpool0-893):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 0
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 46825 (btpool0-892):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 0
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 46822 (btpool0-891):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 1
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 46811 (btpool0-890):
    State: RUNNABLE
    Blocked count: 3
    Waited count: 1
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 46810 (btpool0-889):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 0
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 46803 (btpool0-888):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 8
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 46408 (btpool0-887):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 41
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 46402 (btpool0-886):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 26
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 46396 (btpool0-885):
    State: RUNNABLE
    Blocked count: 4
    Waited count: 48
    Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.read(
    LocalFileSystem.java:113)
    org.apache.hadoop.fs.FSDataInputStream$Checker.read(
    FSDataInputStream.java:91)
    org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(
    FSDataInputStream.java:189)
    java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
    java.io.BufferedInputStream.read(BufferedInputStream.java:313)
    java.io.DataInputStream.read(DataInputStream.java:80)
    org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(
    TaskTracker.java:1411)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
    org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)
    org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:356)
    org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)
    org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)
    org.mortbay.jetty.handler.ContextHandlerCollection.handle(
    ContextHandlerCollection.java:149)
    org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:141)
    org.mortbay.jetty.Server.handle(Server.java:269)
    org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:430)
    org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(
    HttpConnection.java:687)
    Thread 46345 (btpool0-883):
    State: TIMED_WAITING
    Blocked count: 0
    Waited count: 26
    Stack:
    java.lang.Object.wait(Native Method)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:469)
    Thread 46333 (btpool0-881):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 42
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 46305 (btpool0-880):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 43
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 46304 (btpool0-879):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 37
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 46295 (btpool0-878):
    State: RUNNABLE
    Blocked count: 1
    Waited count: 36
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 46253 (btpool0-877):
    State: RUNNABLE
    Blocked count: 1
    Waited count: 48
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 46252 (btpool0-876):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 46
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 46251 (btpool0-875):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 51
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 46242 (btpool0-874):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 43
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 45039 (Thread-28307):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 0
    Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
    org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
    Thread 45038 (process reaper):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 0
    Stack:
    java.lang.UNIXProcess.waitForProcessExit(Native Method)
    java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
    java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
    Thread 45031 (Thread-28302):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 0
    Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
    org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
    Thread 45030 (process reaper):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 0
    Stack:
    java.lang.UNIXProcess.waitForProcessExit(Native Method)
    java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
    java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
    Thread 45026 (Thread-28299):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 0
    Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
    org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
    Thread 45025 (process reaper):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 0
    Stack:
    java.lang.UNIXProcess.waitForProcessExit(Native Method)
    java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
    java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
    Thread 44998 (Thread-28283):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 0
    Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
    org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
    Thread 44997 (process reaper):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 0
    Stack:
    java.lang.UNIXProcess.waitForProcessExit(Native Method)
    java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
    java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
    Thread 44987 (Thread-28277):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 0
    Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
    org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
    Thread 44986 (process reaper):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 0
    Stack:
    java.lang.UNIXProcess.waitForProcessExit(Native Method)
    java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
    java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
    Thread 44905 (btpool0-872):
    State: TIMED_WAITING
    Blocked count: 2
    Waited count: 87
    Stack:
    java.lang.Object.wait(Native Method)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:469)
    Thread 44903 (btpool0-870):
    State: TIMED_WAITING
    Blocked count: 1
    Waited count: 103
    Stack:
    java.lang.Object.wait(Native Method)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:469)
    Thread 44891 (btpool0-865):
    State: RUNNABLE
    Blocked count: 5
    Waited count: 55
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 44888 (btpool0-863):
    State: RUNNABLE
    Blocked count: 7
    Waited count: 135
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 44887 (btpool0-862):
    State: RUNNABLE
    Blocked count: 4
    Waited count: 131
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 44882 (btpool0-857):
    State: RUNNABLE
    Blocked count: 9
    Waited count: 138
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 44881 (btpool0-856):
    State: RUNNABLE
    Blocked count: 3
    Waited count: 127
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 44841 (btpool0-851):
    State: RUNNABLE
    Blocked count: 6
    Waited count: 127
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 44752 (btpool0-849):
    State: RUNNABLE
    Blocked count: 7
    Waited count: 134
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 43631 (Thread-27401):
    State: RUNNABLE
    Blocked count: 114
    Waited count: 1730
    Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
    java.io.BufferedInputStream.read(BufferedInputStream.java:313)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
    org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
    Thread 43610 (Thread-27384):
    State: RUNNABLE
    Blocked count: 116
    Waited count: 1687
    Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
    java.io.BufferedInputStream.read(BufferedInputStream.java:313)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
    org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
    Thread 43591 (Thread-27369):
    State: RUNNABLE
    Blocked count: 79
    Waited count: 1690
    Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
    java.io.BufferedInputStream.read(BufferedInputStream.java:313)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
    org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
    Thread 43583 (Thread-27361):
    State: RUNNABLE
    Blocked count: 109
    Waited count: 1690
    Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
    java.io.BufferedInputStream.read(BufferedInputStream.java:313)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
    org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
    Thread 43575 (Thread-27353):
    State: RUNNABLE
    Blocked count: 128
    Waited count: 1726
    Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
    java.io.BufferedInputStream.read(BufferedInputStream.java:313)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
    org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
    Thread 43052 (btpool0-844):
    State: RUNNABLE
    Blocked count: 6
    Waited count: 217
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 43049 (btpool0-842):
    State: RUNNABLE
    Blocked count: 4
    Waited count: 213
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 43042 (btpool0-835):
    State: RUNNABLE
    Blocked count: 12
    Waited count: 229
    Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.read(
    LocalFileSystem.java:113)
    org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(
    FSDataInputStream.java:189)
    java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
    java.io.BufferedInputStream.read(BufferedInputStream.java:313)
    java.io.DataInputStream.readFully(DataInputStream.java:176)
    java.io.DataInputStream.readFully(DataInputStream.java:152)
    org.apache.hadoop.fs.FSDataInputStream$Checker.<init>(
    FSDataInputStream.java:60)
    org.apache.hadoop.fs.FSDataInputStream.<init>(FSDataInputStream.java
    :279)
    org.apache.hadoop.fs.FileSystem.open(FileSystem.java:156)
    org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(
    TaskTracker.java:1409)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
    org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)
    org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:356)
    org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)
    org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)
    org.mortbay.jetty.handler.ContextHandlerCollection.handle(
    ContextHandlerCollection.java:149)
    Thread 40895 (btpool0-803):
    State: RUNNABLE
    Blocked count: 3
    Waited count: 361
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 40380 (btpool0-790):
    State: RUNNABLE
    Blocked count: 7
    Waited count: 391
    Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.read(
    LocalFileSystem.java:113)
    org.apache.hadoop.fs.FSDataInputStream$Checker.read(
    FSDataInputStream.java:91)
    org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(
    FSDataInputStream.java:189)
    java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
    java.io.BufferedInputStream.read(BufferedInputStream.java:313)
    java.io.DataInputStream.read(DataInputStream.java:80)
    org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(
    TaskTracker.java:1411)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
    org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)
    org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:356)
    org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)
    org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)
    org.mortbay.jetty.handler.ContextHandlerCollection.handle(
    ContextHandlerCollection.java:149)
    org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:141)
    org.mortbay.jetty.Server.handle(Server.java:269)
    org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:430)
    org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(
    HttpConnection.java:687)
    Thread 39121 (btpool0-784):
    State: RUNNABLE
    Blocked count: 16
    Waited count: 439
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 39112 (btpool0-775):
    State: RUNNABLE
    Blocked count: 13
    Waited count: 396
    Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(
    SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(
    SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 28 (org.apache.hadoop.dfs.DFSClient$LeaseChecker@1a7ddcf):
    State: TIMED_WAITING
    Blocked count: 0
    Waited count: 0
    Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:462)
    java.lang.Thread.run(Thread.java:595)
    Thread 19 (org.apache.hadoop.io.ObjectWritable Connection Culler):
    State: TIMED_WAITING
    Blocked count: 154
    Waited count: 0
    Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.ipc.Client$ConnectionCuller.run(Client.java:388)
    Thread 18 (IPC Server handler 4 on 50050):
    State: TIMED_WAITING
    Blocked count: 1769
    Waited count: 51161
    Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
    Thread 17 (IPC Server handler 3 on 50050):
    State: TIMED_WAITING
    Blocked count: 1673
    Waited count: 51202
    Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
    Thread 16 (IPC Server handler 2 on 50050):
    State: TIMED_WAITING
    Blocked count: 1685
    Waited count: 51188
    Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
    Thread 15 (IPC Server handler 1 on 50050):
    State: TIMED_WAITING
    Blocked count: 1694
    Waited count: 51183
    Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
    Thread 14 (IPC Server handler 0 on 50050):
    State: TIMED_WAITING
    Blocked count: 1913
    Waited count: 51137
    Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
    Thread 13 (IPC Server listener on 50050):
    State: RUNNABLE
    Blocked count: 634
    Waited count: 0
    Stack:
    sun.nio.ch.PollArrayWrapper.poll0(Native Method)
    sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
    sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener.run(Server.java:224)
    Thread 12 (btpool0-1 - Acceptor0 SocketConnector @ 0.0.0.0:50060):
    State: RUNNABLE
    Blocked count: 85
    Waited count: 1517
    Stack:
    java.net.PlainSocketImpl.socketAccept(Native Method)
    java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    java.net.ServerSocket.implAccept(ServerSocket.java:450)
    java.net.ServerSocket.accept(ServerSocket.java:421)
    org.mortbay.jetty.bio.SocketConnector.accept(SocketConnector.java:97)
    org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java
    :630)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 11 (btpool0-0 - Invalidator - /):
    State: TIMED_WAITING
    Blocked count: 0
    Waited count: 1
    Stack:
    java.lang.Thread.sleep(Native Method)
    org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(
    AbstractSessionManager.java:933)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(
    BoundedThreadPool.java:475)
    Thread 9 (taskCleanup):
    State: WAITING
    Blocked count: 457
    Waited count: 118
    Waiting on java.util.ArrayList@15192a
    Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:474)
    org.apache.hadoop.mapred.TaskTracker$BlockingQueue.take(TaskTracker.java
    :783)
    org.apache.hadoop.mapred.TaskTracker$1.run(TaskTracker.java:143)
    java.lang.Thread.run(Thread.java:595)
    Thread 4 (Signal Dispatcher):
    State: RUNNABLE
    Blocked count: 0
    Waited count: 0
    Stack:
    Thread 3 (Finalizer):
    State: WAITING
    Blocked count: 99047
    Waited count: 39648
    Waiting on java.lang.ref.ReferenceQueue$Lock@ac6214
    Stack:
    java.lang.Object.wait(Native Method)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
    java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
    Thread 2 (Reference Handler):
    State: WAITING
    Blocked count: 102986
    Waited count: 3131
    Waiting on java.lang.ref.Reference$Lock@15b40a1
    Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:474)
    java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    Thread 1 (main):
    State: RUNNABLE
    Blocked count: 1538
    Waited count: 107465
    Stack:
    sun.management.ThreadImpl.getThreadInfo0(Native Method)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:144)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:120)
    org.apache.hadoop.util.ReflectionUtils.printThreadInfo(
    ReflectionUtils.java:93)
    org.apache.hadoop.util.ReflectionUtils.logThreadInfo(
    ReflectionUtils.java:141)
    org.apache.hadoop.mapred.TaskTracker.markUnresponsiveTasks(
    TaskTracker.java:553)
    org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:449)
    org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:720)
    org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1374)


    /Johan

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedNov 17, '06 at 5:21p
activeNov 17, '06 at 7:58p
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Albert Chern: 1 post Johan Oskarsson: 1 post

People

Translate

site design / logo © 2022 Grokbase