Actually I enabled all level logs. But I didn't realize to check logs in .out files and only looked at .log file and didn't see any error msgs. now I opened the .out file and saw the following logged exception:
Exception in thread "IPC Server handler 5 on 50002" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:133)
at org.apache.hadoop.ipc.Server.setupResponse(Server.java:1087)
at org.apache.hadoop.ipc.Server.access$2400(Server.java:77)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:984)
Exception in thread "IPC Client (47) connection to servername/serverip:50001 from auser" java.lang.OutOfMemoryError: Java heap space
at java.nio.ByteBuffer.wrap(ByteBuffer.java:350)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.DataInputStream.readInt(DataInputStream.java:370)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
Exception in thread "LeaseChecker" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "IPC Server handler 4 on 50002" java.lang.OutOfMemoryError: Java heap space
Exception in thread "IPC Server handler 8 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "IPC Server handler 9 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "IPC Server handler 2 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "IPC Server handler 6 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "ResponseProcessor for block blk_-573819335330670501_1268378" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "IPC Server handler 7 on 50002" java.lang.OutOfMemoryError: Java heap space
50001 and 50002 used for dfs and jobtracker. But it's normal that IPC threads consume such big memory?
It may not be IPC threads that are causing the problem, though IException in thread "IPC Server handler 5 on 50002" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:133)
at org.apache.hadoop.ipc.Server.setupResponse(Server.java:1087)
at org.apache.hadoop.ipc.Server.access$2400(Server.java:77)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:984)
Exception in thread "IPC Client (47) connection to servername/serverip:50001 from auser" java.lang.OutOfMemoryError: Java heap space
at java.nio.ByteBuffer.wrap(ByteBuffer.java:350)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.DataInputStream.readInt(DataInputStream.java:370)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
Exception in thread "LeaseChecker" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "IPC Server handler 4 on 50002" java.lang.OutOfMemoryError: Java heap space
Exception in thread "IPC Server handler 8 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "IPC Server handler 9 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "IPC Server handler 2 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "IPC Server handler 6 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "ResponseProcessor for block blk_-573819335330670501_1268378" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "IPC Server handler 7 on 50002" java.lang.OutOfMemoryError: Java heap space
50001 and 50002 used for dfs and jobtracker. But it's normal that IPC threads consume such big memory?
remember at least one IPC buffer related OOM somewhere. A more typical
case is that the daemon is running out of memory and the IPC threads
are just manifesting the error because they're the entry-points into
the application. I think we still need to go down the path that Alex
was suggesting. What is the heap you have given the JobTracker ? What
is number of Jobs / tasks / users ? How many nodes are there in the
cluster ?
--- On Fri, 7/30/10, Alex Loddengaard wrote:
From: Alex Loddengaard <alex@cloudera.com>
Subject: Re: jobtracker.jsp reports "GC overhead limit exceeded"
To: common-user@hadoop.apache.org
Date: Friday, July 30, 2010, 2:19 PM
What does "ps" show you? How much memory is being used by the jobtracker,
and how large is its heap (loop for HADOOP_HEAPSIZE in hadoop-env.sh)? Also
consider turning on GC logging, which will find its way to the jobtracker
.out log in /var/log/hadoop:
<http://java.sun.com/developer/technicalArticles/Programming/GCPortal/>
Alex
From: Alex Loddengaard <alex@cloudera.com>
Subject: Re: jobtracker.jsp reports "GC overhead limit exceeded"
To: common-user@hadoop.apache.org
Date: Friday, July 30, 2010, 2:19 PM
What does "ps" show you? How much memory is being used by the jobtracker,
and how large is its heap (loop for HADOOP_HEAPSIZE in hadoop-env.sh)? Also
consider turning on GC logging, which will find its way to the jobtracker
.out log in /var/log/hadoop:
<http://java.sun.com/developer/technicalArticles/Programming/GCPortal/>
Alex
On Fri, Jul 30, 2010 at 3:10 PM, jiang licht wrote:
http://server:50030/jobtracker.jsp generates the following error message:
HTTP ERROR: 500
GC overhead limit exceeded
RequestURI=/jobtracker.jsp
Caused by:
java.lang.OutOfMemoryError: GC overhead limit exceeded
Powered by Jetty://
The jobtracker is running below the limit. But "hadoop job -status" seems
to halt and does not response ...
The last 2 lines of jobtracker logs:
2010-07-30 13:53:18,482 DEBUG org.apache.hadoop.mapred.JobTracker: Got
heartbeat from: tracker_host1:localhost.localdomain/127.0.0.1:53914(restarted: false initialContact: false acceptNewTasks: true) with
responseId: -31252
2010-07-30 13:55:32,917 DEBUG org.apache.hadoop.mapred.JobTracker: Starting
launching task sweep
Any thought about this?
Thanks!
--Michael
http://server:50030/jobtracker.jsp generates the following error message:
HTTP ERROR: 500
GC overhead limit exceeded
RequestURI=/jobtracker.jsp
Caused by:
java.lang.OutOfMemoryError: GC overhead limit exceeded
Powered by Jetty://
The jobtracker is running below the limit. But "hadoop job -status" seems
to halt and does not response ...
The last 2 lines of jobtracker logs:
2010-07-30 13:53:18,482 DEBUG org.apache.hadoop.mapred.JobTracker: Got
heartbeat from: tracker_host1:localhost.localdomain/127.0.0.1:53914(restarted: false initialContact: false acceptNewTasks: true) with
responseId: -31252
2010-07-30 13:55:32,917 DEBUG org.apache.hadoop.mapred.JobTracker: Starting
launching task sweep
Any thought about this?
Thanks!
--Michael
