FAQ
http://server:50030/jobtracker.jsp generates the following error message:

HTTP ERROR: 500

GC overhead limit exceeded

RequestURI=/jobtracker.jsp
Caused by:

java.lang.OutOfMemoryError: GC overhead limit exceeded

Powered by Jetty://

The jobtracker is running below the limit. But "hadoop job -status" seems to halt and does not response ...

The last 2 lines of jobtracker logs:

2010-07-30 13:53:18,482 DEBUG org.apache.hadoop.mapred.JobTracker: Got heartbeat from: tracker_host1:localhost.localdomain/127.0.0.1:53914 (restarted: false initialContact: false acceptNewTasks: true) with responseId: -31252
2010-07-30 13:55:32,917 DEBUG org.apache.hadoop.mapred.JobTracker: Starting launching task sweep

Any thought about this?

Thanks!
--Michael

Search Discussions

  • Alex Loddengaard at Jul 30, 2010 at 7:20 pm
    What does "ps" show you? How much memory is being used by the jobtracker,
    and how large is its heap (loop for HADOOP_HEAPSIZE in hadoop-env.sh)? Also
    consider turning on GC logging, which will find its way to the jobtracker
    .out log in /var/log/hadoop:

    <http://java.sun.com/developer/technicalArticles/Programming/GCPortal/>

    Alex
    On Fri, Jul 30, 2010 at 3:10 PM, jiang licht wrote:

    http://server:50030/jobtracker.jsp generates the following error message:

    HTTP ERROR: 500

    GC overhead limit exceeded

    RequestURI=/jobtracker.jsp
    Caused by:

    java.lang.OutOfMemoryError: GC overhead limit exceeded

    Powered by Jetty://

    The jobtracker is running below the limit. But "hadoop job -status" seems
    to halt and does not response ...

    The last 2 lines of jobtracker logs:

    2010-07-30 13:53:18,482 DEBUG org.apache.hadoop.mapred.JobTracker: Got
    heartbeat from: tracker_host1:localhost.localdomain/127.0.0.1:53914(restarted: false initialContact: false acceptNewTasks: true) with
    responseId: -31252
    2010-07-30 13:55:32,917 DEBUG org.apache.hadoop.mapred.JobTracker: Starting
    launching task sweep

    Any thought about this?

    Thanks!
    --Michael

  • Alex Loddengaard at Jul 30, 2010 at 7:20 pm
    err, "ps aux", not "ps".

    Alex
    On Fri, Jul 30, 2010 at 3:19 PM, Alex Loddengaard wrote:

    What does "ps" show you? How much memory is being used by the jobtracker,
    and how large is its heap (loop for HADOOP_HEAPSIZE in hadoop-env.sh)? Also
    consider turning on GC logging, which will find its way to the jobtracker
    .out log in /var/log/hadoop:

    <http://java.sun.com/developer/technicalArticles/Programming/GCPortal/>

    Alex

    On Fri, Jul 30, 2010 at 3:10 PM, jiang licht wrote:

    http://server:50030/jobtracker.jsp generates the following error message:

    HTTP ERROR: 500

    GC overhead limit exceeded

    RequestURI=/jobtracker.jsp
    Caused by:

    java.lang.OutOfMemoryError: GC overhead limit exceeded

    Powered by Jetty://

    The jobtracker is running below the limit. But "hadoop job -status" seems
    to halt and does not response ...

    The last 2 lines of jobtracker logs:

    2010-07-30 13:53:18,482 DEBUG org.apache.hadoop.mapred.JobTracker: Got
    heartbeat from: tracker_host1:localhost.localdomain/127.0.0.1:53914(restarted: false initialContact: false acceptNewTasks: true) with
    responseId: -31252
    2010-07-30 13:55:32,917 DEBUG org.apache.hadoop.mapred.JobTracker:
    Starting launching task sweep

    Any thought about this?

    Thanks!
    --Michael

  • Jiang licht at Jul 30, 2010 at 8:21 pm
    Thanks, Alex.

    Actually I enabled all level logs. But I didn't realize to check logs in .out files and only looked at .log file and didn't see any error msgs. now I opened the .out file and saw the following logged exception:

    Exception in thread "IPC Server handler 5 on 50002" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2786)
    at java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:133)
    at org.apache.hadoop.ipc.Server.setupResponse(Server.java:1087)
    at org.apache.hadoop.ipc.Server.access$2400(Server.java:77)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:984)

    Exception in thread "IPC Client (47) connection to servername/serverip:50001 from auser" java.lang.OutOfMemoryError: Java heap space
    at java.nio.ByteBuffer.wrap(ByteBuffer.java:350)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
    at java.io.FilterInputStream.read(FilterInputStream.java:116)
    at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
    at java.io.DataInputStream.readInt(DataInputStream.java:370)
    at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)

    Exception in thread "LeaseChecker" java.lang.OutOfMemoryError: GC overhead limit exceeded
    Exception in thread "IPC Server handler 4 on 50002" java.lang.OutOfMemoryError: Java heap space
    Exception in thread "IPC Server handler 8 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
    Exception in thread "IPC Server handler 9 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
    Exception in thread "IPC Server handler 2 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
    Exception in thread "IPC Server handler 6 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
    Exception in thread "ResponseProcessor for block blk_-573819335330670501_1268378" java.lang.OutOfMemoryError: GC overhead limit exceeded
    Exception in thread "IPC Server handler 7 on 50002" java.lang.OutOfMemoryError: Java heap space

    50001 and 50002 used for dfs and jobtracker. But it's normal that IPC threads consume such big memory?

    Thanks,

    Michael

    --- On Fri, 7/30/10, Alex Loddengaard wrote:

    From: Alex Loddengaard <alex@cloudera.com>
    Subject: Re: jobtracker.jsp reports "GC overhead limit exceeded"
    To: common-user@hadoop.apache.org
    Date: Friday, July 30, 2010, 2:19 PM

    What does "ps" show you?  How much memory is being used by the jobtracker,
    and how large is its heap (loop for HADOOP_HEAPSIZE in hadoop-env.sh)?  Also
    consider turning on GC logging, which will find its way to the jobtracker
    .out log in /var/log/hadoop:

    <http://java.sun.com/developer/technicalArticles/Programming/GCPortal/>

    Alex
    On Fri, Jul 30, 2010 at 3:10 PM, jiang licht wrote:

    http://server:50030/jobtracker.jsp generates the following error message:

    HTTP ERROR: 500

    GC overhead limit exceeded

    RequestURI=/jobtracker.jsp
    Caused by:

    java.lang.OutOfMemoryError: GC overhead limit exceeded

    Powered by Jetty://

    The jobtracker is running below the limit. But "hadoop job -status" seems
    to halt and does not response ...

    The last 2 lines of jobtracker logs:

    2010-07-30 13:53:18,482 DEBUG org.apache.hadoop.mapred.JobTracker: Got
    heartbeat from: tracker_host1:localhost.localdomain/127.0.0.1:53914(restarted: false initialContact: false acceptNewTasks: true) with
    responseId: -31252
    2010-07-30 13:55:32,917 DEBUG org.apache.hadoop.mapred.JobTracker: Starting
    launching task sweep

    Any thought about this?

    Thanks!
    --Michael

  • Hemanth Yamijala at Aug 2, 2010 at 5:01 am
    Hi,
    Actually I enabled all level logs. But I didn't realize to check logs in .out files and only looked at .log file and didn't see any error msgs. now I opened the .out file and saw the following logged exception:

    Exception in thread "IPC Server handler 5 on 50002" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2786)
    at java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:133)
    at org.apache.hadoop.ipc.Server.setupResponse(Server.java:1087)
    at org.apache.hadoop.ipc.Server.access$2400(Server.java:77)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:984)

    Exception in thread "IPC Client (47) connection to servername/serverip:50001 from auser" java.lang.OutOfMemoryError: Java heap space
    at java.nio.ByteBuffer.wrap(ByteBuffer.java:350)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
    at java.io.FilterInputStream.read(FilterInputStream.java:116)
    at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
    at java.io.DataInputStream.readInt(DataInputStream.java:370)
    at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)

    Exception in thread "LeaseChecker" java.lang.OutOfMemoryError: GC overhead limit exceeded
    Exception in thread "IPC Server handler 4 on 50002" java.lang.OutOfMemoryError: Java heap space
    Exception in thread "IPC Server handler 8 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
    Exception in thread "IPC Server handler 9 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
    Exception in thread "IPC Server handler 2 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
    Exception in thread "IPC Server handler 6 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
    Exception in thread "ResponseProcessor for block blk_-573819335330670501_1268378" java.lang.OutOfMemoryError: GC overhead limit exceeded
    Exception in thread "IPC Server handler 7 on 50002" java.lang.OutOfMemoryError: Java heap space

    50001 and 50002 used for dfs and jobtracker. But it's normal that IPC threads consume such big memory?
    It may not be IPC threads that are causing the problem, though I
    remember at least one IPC buffer related OOM somewhere. A more typical
    case is that the daemon is running out of memory and the IPC threads
    are just manifesting the error because they're the entry-points into
    the application. I think we still need to go down the path that Alex
    was suggesting. What is the heap you have given the JobTracker ? What
    is number of Jobs / tasks / users ? How many nodes are there in the
    cluster ?
    --- On Fri, 7/30/10, Alex Loddengaard wrote:

    From: Alex Loddengaard <alex@cloudera.com>
    Subject: Re: jobtracker.jsp reports "GC overhead limit exceeded"
    To: common-user@hadoop.apache.org
    Date: Friday, July 30, 2010, 2:19 PM

    What does "ps" show you?  How much memory is being used by the jobtracker,
    and how large is its heap (loop for HADOOP_HEAPSIZE in hadoop-env.sh)?  Also
    consider turning on GC logging, which will find its way to the jobtracker
    .out log in /var/log/hadoop:

    <http://java.sun.com/developer/technicalArticles/Programming/GCPortal/>

    Alex
    On Fri, Jul 30, 2010 at 3:10 PM, jiang licht wrote:

    http://server:50030/jobtracker.jsp generates the following error message:

    HTTP ERROR: 500

    GC overhead limit exceeded

    RequestURI=/jobtracker.jsp
    Caused by:

    java.lang.OutOfMemoryError: GC overhead limit exceeded

    Powered by Jetty://

    The jobtracker is running below the limit. But "hadoop job -status" seems
    to halt and does not response ...

    The last 2 lines of jobtracker logs:

    2010-07-30 13:53:18,482 DEBUG org.apache.hadoop.mapred.JobTracker: Got
    heartbeat from: tracker_host1:localhost.localdomain/127.0.0.1:53914(restarted: false initialContact: false acceptNewTasks: true) with
    responseId: -31252
    2010-07-30 13:55:32,917 DEBUG org.apache.hadoop.mapred.JobTracker: Starting
    launching task sweep

    Any thought about this?

    Thanks!
    --Michael


Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJul 30, '10 at 7:10p
activeAug 2, '10 at 5:01a
posts5
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase