FAQ
Hi all,



We are using hadoop-0.17.2 for some time now. Since yesterday, We have
been seeing jobTracker failing to respond with an OutOfMemory Error very
frequently. Things are going fine after restarting it. But the problem
is occurring after a while. Below is the exception that we are seeing in
jobtracker logs. Can someone please suggest what is going wrong in
this?





2008-11-19 14:17:46,059 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 9 on 9001, call
heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@2b152f32, false,
true, 16068)

from 205.188.170.107:51492: error: java.io.IOException:
java.lang.OutOfMemoryError: Java heap space

java.io.IOException: java.lang.OutOfMemoryError: Java heap space

at java.util.regex.Pattern.compile(Pattern.java:1452)

at java.util.regex.Pattern.(Pattern.java:847)

at java.lang.String.replace(String.java:2208)

at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)

at org.apache.hadoop.fs.Path.initialize(Path.java:137)

at org.apache.hadoop.fs.Path.(Path.java:50)

at
org.apache.hadoop.mapred.Task.getTaskOutputPath(Task.java:214)

at org.apache.hadoop.mapred.Task.setConf(Task.java:517)

at
org.apache.hadoop.mapred.TaskInProgress.getTaskToRun(TaskInProgress.java
:745)

at
org.apache.hadoop.mapred.JobInProgress.obtainNewMapTask(JobInProgress.ja
va:664)

at
org.apache.hadoop.mapred.JobTracker.getNewTaskForTaskTracker(JobTracker.
java:1585)

at
org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:1309)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)

at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

2008-11-19 14:18:11,869 WARN org.apache.hadoop.ipc.Server: IPC Server
handler 4 on 9001, call
heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@24d86be, false,
true, 16077) f

rom 205.188.170.84:54871: discarded for being too old (133957)

2008-11-19 14:18:11,869 WARN org.apache.hadoop.ipc.Server: IPC Server
handler 4 on 9001, call
heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@3ea47062, false,
true, 16082)

from 205.188.170.90:32934: discarded for being too old (133957)




Thanks

Pallavi

Search Discussions

  • Charles du at Dec 5, 2008 at 6:58 pm
    Any update on this?

    We got a similar problem after we ran a hadoop job with a lot of mappers.
    Restarting jobtracker solved the problem for a few times. But right now, we
    got the out of memory error right after we restarted the jobtracker. Thanks.




    On Wed, Nov 19, 2008 at 8:40 PM, Palleti, Pallavi wrote:

    Hi all,



    We are using hadoop-0.17.2 for some time now. Since yesterday, We have
    been seeing jobTracker failing to respond with an OutOfMemory Error very
    frequently. Things are going fine after restarting it. But the problem
    is occurring after a while. Below is the exception that we are seeing in
    jobtracker logs. Can someone please suggest what is going wrong in
    this?





    2008-11-19 14:17:46,059 INFO org.apache.hadoop.ipc.Server: IPC Server
    handler 9 on 9001, call
    heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@2b152f32, false,
    true, 16068)

    from 205.188.170.107:51492: error: java.io.IOException:
    java.lang.OutOfMemoryError: Java heap space

    java.io.IOException: java.lang.OutOfMemoryError: Java heap space

    at java.util.regex.Pattern.compile(Pattern.java:1452)

    at java.util.regex.Pattern.<init>(Pattern.java:1133)

    at java.util.regex.Pattern.compile(Pattern.java:847)

    at java.lang.String.replace(String.java:2208)

    at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)

    at org.apache.hadoop.fs.Path.initialize(Path.java:137)

    at org.apache.hadoop.fs.Path.<init>(Path.java:126)

    at org.apache.hadoop.fs.Path.<init>(Path.java:50)

    at
    org.apache.hadoop.mapred.Task.getTaskOutputPath(Task.java:214)

    at org.apache.hadoop.mapred.Task.setConf(Task.java:517)

    at
    org.apache.hadoop.mapred.TaskInProgress.getTaskToRun(TaskInProgress.java
    :745)

    at
    org.apache.hadoop.mapred.JobInProgress.obtainNewMapTask(JobInProgress.ja
    va:664)

    at
    org.apache.hadoop.mapred.JobTracker.getNewTaskForTaskTracker(JobTracker.
    java:1585)

    at
    org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:1309)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)

    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

    2008-11-19 14:18:11,869 WARN org.apache.hadoop.ipc.Server: IPC Server
    handler 4 on 9001, call
    heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@24d86be, false,
    true, 16077) f

    rom 205.188.170.84:54871: discarded for being too old (133957)

    2008-11-19 14:18:11,869 WARN org.apache.hadoop.ipc.Server: IPC Server
    handler 4 on 9001, call
    heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@3ea47062, false,
    true, 16082)

    from 205.188.170.90:32934: discarded for being too old (133957)




    Thanks

    Pallavi

    --
    tp
  • Charles du at Dec 5, 2008 at 7:03 pm
    I found the following error message in
    hadoop-middleware-jobtracker-dd-9c32d01.off.tn.ask.com.out

    Java HotSpot(TM) Server VM warning: Exception java.lang.OutOfMemoryError
    occurred dispatching signal SIGTERM to handler- the VM may need to be
    forcibly terminated

    On Fri, Dec 5, 2008 at 10:58 AM, charles du wrote:

    Any update on this?

    We got a similar problem after we ran a hadoop job with a lot of mappers.
    Restarting jobtracker solved the problem for a few times. But right now, we
    got the out of memory error right after we restarted the jobtracker. Thanks.





    On Wed, Nov 19, 2008 at 8:40 PM, Palleti, Pallavi <
    pallavi.palleti@corp.aol.com> wrote:
    Hi all,



    We are using hadoop-0.17.2 for some time now. Since yesterday, We have
    been seeing jobTracker failing to respond with an OutOfMemory Error very
    frequently. Things are going fine after restarting it. But the problem
    is occurring after a while. Below is the exception that we are seeing in
    jobtracker logs. Can someone please suggest what is going wrong in
    this?





    2008-11-19 14:17:46,059 INFO org.apache.hadoop.ipc.Server: IPC Server
    handler 9 on 9001, call
    heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@2b152f32, false,
    true, 16068)

    from 205.188.170.107:51492: error: java.io.IOException:
    java.lang.OutOfMemoryError: Java heap space

    java.io.IOException: java.lang.OutOfMemoryError: Java heap space

    at java.util.regex.Pattern.compile(Pattern.java:1452)

    at java.util.regex.Pattern.<init>(Pattern.java:1133)

    at java.util.regex.Pattern.compile(Pattern.java:847)

    at java.lang.String.replace(String.java:2208)

    at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)

    at org.apache.hadoop.fs.Path.initialize(Path.java:137)

    at org.apache.hadoop.fs.Path.<init>(Path.java:126)

    at org.apache.hadoop.fs.Path.<init>(Path.java:50)

    at
    org.apache.hadoop.mapred.Task.getTaskOutputPath(Task.java:214)

    at org.apache.hadoop.mapred.Task.setConf(Task.java:517)

    at
    org.apache.hadoop.mapred.TaskInProgress.getTaskToRun(TaskInProgress.java
    :745)

    at
    org.apache.hadoop.mapred.JobInProgress.obtainNewMapTask(JobInProgress.ja
    va:664)

    at
    org.apache.hadoop.mapred.JobTracker.getNewTaskForTaskTracker(JobTracker.
    java:1585)

    at
    org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:1309)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)

    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

    2008-11-19 14:18:11,869 WARN org.apache.hadoop.ipc.Server: IPC Server
    handler 4 on 9001, call
    heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@24d86be, false,
    true, 16077) f

    rom 205.188.170.84:54871: discarded for being too old (133957)

    2008-11-19 14:18:11,869 WARN org.apache.hadoop.ipc.Server: IPC Server
    handler 4 on 9001, call
    heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@3ea47062, false,
    true, 16082)

    from 205.188.170.90:32934: discarded for being too old (133957)




    Thanks

    Pallavi

    --
    tp


    --
    tp
  • Arun C Murthy at Dec 6, 2008 at 9:46 am

    On Dec 5, 2008, at 10:58 AM, charles du wrote:

    Any update on this?
    What is the available heapsize for the JobTracker? (HADOOP_HEAPSIZE or
    set it in HADOOP_JOBTRACKER_OPTS in conf/hadoop-env.sh).

    Do you remember how many total tasks (across all jobs) were executed
    before the OOM?

    Arun
    We got a similar problem after we ran a hadoop job with a lot of
    mappers.
    Restarting jobtracker solved the problem for a few times. But right
    now, we
    got the out of memory error right after we restarted the jobtracker.
    Thanks.





    On Wed, Nov 19, 2008 at 8:40 PM, Palleti, Pallavi <
    pallavi.palleti@corp.aol.com> wrote:
    Hi all,



    We are using hadoop-0.17.2 for some time now. Since yesterday, We
    have
    been seeing jobTracker failing to respond with an OutOfMemory Error
    very
    frequently. Things are going fine after restarting it. But the
    problem
    is occurring after a while. Below is the exception that we are
    seeing in
    jobtracker logs. Can someone please suggest what is going wrong in
    this?





    2008-11-19 14:17:46,059 INFO org.apache.hadoop.ipc.Server: IPC Server
    handler 9 on 9001, call
    heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@2b152f32, false,
    true, 16068)

    from 205.188.170.107:51492: error: java.io.IOException:
    java.lang.OutOfMemoryError: Java heap space

    java.io.IOException: java.lang.OutOfMemoryError: Java heap space

    at java.util.regex.Pattern.compile(Pattern.java:1452)

    at java.util.regex.Pattern.<init>(Pattern.java:1133)

    at java.util.regex.Pattern.compile(Pattern.java:847)

    at java.lang.String.replace(String.java:2208)

    at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)

    at org.apache.hadoop.fs.Path.initialize(Path.java:137)

    at org.apache.hadoop.fs.Path.<init>(Path.java:126)

    at org.apache.hadoop.fs.Path.<init>(Path.java:50)

    at
    org.apache.hadoop.mapred.Task.getTaskOutputPath(Task.java:214)

    at org.apache.hadoop.mapred.Task.setConf(Task.java:517)

    at
    org
    .apache.hadoop.mapred.TaskInProgress.getTaskToRun(TaskInProgress.java
    :745)

    at
    org
    .apache.hadoop.mapred.JobInProgress.obtainNewMapTask(JobInProgress.ja
    va:664)

    at
    org
    .apache.hadoop.mapred.JobTracker.getNewTaskForTaskTracker(JobTracker.
    java:1585)

    at
    org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:1309)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun
    .reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun
    .reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)

    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

    2008-11-19 14:18:11,869 WARN org.apache.hadoop.ipc.Server: IPC Server
    handler 4 on 9001, call
    heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@24d86be, false,
    true, 16077) f

    rom 205.188.170.84:54871: discarded for being too old (133957)

    2008-11-19 14:18:11,869 WARN org.apache.hadoop.ipc.Server: IPC Server
    handler 4 on 9001, call
    heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@3ea47062, false,
    true, 16082)

    from 205.188.170.90:32934: discarded for being too old (133957)




    Thanks

    Pallavi

    --
    tp
  • Charles du at Dec 6, 2008 at 7:41 pm
    I used the default value, which I believe is 1000 MB. My cluster has about
    30 machines. Each machine is configured to run up to 5 tasks. We run hourly,
    daily jobs on the cluster. When OOM happened, I was running a job with 1500
    - 1600 mappers and 40 reducers.

    I noticed that the memory usage of the job tracker keeps getting up. In one
    or two days, the job tracker uses about 1Gbytes memory, and stops responding
    to any request. Thanks.


    On Sat, Dec 6, 2008 at 1:45 AM, Arun C Murthy wrote:


    On Dec 5, 2008, at 10:58 AM, charles du wrote:

    Any update on this?
    What is the available heapsize for the JobTracker? (HADOOP_HEAPSIZE or set
    it in HADOOP_JOBTRACKER_OPTS in conf/hadoop-env.sh).

    Do you remember how many total tasks (across all jobs) were executed before
    the OOM?

    Arun


    We got a similar problem after we ran a hadoop job with a lot of mappers.
    Restarting jobtracker solved the problem for a few times. But right now,
    we
    got the out of memory error right after we restarted the jobtracker.
    Thanks.





    On Wed, Nov 19, 2008 at 8:40 PM, Palleti, Pallavi <
    pallavi.palleti@corp.aol.com> wrote:

    Hi all,


    We are using hadoop-0.17.2 for some time now. Since yesterday, We have
    been seeing jobTracker failing to respond with an OutOfMemory Error very
    frequently. Things are going fine after restarting it. But the problem
    is occurring after a while. Below is the exception that we are seeing in
    jobtracker logs. Can someone please suggest what is going wrong in
    this?





    2008-11-19 14:17:46,059 INFO org.apache.hadoop.ipc.Server: IPC Server
    handler 9 on 9001, call
    heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@2b152f32, false,
    true, 16068)

    from 205.188.170.107:51492: error: java.io.IOException:
    java.lang.OutOfMemoryError: Java heap space

    java.io.IOException: java.lang.OutOfMemoryError: Java heap space

    at java.util.regex.Pattern.compile(Pattern.java:1452)

    at java.util.regex.Pattern.<init>(Pattern.java:1133)

    at java.util.regex.Pattern.compile(Pattern.java:847)

    at java.lang.String.replace(String.java:2208)

    at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)

    at org.apache.hadoop.fs.Path.initialize(Path.java:137)

    at org.apache.hadoop.fs.Path.<init>(Path.java:126)

    at org.apache.hadoop.fs.Path.<init>(Path.java:50)

    at
    org.apache.hadoop.mapred.Task.getTaskOutputPath(Task.java:214)

    at org.apache.hadoop.mapred.Task.setConf(Task.java:517)

    at
    org.apache.hadoop.mapred.TaskInProgress.getTaskToRun(TaskInProgress.java
    :745)

    at
    org.apache.hadoop.mapred.JobInProgress.obtainNewMapTask(JobInProgress.ja
    va:664)

    at
    org.apache.hadoop.mapred.JobTracker.getNewTaskForTaskTracker(JobTracker.
    java:1585)

    at
    org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:1309)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)

    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

    2008-11-19 14:18:11,869 WARN org.apache.hadoop.ipc.Server: IPC Server
    handler 4 on 9001, call
    heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@24d86be, false,
    true, 16077) f

    rom 205.188.170.84:54871: discarded for being too old (133957)

    2008-11-19 14:18:11,869 WARN org.apache.hadoop.ipc.Server: IPC Server
    handler 4 on 9001, call
    heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@3ea47062, false,
    true, 16082)

    from 205.188.170.90:32934: discarded for being too old (133957)




    Thanks

    Pallavi

    --
    tp

    --
    tp
  • Arun C Murthy at Dec 6, 2008 at 7:56 pm

    On Dec 6, 2008, at 11:40 AM, charles du wrote:

    I used the default value, which I believe is 1000 MB. My cluster has
    about
    30 machines. Each machine is configured to run up to 5 tasks. We run
    hourly,
    daily jobs on the cluster. When OOM happened, I was running a job
    with 1500
    - 1600 mappers and 40 reducers.

    I noticed that the memory usage of the job tracker keeps getting
    up. In one
    or two days, the job tracker uses about 1Gbytes memory, and stops
    responding
    to any request. Thanks.
    Do you know how many total tasks (across all jobs) were executed in
    the day or two by the JT?

    Couple of workarounds:
    1. Move to hadoop-0.18 - we've fixed https://issues.apache.org/jira/browse/HADOOP-3670
    .
    2. Increase the JT's heapsize to 2G or 3G.

    Arun
  • Charles du at Dec 7, 2008 at 10:48 pm
    Thanks for the information. It helps a lot.
    On Sat, Dec 6, 2008 at 11:54 AM, Arun C Murthy wrote:


    On Dec 6, 2008, at 11:40 AM, charles du wrote:

    I used the default value, which I believe is 1000 MB. My cluster has about
    30 machines. Each machine is configured to run up to 5 tasks. We run
    hourly,
    daily jobs on the cluster. When OOM happened, I was running a job with
    1500
    - 1600 mappers and 40 reducers.

    I noticed that the memory usage of the job tracker keeps getting up. In
    one
    or two days, the job tracker uses about 1Gbytes memory, and stops
    responding
    to any request. Thanks.

    Do you know how many total tasks (across all jobs) were executed in the day
    or two by the JT?

    Couple of workarounds:
    1. Move to hadoop-0.18 - we've fixed
    https://issues.apache.org/jira/browse/HADOOP-3670.
    2. Increase the JT's heapsize to 2G or 3G.

    Arun

    --
    tp

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedNov 20, '08 at 4:43a
activeDec 7, '08 at 10:48p
posts7
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase