Hello everyone,
I'm running a small toy cluster (3 nodes) on EC2 configured as follows:

* one node as JT+NN
* two nodes as DN+TT

I use whirr to build such cluster on demand (config file here
http://pastebin.com/JXHYvMNb). Since my jobs are memory intensive I'd
like to exploit the 8GB of ram the m1.large instance offers. Thus I
added mapred.map.child.java.opts=-Xmx1g and
mapred.reduce.child.java.opts=-Xmx1g (adding just
hadoop-mapreduce.mapred.child.java.opts=-Xmx1g produced no effect
since the map tasks were allocated the default 200MB). The problem is
that with these settings I cannot have any job running because I
always get

1/12/11 18:14:24 INFO mapred.JobClient: Running job: job_201112111644_0002
11/12/11 18:14:25 INFO mapred.JobClient: map 0% reduce 0%
11/12/11 18:14:28 INFO mapred.JobClient: Task Id :
attempt_201112111644_0002_m_000004_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)

11/12/11 18:14:28 WARN mapred.JobClient: Error reading task
outputip-10-87-1-170.ec2.internal
11/12/11 18:14:28 WARN mapred.JobClient: Error reading task
outputip-10-87-1-170.ec2.internal
11/12/11 18:14:30 INFO mapred.JobClient: Task Id :
attempt_201112111644_0002_r_000001_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)

And the only log I can get from the job_201112111644_0002 is the
stdout and the stderr whose combined output is

Could not create the Java virtual machine.
Error occurred during initialization of VM
Could not reserve enough space for object heap

I really cannot understand why the jvm cannot allocate enough space:
there's plenty of ram. I also tried to reduce the number of map slots
to two: nothing changed. I'm out of ideas. I hope you can shed some
light :)

FYI I use cloudera distribution for hadoop, latest stable release available.

Thanks for your attention.

MD

Search Discussions

  • Harsh J at Dec 11, 2011 at 8:38 pm
    Marco,

    Are you using a 64-bit JVM on your nodes or a 32-bit one?

    Sun JRE should say something like (for hadoop -version):
    "Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)"

    If you are, could you post what 'free' says on your slave nodes?
    On Sun, Dec 11, 2011 at 11:29 PM, Marco Didonna wrote:
    Hello everyone,
    I'm running a small toy cluster (3 nodes) on EC2 configured as follows:

    * one node as JT+NN
    * two nodes as DN+TT

    I use whirr to build such cluster on demand (config file here
    http://pastebin.com/JXHYvMNb). Since my jobs are memory intensive I'd
    like to exploit the 8GB of ram the m1.large instance offers. Thus I
    added mapred.map.child.java.opts=-Xmx1g and
    mapred.reduce.child.java.opts=-Xmx1g (adding just
    hadoop-mapreduce.mapred.child.java.opts=-Xmx1g produced no effect
    since the map tasks were allocated the default 200MB). The problem is
    that with these settings I cannot have any job running because I
    always get

    1/12/11 18:14:24 INFO mapred.JobClient: Running job: job_201112111644_0002
    11/12/11 18:14:25 INFO mapred.JobClient:  map 0% reduce 0%
    11/12/11 18:14:28 INFO mapred.JobClient: Task Id :
    attempt_201112111644_0002_m_000004_0, Status : FAILED
    java.lang.Throwable: Child Error
           at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
    Caused by: java.io.IOException: Task process exit with nonzero status of 1.
           at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)

    11/12/11 18:14:28 WARN mapred.JobClient: Error reading task
    outputip-10-87-1-170.ec2.internal
    11/12/11 18:14:28 WARN mapred.JobClient: Error reading task
    outputip-10-87-1-170.ec2.internal
    11/12/11 18:14:30 INFO mapred.JobClient: Task Id :
    attempt_201112111644_0002_r_000001_0, Status : FAILED
    java.lang.Throwable: Child Error
           at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
    Caused by: java.io.IOException: Task process exit with nonzero status of 1.
           at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)

    And the only log I can get from the job_201112111644_0002 is the
    stdout and the stderr whose combined output is

    Could not create the Java virtual machine.
    Error occurred during initialization of VM
    Could not reserve enough space for object heap

    I really cannot understand why the jvm cannot allocate enough space:
    there's plenty of ram. I also tried to reduce the number of map slots
    to two: nothing changed. I'm out of ideas. I hope you can shed some
    light :)

    FYI I use cloudera distribution for hadoop, latest stable release available.

    Thanks for your attention.

    MD


    --
    Harsh J
  • Marco Didonna at Dec 12, 2011 at 8:41 am
    Hi, thanks for your answer.
    I am indeed using a 64bit jvm on all nodes. In fact I get the
    following answer when executing java -version

    java version "1.6.0_26"
    Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
    Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

    And I do have a looot of free RAM:

    total used free shared buffers cached
    Mem: 7700 930 6769 0 14 462
    -/+ buffers/cache: 453 7246
    Swap: 0 0 0


    And still I cannot make use of such amount of ram. Again I tried to
    set mapred.child.java.opts to -Xmx1g but the map task got only the
    default amount of heap space (I detect this using the ps command).
    Only If i use mapred.{map|reduce}.child.java.opts=-Xmx1g this setting
    affects the job execution and I end up with the error we're trying to
    debug.
    The other non-default setting can be seen from the attached whirr
    configuration file (previous message).

    Thanks again,
    Marco
    On 11 December 2011 21:37, Harsh J wrote:
    Marco,

    Are you using a 64-bit JVM on your nodes or a 32-bit one?

    Sun JRE should say something like (for hadoop -version):
    "Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)"

    If you are, could you post what 'free' says on your slave nodes?
    On Sun, Dec 11, 2011 at 11:29 PM, Marco Didonna wrote:
    Hello everyone,
    I'm running a small toy cluster (3 nodes) on EC2 configured as follows:

    * one node as JT+NN
    * two nodes as DN+TT

    I use whirr to build such cluster on demand (config file here
    http://pastebin.com/JXHYvMNb). Since my jobs are memory intensive I'd
    like to exploit the 8GB of ram the m1.large instance offers. Thus I
    added mapred.map.child.java.opts=-Xmx1g and
    mapred.reduce.child.java.opts=-Xmx1g (adding just
    hadoop-mapreduce.mapred.child.java.opts=-Xmx1g produced no effect
    since the map tasks were allocated the default 200MB). The problem is
    that with these settings I cannot have any job running because I
    always get

    1/12/11 18:14:24 INFO mapred.JobClient: Running job: job_201112111644_0002
    11/12/11 18:14:25 INFO mapred.JobClient:  map 0% reduce 0%
    11/12/11 18:14:28 INFO mapred.JobClient: Task Id :
    attempt_201112111644_0002_m_000004_0, Status : FAILED
    java.lang.Throwable: Child Error
           at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
    Caused by: java.io.IOException: Task process exit with nonzero status of 1.
           at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)

    11/12/11 18:14:28 WARN mapred.JobClient: Error reading task
    outputip-10-87-1-170.ec2.internal
    11/12/11 18:14:28 WARN mapred.JobClient: Error reading task
    outputip-10-87-1-170.ec2.internal
    11/12/11 18:14:30 INFO mapred.JobClient: Task Id :
    attempt_201112111644_0002_r_000001_0, Status : FAILED
    java.lang.Throwable: Child Error
           at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
    Caused by: java.io.IOException: Task process exit with nonzero status of 1.
           at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)

    And the only log I can get from the job_201112111644_0002 is the
    stdout and the stderr whose combined output is

    Could not create the Java virtual machine.
    Error occurred during initialization of VM
    Could not reserve enough space for object heap

    I really cannot understand why the jvm cannot allocate enough space:
    there's plenty of ram. I also tried to reduce the number of map slots
    to two: nothing changed. I'm out of ideas. I hope you can shed some
    light :)

    FYI I use cloudera distribution for hadoop, latest stable release available.

    Thanks for your attention.

    MD


    --
    Harsh J

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedDec 11, '11 at 6:00p
activeDec 12, '11 at 8:41a
posts3
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Marco Didonna: 2 posts Harsh J: 1 post

People

Translate

site design / logo © 2022 Grokbase