FAQ
Hej

I am running an aplication based on Nutch in a cluster with 1 master and 6
slaves in Amazon (with the same instances for all of them with 1.7GB RAM
memory)

My configuration is the following:

HADOOP_HEAPSIZE=1300
HADOOP_NAMENODE_OPTS=-Xmx400m
HADOOP_SECONDARYNAMENODE_OPTS=-Xmx400m
HADOOP_JOBTRACKER_OPTS=-Xmx400m
dfs.replication=3
mapred.map.tasks=6
mapred.reduce.tasks=6
mapred.child.java.opts=-Xmx950m

But in the second depth fetch, I got the following errors in some instances
(while the other ones seems they fetched correctly) :

2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - java.io.IOException: Spill
failed
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:822)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:466)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.nutch.fetcher.Fetcher$FetcherThread.output(Fetcher.java:907)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:670)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - Caused by:
java.io.IOException: Cannot run program "bash": java.io.IOException:
error=12, Cannot allocate memory
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.util.Shell.run(Shell.java:134)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.fs.DF.getAvailable(DF.java:73)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1183)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:648)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1135)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - Caused by:
java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
java.lang.ProcessImpl.start(ProcessImpl.java:65)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - at
java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
2010-02-10 03:18:31,185 FATAL fetcher.Fetcher - ... 9 more
.
.
.
.
.
2010-02-10 03:18:31,463 WARN mapred.TaskTracker - Error running child
java.io.IOException: Spill failed
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1085)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:359)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.io.IOException: Cannot run program "bash":
java.io.IOException: error=12, Cannot allocate memory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
at org.apache.hadoop.util.Shell.run(Shell.java:134)
at org.apache.hadoop.fs.DF.getAvailable(DF.java:73)
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1183)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:648)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1135)
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
allocate memory
at java.lang.UNIXProcess.(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
... 9 more

Any idea??

Thanks in advance :)
--
View this message in context: http://old.nabble.com/Spill-failed-tp27527092p27527092.html
Sent from the Hadoop core-dev mailing list archive at Nabble.com.

Search Discussions

  • Steve Loughran at Feb 10, 2010 at 11:13 am

    Santiago Pérez wrote:
    Hej

    I am running an aplication based on Nutch in a cluster with 1 master and 6
    slaves in Amazon (with the same instances for all of them with 1.7GB RAM
    memory)
    .
    2010-02-10 03:18:31,463 WARN mapred.TaskTracker - Error running child
    java.io.IOException: Spill failed
    at
    org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1085)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:359)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
    Caused by: java.io.IOException: Cannot run program "bash":
    java.io.IOException: error=12, Cannot allocate memory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
    at org.apache.hadoop.util.Shell.run(Shell.java:134)
    at org.apache.hadoop.fs.DF.getAvailable(DF.java:73)
    at
    org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329)
    at
    org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
    at
    org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
    at
    org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1183)
    at
    org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:648)
    at
    org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1135)
    Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
    allocate memory
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
    at java.lang.ProcessImpl.start(ProcessImpl.java:65)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
    ... 9 more
    The classic JVM fork failure? If so, give your TT less memory, so
    forking becomes cheaper

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedFeb 10, '10 at 8:54a
activeFeb 10, '10 at 11:13a
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Steve Loughran: 1 post Santiago Pérez: 1 post

People

Translate

site design / logo © 2022 Grokbase