FAQ
It happens in "reduce"

2009/7/21 George Pang <p0941p@gmail.com>
Hi users,

Please help with this one - I got an error at running a two - node cluster
on big files, the error is :

2365222 [main] ERROR
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher -
Error message from task (map) task_200907201602_0002_m_000010*java.io.IOException:
Spill failed*
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:425)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:96)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:172)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:158)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:81)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2198)
*Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could
not find any valid local directory for
taskTracker/jobcache/job_200907201602_0002/attempt_200907201602_0002_m_000010_0/output/spill0.out
*

*Cluster specification: *

Hadoop -version 0.18.3
Ubuntu -version 8.04

*etc/hosts*

127.0.0.1 localhost
127.0.1.1 wpang-laptop
192.168.0.1 master
192.168.0.2 slave

*hadoop-site.xml*

<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop-datastore/hadoop-${user.name}</value>
</property>

<property>
<name>fs.default.name</name>
<value>hdfs://master:9000</value>
</property>

<property>
<name>mapred.job.tracker</name>
<value>master:9001</value>
</property>

<property>
<name>dfs.replication</name>
<value>2</value>
</property>


*Hadoop web interface ( http://master:50070)
* Cluster Summary * * * 54 files and directories, 196 blocks = 250 total.
Heap Size is 20.5 MB / 963 MB (2%)
* Capacity : 54.49 GB DFS Remaining : 18.21 GB DFS Used : 22.59 GB DFS
Used% : 41.45 % Live Nodes<http://localhost:50070/dfshealth.jsp#LiveNodes>
: 2 Dead Nodes <http://localhost:50070/dfshealth.jsp#DeadNodes> : 0
George

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 1 | next ›
Discussion Overview
groupcommon-user @
categorieshadoop
postedJul 21, '09 at 7:24p
activeJul 21, '09 at 7:24p
posts1
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

George Pang: 1 post

People

Translate

site design / logo © 2021 Grokbase