More observations: during hadoop job running, this "filesystem closed" error
2010-10-02 05:29:58,951 WARN mapred.TaskTracker - Error running child
java.io.IOException: Filesystem closed
2010-10-02 05:29:58,979 WARN mapred.TaskRunner - Parent died. Exiting
Could this error turns on the safemode in hadoop? I suspect this because the
next hadoop job is supposed to create a segment directory and write out
segment results, but it does not create the directory. Anything else could
happen to hdfs?
On Tue, Sep 28, 2010 at 4:40 PM, AJ Chen wrote:
I'm doing web crawling using nutch, which runs on hadoop in distributed
mode. When the crawldb has tens of millions of urls, I have started to see
strange failure in generating new segment and updating crawldb.
For generating segment, the hadoop job for select is completed successfully
and generate-temp-1285641291765 is created. but it does not start the
partition job and the segment is not created in segments directory. I try to
understand where it fails. There is no error message except for a few WARN
messages about connection reset by peer. Hadoop fsck and dfsadmin show the
nodes and directories are healthy. Is this a hadoop problem or nutch
problem? I'll appreciate any suggestion for how to debug this fatal
Similar problem is seen for updatedb step, which creates the temp dir but
never actually update the crawldb.
AJ Chen, PhD
Chair, Semantic Web SIG, sdforum.org
Palo Alto, CA, USA