FAQ
Hi,

We have a small cluster of 7 machines, of which 4 are also tasktrackers,
running Hadoop 0.19 (yes, we should move to 0.20...).

There is no Reduce phase; instead each Map task writes output to an HDFS
file. On completion of a job, there is a listener that reads every such file
& writes the results to a Solr index.

We observed a very strange scenario this weekend: the master complains about
too many open files every 24 hours or so, and stops working, though the HDFS
cluster itself remains up. We get two errors:

11/07/10 05:16:57 ERROR our.Class: java.io.FileNotFoundException:
/home/user/project/conf/hadoop-default.xml (Too many open files) :
org.apache.hadoop.conf.Configuration : 1043

11/07/09 05:01:20 ERROR namenode.LeaseManager: In
org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor
java.util.ConcurrentModificationException
at
java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
at java.util.TreeMap$KeyIterator.next(TreeMap.java:1154)
at
org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:355)
at java.lang.Thread.run(Thread.java:619)

When i checked /proc/sys/fs/file-nr it showed about 37500 or so.

This issue has never cropped up before... Please help! :)

Thanks,

Meghana

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJul 11, '11 at 6:31a
activeJul 11, '11 at 6:31a
posts1
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Meghana: 1 post

People

Translate

site design / logo © 2022 Grokbase