I was using 0.17.2.1, but I installed 0.18.3 a couple of days ago. I
also separated out my secondarynamenode and jobtracker to another
machine. In addition my network operations people had misconfigured
some switches which ended up being my bottleneck.
After all of that my writer and Hadoop is working great.
From: Raghu Angadi
Sent: Wednesday, February 18, 2009 11:49 AM
Subject: Re: Hadoop Write Performance
what is the hadoop version?
You could check log on a datanode around that time. You could post any
suspicious errors. For e.g. you can trace a particular block in client
and datanode logs.
Most likely it not a NameNode issue, but you can check NameNode log as
Xavier Stevens wrote:
Does anyone have an expected or experienced write speed to HDFS
outside of Map/Reduce? Any recommendations on properties to tweak in
Currently I have a multi-threaded writer where each thread is writing
to a different file. But after a while I get this:
java.io.IOException: Could not get block locations. Aborting...
Which is perhaps indicating that the namenode is overwhelmed?