On Wed, Jul 21, 2010 at 3:01 PM, Allen Wittenauer wrote:
a) During every maintenance, we'd blow away the mapred temp dirs. The
TaskTracker does a very bad job of cleaning up after jobs and there is
usually a lot of cruft. If you have a 'flat' disk/fs structure such that MR
temp and HDFS is shared, this is a huge problem.
On Jul 21, 2010, at 12:45 PM, Travis Crawford wrote:
Any tips on how to avoid getting into this situation?
What we started to do was two-fold:Any tips on how to avoid getting into this situation?
a) During every maintenance, we'd blow away the mapred temp dirs. The
TaskTracker does a very bad job of cleaning up after jobs and there is
usually a lot of cruft. If you have a 'flat' disk/fs structure such that MR
temp and HDFS is shared, this is a huge problem.
I've also found that sometimes userlogs aren't cleaned up correctly, so
setting up a cron job to delete old files in userlogs is also a good idea.
Good luck, Travis!
Alex
