FAQ
Hi all,

I found my cluster’s space usage increase over time although I did not
upload new data. And there's a lot of files under folder /tmp .

So I guess hadoop won’t delete the intermediate data(output of mapper).

Am I right ?


Thank you.

Jeff zhang

Search Discussions

  • Chandraprakash Bhagtani at Aug 31, 2009 at 9:20 am
    Hadoop does delete the intermediate data after the job completes.
    Jobtracker sends signal to Tasktracker to delete intermediate data
    when the job completes.

    The problem in your case might be some of your running job might not
    have been killed gracefully or Jobtracker failed for some reason.

    --
    Thanks & Regards,
    Chandra Prakash Bhagtani,
    On Tue, Aug 25, 2009 at 6:19 AM, zhang jianfeng wrote:

    Hi all,

    I found my cluster’s space usage increase over time although I did not
    upload new data. And there's a lot of files under folder /tmp .

    So I guess hadoop won’t delete the intermediate data(output of mapper).

    Am I right ?


    Thank you.

    Jeff zhang
  • Jim Twensky at Aug 31, 2009 at 4:48 pm
    Hi Jeff,

    The problem may also be related to the large log files if you use the
    cluster for too many jobs. Check out your hadoop log directory and see
    how big it is. You can decrease the maximum size of a log file using
    one of the hadoop configuration files under conf.

    Jim

    On Mon, Aug 31, 2009 at 2:19 AM, Chandraprakash
    Bhagtaniwrote:
    Hadoop does delete the intermediate data after the job completes.
    Jobtracker sends signal to Tasktracker to delete intermediate data
    when the job completes.

    The problem in your case might be some of your running job might not
    have been killed gracefully or Jobtracker failed for some reason.

    --
    Thanks & Regards,
    Chandra Prakash Bhagtani,
    On Tue, Aug 25, 2009 at 6:19 AM, zhang jianfeng wrote:

    Hi all,

    I found my cluster’s space usage increase over time although I did not
    upload new data.  And there's a lot of files under folder /tmp .

    So I guess hadoop won’t delete the intermediate data(output of mapper).

    Am I right ?


    Thank you.

    Jeff zhang

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedAug 25, '09 at 12:49a
activeAug 31, '09 at 4:48p
posts3
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase