FAQ
Hi All,

Do you know if the tmp directory on every map/reduce task will be deleted
automatically after the map task finishes or will do I have to delete them?

I mean the tmp directory that automatically created by on current directory.

Thanks a lot
--Q

Search Discussions

  • Pankil Doshi at Jun 22, 2009 at 8:20 pm
    Yes, If your job gets completed successfully .possibly it removes after
    completion of both map and reduce tasks.

    Pankil
    On Mon, Jun 22, 2009 at 3:15 PM, Qin Gao wrote:

    Hi All,

    Do you know if the tmp directory on every map/reduce task will be deleted
    automatically after the map task finishes or will do I have to delete them?

    I mean the tmp directory that automatically created by on current
    directory.

    Thanks a lot
    --Q
  • Qin Gao at Jun 22, 2009 at 8:25 pm
    Thanks!

    But what if the jobs get killed or failed? Does hadoop try to clean it? we
    are considering bad situations - if job gets killed, will the tmp dirs sit
    on local disks forever and eats up all the diskspace?

    I guess this should be considered in distributed cache, but those files are
    read-only, and our program will generate new temporary files.


    --Q

    On Mon, Jun 22, 2009 at 4:19 PM, Pankil Doshi wrote:

    Yes, If your job gets completed successfully .possibly it removes after
    completion of both map and reduce tasks.

    Pankil
    On Mon, Jun 22, 2009 at 3:15 PM, Qin Gao wrote:

    Hi All,

    Do you know if the tmp directory on every map/reduce task will be deleted
    automatically after the map task finishes or will do I have to delete them?
    I mean the tmp directory that automatically created by on current
    directory.

    Thanks a lot
    --Q
  • Pankil Doshi at Jun 22, 2009 at 8:35 pm
    No..If your job gets killed or failed.Temp wont clean up.. and In that case
    you will have to carefully clean that on your own. If you dont clean it up
    yourself it will eat up your disk space.

    Pankil
    On Mon, Jun 22, 2009 at 4:24 PM, Qin Gao wrote:

    Thanks!

    But what if the jobs get killed or failed? Does hadoop try to clean it? we
    are considering bad situations - if job gets killed, will the tmp dirs sit
    on local disks forever and eats up all the diskspace?

    I guess this should be considered in distributed cache, but those files are
    read-only, and our program will generate new temporary files.


    --Q

    On Mon, Jun 22, 2009 at 4:19 PM, Pankil Doshi wrote:

    Yes, If your job gets completed successfully .possibly it removes after
    completion of both map and reduce tasks.

    Pankil
    On Mon, Jun 22, 2009 at 3:15 PM, Qin Gao wrote:

    Hi All,

    Do you know if the tmp directory on every map/reduce task will be
    deleted
    automatically after the map task finishes or will do I have to delete them?
    I mean the tmp directory that automatically created by on current
    directory.

    Thanks a lot
    --Q
  • Qin Gao at Jun 22, 2009 at 8:46 pm
    Thanks, then I will try keep a log on the files and clean them out, thanks.
    --Q

    On Mon, Jun 22, 2009 at 4:34 PM, Pankil Doshi wrote:

    No..If your job gets killed or failed.Temp wont clean up.. and In that case
    you will have to carefully clean that on your own. If you dont clean it up
    yourself it will eat up your disk space.

    Pankil
    On Mon, Jun 22, 2009 at 4:24 PM, Qin Gao wrote:

    Thanks!

    But what if the jobs get killed or failed? Does hadoop try to clean it? we
    are considering bad situations - if job gets killed, will the tmp dirs sit
    on local disks forever and eats up all the diskspace?

    I guess this should be considered in distributed cache, but those files are
    read-only, and our program will generate new temporary files.


    --Q

    On Mon, Jun 22, 2009 at 4:19 PM, Pankil Doshi wrote:

    Yes, If your job gets completed successfully .possibly it removes after
    completion of both map and reduce tasks.

    Pankil
    On Mon, Jun 22, 2009 at 3:15 PM, Qin Gao wrote:

    Hi All,

    Do you know if the tmp directory on every map/reduce task will be
    deleted
    automatically after the map task finishes or will do I have to delete them?
    I mean the tmp directory that automatically created by on current
    directory.

    Thanks a lot
    --Q
  • Allen Wittenauer at Jun 22, 2009 at 8:45 pm

    On 6/22/09 12:15 PM, "Qin Gao" wrote:
    Do you know if the tmp directory on every map/reduce task will be deleted
    automatically after the map task finishes or will do I have to delete them?

    I mean the tmp directory that automatically created by on current directory.
    Past experience says that users will find writable space on nodes and fill
    it, regardless of what Hadoop may do to try and keep it clean. It is a good
    idea to just wipe those spaces clean during hadoop upgrades and other
    planned downtimes.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJun 22, '09 at 7:16p
activeJun 22, '09 at 8:46p
posts6
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase