Grokbase Groups Pig user October 2010
FAQ
Hello,

I face an issue with PIG temporary files: they are not deleted once
a job is terminated.
I got my HDFS storage full of PIG temporary files.

I use PIG from Java using a PigServer object.

Is there is way to ask PIG to cleanup all that it created to perform
a job ?

Thanks a lot

Search Discussions

  • Vincent Barat at Oct 23, 2010 at 11:54 pm
    I forgot to mention that I use PIG 0.6.0, it may help !

    Le 23/10/10 13:29, Vincent Barat a écrit :
    Hello,

    I face an issue with PIG temporary files: they are not deleted
    once a job is terminated.
    I got my HDFS storage full of PIG temporary files.

    I use PIG from Java using a PigServer object.

    Is there is way to ask PIG to cleanup all that it created to
    perform a job ?

    Thanks a lot
  • Rekha Joshi at Oct 25, 2010 at 6:27 am
    Hi Vincent,

    I recall it happens under some specific condition and is resolved in later versions of pig/hadoop by adding an explicit cleanup before final exit.
    As a workaround in 0.6 you can code finally{} in your java file to remove files from your user specific folder under hadoop.tmp.dir.HTH!

    Thanks & Regards,
    /Rekha.

    On 10/24/10 5:24 AM, "Vincent Barat" wrote:

    I forgot to mention that I use PIG 0.6.0, it may help !

    Le 23/10/10 13:29, Vincent Barat a écrit :
    Hello,

    I face an issue with PIG temporary files: they are not deleted
    once a job is terminated.
    I got my HDFS storage full of PIG temporary files.

    I use PIG from Java using a PigServer object.

    Is there is way to ask PIG to cleanup all that it created to
    perform a job ?

    Thanks a lot
  • Vincent Barat at Oct 25, 2010 at 6:30 pm
    Thanks for your answer, I will investigate on that.

    Le 25/10/10 08:26, Rekha Joshi a écrit :
    Hi Vincent,

    I recall it happens under some specific condition and is resolved in later versions of pig/hadoop by adding an explicit cleanup before final exit.
    As a workaround in 0.6 you can code finally{} in your java file to remove files from your user specific folder under hadoop.tmp.dir.HTH!

    Thanks& Regards,
    /Rekha.

    On 10/24/10 5:24 AM, "Vincent Barat"wrote:

    I forgot to mention that I use PIG 0.6.0, it may help !

    Le 23/10/10 13:29, Vincent Barat a écrit :
    Hello,

    I face an issue with PIG temporary files: they are not deleted
    once a job is terminated.
    I got my HDFS storage full of PIG temporary files.

    I use PIG from Java using a PigServer object.

    Is there is way to ask PIG to cleanup all that it created to
    perform a job ?

    Thanks a lot
  • Rakesh kothari at Nov 22, 2010 at 10:20 pm
    This is is happening on PIG 0.7 as well. When I use PigServer to launch pig jobs from Java lots of tmp files (e.g. directory /tmp/temp-203305867) gets created that are not deleted at the end of the Job.

    "hadoop.tmp.dir" points to some other directory "/tmp/hadoop-userA". Any ideas on how to get path to these tmp folders so that I can explicitly delete them at the end of the Job ?

    Thanks,
    -Rakesh
    Date: Mon, 25 Oct 2010 20:30:24 +0200
    From: vbarat@ubikod.com
    To: user@pig.apache.org
    Subject: Re: How to make PIG delete its temporary files ?

    Thanks for your answer, I will investigate on that.

    Le 25/10/10 08:26, Rekha Joshi a écrit :
    Hi Vincent,

    I recall it happens under some specific condition and is resolved in later versions of pig/hadoop by adding an explicit cleanup before final exit.
    As a workaround in 0.6 you can code finally{} in your java file to remove files from your user specific folder under hadoop.tmp.dir.HTH!

    Thanks& Regards,
    /Rekha.

    On 10/24/10 5:24 AM, "Vincent Barat"wrote:

    I forgot to mention that I use PIG 0.6.0, it may help !

    Le 23/10/10 13:29, Vincent Barat a écrit :
    Hello,

    I face an issue with PIG temporary files: they are not deleted
    once a job is terminated.
    I got my HDFS storage full of PIG temporary files.

    I use PIG from Java using a PigServer object.

    Is there is way to ask PIG to cleanup all that it created to
    perform a job ?

    Thanks a lot
  • Alan Gates at Nov 29, 2010 at 9:59 pm
    Pig is supposed to remove all these temporary files, as long as the
    java process finishes in such a way that it has a chance to clean up
    (ie, no one does a kill -9 on it or something). Can you file a JIRA
    with a reproducible case so we can track this down and fix it?

    Alan.
    On Nov 22, 2010, at 2:20 PM, rakesh kothari wrote:


    This is is happening on PIG 0.7 as well. When I use PigServer to
    launch pig jobs from Java lots of tmp files (e.g. directory /tmp/
    temp-203305867) gets created that are not deleted at the end of the
    Job.

    "hadoop.tmp.dir" points to some other directory "/tmp/hadoop-userA".
    Any ideas on how to get path to these tmp folders so that I can
    explicitly delete them at the end of the Job ?

    Thanks,
    -Rakesh
    Date: Mon, 25 Oct 2010 20:30:24 +0200
    From: vbarat@ubikod.com
    To: user@pig.apache.org
    Subject: Re: How to make PIG delete its temporary files ?

    Thanks for your answer, I will investigate on that.

    Le 25/10/10 08:26, Rekha Joshi a écrit :
    Hi Vincent,

    I recall it happens under some specific condition and is resolved
    in later versions of pig/hadoop by adding an explicit cleanup
    before final exit.
    As a workaround in 0.6 you can code finally{} in your java file to
    remove files from your user specific folder under
    hadoop.tmp.dir.HTH!

    Thanks& Regards,
    /Rekha.

    On 10/24/10 5:24 AM, "Vincent Barat"wrote:

    I forgot to mention that I use PIG 0.6.0, it may help !

    Le 23/10/10 13:29, Vincent Barat a écrit :
    Hello,

    I face an issue with PIG temporary files: they are not deleted
    once a job is terminated.
    I got my HDFS storage full of PIG temporary files.

    I use PIG from Java using a PigServer object.

    Is there is way to ask PIG to cleanup all that it created to
    perform a job ?

    Thanks a lot

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedOct 23, '10 at 11:30a
activeNov 29, '10 at 9:59p
posts6
users4
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase