Grokbase Groups Pig user January 2010
FAQ
Thanks Rekha.

These issues seem to be related to cleaning up Pig/Hadoop file upon shutdown
of the VM. I just checked and when I shut down the VM, all files are cleaned
up as expected.

My issue is that I have Pig jobs that run in an app server which are
triggered by quartz. It might be days or weeks between app server bounces.
If anyone knows a way to configure or kick off some sort of cleanup process
without shutting down the VM, please let me know.

Otherwise, I need to deploy a hacky crontab script like this:

find /tmp/Job[0-9]*.jar -type f -mmin +50 -exec rm {} \;

On Tue, Jan 26, 2010 at 8:40 PM, Rekha Joshi wrote:

You might like to check up PIG-116 and HADOOP-5175.Also think there is a
JobCleanup task which takes care of cleaning.., AFAIK.., unless its failed
job.
Cheers,
/R



On 1/27/10 12:01 AM, "Bill Graham" wrote:

Hi,

Every time I run a Pig script I get a number of Job jars left in the /tmp
directory of my client, 1 per MR job it seems. The file names look like
/tmp/Job875278192.jar.

I have scripts that run every five minutes and fire 10 MR jobs each, so the
amount of space used by these jars grows rapidly. Is there a way to tell
Pig
to clean up after itself and remove these jars, or do I need to just write
my own clean-up script?

thanks,
Bill

Search Discussions

Discussion Posts

Previous

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 3 of 3 | next ›
Discussion Overview
groupuser @
categoriespig, hadoop
postedJan 26, '10 at 6:31p
activeJan 27, '10 at 7:33p
posts3
users2
websitepig.apache.org

2 users in discussion

Bill Graham: 2 posts Rekha Joshi: 1 post

People

Translate

site design / logo © 2022 Grokbase