Yep I can confirm that if you call it enough times within the same java process you will run out of memory eventually. I've tried this before, monitored this with jconsole and saw the memory gradually increasing over 50 or so iterations, each iteration also created its own set of threads that never died but this might be in the hadoop client itself.
I even tried using a whole different set of classloaders to try and unload classes after each call but this did not work either
----- Original Message -----
From: Vincent Barat <email@example.com>
To: firstname.lastname@example.org <email@example.com>; firstname.lastname@example.org <email@example.com>
Sent: Thu Aug 05 07:08:06 2010
Subject: Re: Call Pig from Java
No. PigServer is not reentrant at this time, afaik, and even if you create several pigserver objects you will run into trouble, as there is a small set of global data shared between them. It may work for a time, but it will fail at a point. The only way is to create different processes to handle your requests.
Wenhao Xu <firstname.lastname@example.org> a écrit :
Can PigServer handle concurrent requests? Because the store is a
synchronous interface, is there any asynchronous one?
On Wed, Aug 4, 2010 at 9:01 PM, Harsh J wrote:
You need to use the class PigServer.
PigServer pigServer = new PigServer("mapreduce"); // Or "local" for local
pigServer.registerQuery("A = LOAD ...");
(...) // Your statements here.
On Thu, Aug 5, 2010 at 9:26 AM, Wenhao Xu wrote:
I am new to pig. I am wondering is there any recommended way to call Pig
code from Java?
Is there any Java interface which can be called directly from Java and
makes them work smoothly? It seems each keyword (filter, group, cogrape,
generate) and data types in Pig can have a counterpart in Java by using
Class, interface and data type. Is these Java interface available to Java
programmers to use? If not, why not?
Thanks very much for help!