[Pig-user] Call Pig from Java

On Thu, Aug 5, 2010 at 9:26 AM, Wenhao Xu wrote:
Hi all,
I am new to pig. I am wondering is there any recommended way to call Pig
code from Java?
Is there any Java interface which can be called directly from Java and
makes them work smoothly? It seems each keyword (filter, group, cogrape,
generate) and data types in Pig can have a counterpart in Java by using
Class, interface and data type. Is these Java interface available to Java
programmers to use? If not, why not?
Thanks very much for help!

regards,
Wenhao

--
~_~

On Thu, Aug 5, 2010 at 9:31 AM, Harsh J wrote:
You need to use the class PigServer.

PigServer pigServer = new PigServer("mapreduce"); // Or "local" for local mode
pigServer.registerQuery("A = LOAD ...");
(...) // Your statements here.
pigServer.store("A", "filename");


--
Harsh J
www.harshj.com

On Wed, Aug 4, 2010 at 9:01 PM, Harsh J wrote:

You need to use the class PigServer.

PigServer pigServer = new PigServer("mapreduce"); // Or "local" for local
mode
pigServer.registerQuery("A = LOAD ...");
(...) // Your statements here.
pigServer.store("A", "filename");


--
Harsh J
www.harshj.com

On Wed, Aug 4, 2010 at 9:08 PM, Wenhao Xu wrote:

Thanks!
Can PigServer handle concurrent requests? Because the store is a
synchronous interface, is there any asynchronous one?

cheers,
W.



--
~_~

On Wed, Aug 4, 2010 at 9:13 PM, Wenhao Xu wrote:

btw, I am considering using it to speedup (parallel) online queries over
large dataset. Is pig suitable for this, or just suitable for offline large
data analysis? Will it be a better choice than distributed(parallel)
database in terms of scalability and latency?

I really like the pig's programming interface. So I want to try to use it
instead of using parallel database.

Thanks!

cheers,
W.
local
and
cogrape,
using


--
~_~

On Thu, Aug 5, 2010 at 12:08 PM, Wenhao Xu wrote:

Thanks!
Can PigServer handle concurrent requests? Because the store is a
synchronous interface, is there any asynchronous one?

cheers,
W.
and
cogrape,
Java


--
~_~

On Wed, Aug 4, 2010 at 9:01 PM, Harsh J wrote:

You need to use the class PigServer.

PigServer pigServer = new PigServer("mapreduce"); // Or "local" for local
mode
pigServer.registerQuery("A = LOAD ...");
(...) // Your statements here.
pigServer.store("A", "filename");


--
Harsh J
www.harshj.com

Thanks!
Can PigServer handle concurrent requests? Because the store is a
synchronous interface, is there any asynchronous one?

cheers,
W.


--
~_~

On Wed, Aug 4, 2010 at 11:27 PM, Gerrit van Vuuren wrote:

Yep I can confirm that if you call it enough times within the same java
process you will run out of memory eventually. I've tried this before,
monitored this with jconsole and saw the memory gradually increasing over 50
or so iterations, each iteration also created its own set of threads that
never died but this might be in the hadoop client itself.
I even tried using a whole different set of classloaders to try and unload
classes after each call but this did not work either


----- Original Message -----
From: Vincent Barat <vbarat@ubikod.com>
To: pig-user@hadoop.apache.org <pig-user@hadoop.apache.org>;
pig-user@hadoop.apache.org <pig-user@hadoop.apache.org>
Sent: Thu Aug 05 07:08:06 2010
Subject: Re: Call Pig from Java

No. PigServer is not reentrant at this time, afaik, and even if you create
several pigserver objects you will run into trouble, as there is a small set
of global data shared between them. It may work for a time, but it will fail
at a point. The only way is to create different processes to handle your
requests.

Wenhao Xu <xuwenhao2008@gmail.com> a écrit :
local
and
cogrape,
using
Java