[Pig-dev] Pig optimizer

Hi, Renato,
There is a description of optimization rule in Pig Latin reference menu:
http://hadoop.apache.org/pig/docs/r0.7.0/piglatin_ref1.html#Optimization+Rules.
Is that enough?

Daniel

Renato Marroquín Mogrovejo wrote:

Hi Daniel,

Thanks, but that was not what I was actually looking. What I want to know is
for example, how the optimizer work when the bags' logical plans are
combined, or if all commands are reduced at the end to CO-GROUP commands,
how is this handled? I know from Pig's paper that the ORDER, and LOAD,
commands generate new MapReduce jobs, are there any optimizations for the
physical plans?
Thanks in advanced.


Renato M.

2010/8/23 Daniel Dai <jianyong@yahoo-inc.com>

Anyone, please?

Renato M.

2010/8/24 Renato Marroquín Mogrovejo <renatoj.marroquin@gmail.com>

Hi, Renato,
I think you are talking about how we organize different operators into
map-reduce jobs. Unfortunately there is no document currently. Basically
we will put as much operators into one map-reduce job as possible.
Co-group/Group, Join, Order, Distinct, Cross, Stream will create a
map-reduce boundary; Most others we will put into existing jobs. The
main logic is inside MRCompiler.java.

Daniel

Renato Marroquín Mogrovejo wrote:

Hi Daniel,

Yeah that is, but like there are two types of optimizations right? I mean
physical and logical optimizations. The physical ones are the ones on how
the operators are distributed along mapreduce jobs and the logical ones are
the ones similar to relational algebra right?
Do you have any tips on how to get a quick grasp on pig logical
optimizations?
Thanks again.


Renato M.


2010/8/26 Daniel Dai <jianyong@yahoo-inc.com>