already been discussed previously

(here<http://wiki.apache.org/pig/PigTypesFunctionalSpec>).There

seem to be a question of what's the right thing to do. It seems clear to me

though. When an operation like '*' is applied, this is clearly an item-wise

operation that is to be applied to each member of the bag. If a function is

aggregate (SUM), then it operates across an entire bag.

When a COGROUP occurs, just do what SQL does. Which is to say, perform cross

join if an aggregate has been applied across several bags. And do so

automatically, so we don't have to type out the separate FLATTEN's

grouped = COGROUP employee BY name, bonuses BY name;

flattened = FOREACH grouped GENERATE group, *FLATTEN(employee),

FLATTEN(bonuses);grouped_again = GROUP flattened BY group;

total_compensation = FOREACH grouped_again GENERATE group,

SUM(employee:salary * bonuses:multiplier);*

So this should do the same:

grouped = COGROUP employee BY name, bonuses BY name;

total_compensation = FOREACH grouped GENERATE group,

SUM(employee:salary * bonuses:multiplier);

automatically, because that can only have one meaning.

Alternatively, if it is desired to stay with a low-level language, the

solution to all of this confusion around UDF's that take bag's and UDF's

that operate on members of bags can be resolved if we do two things.

1.) Allow UDF's to actually become first class citizens. This way we can

pass UDF's to other UDF's.

2.) introduce the concept of map() and reduce() operator over bags.

This two things allows us more freedom and follows the paradigm of

map-reducing more closely.

grouped = COGROUP employee BY name, bonuses BY name;

total_compensation = FOREACH grouped GENERATE group,

reduce(SUM,map(*,employee::salary,bonuses::multiplier));

Actually, this may deserve a separate keyword. Because map and reduce

operate on single bags where as Pig introduces this concept of co-grouping,

so we should have *comap *and *coreduce* that take functions and operate on

multiple bags that are results of a *cogroup*.

grouped = COGROUP employee BY name, bonuses BY name;

total_compensation = FOREACH grouped GENERATE group,

REDUCE(SUM,COMAP(*, employee::salary,bonuses::multiplier));

This allows us to write efficiently, on one line, what would other wise be

several aliases and unnecessary FLATTENed cross products.

A second thing that I see is the recommendation of implementing looping

constructs. I wonder if I may suggest, as a follow up to the above, that we

beef up UDF's as first class citizens and add the ability to create UDF

functions in Pig Latin with the ability to recurse.

The reason why I think this is a better way to loop than *for(;;)* and *

while(){}* and *do{}while()* statements is that recursive calls are

functional and are more easily optimizable than imperative programming. The

PigJournal <http://wiki.apache.org/pig/PigJournal> has an entry for all of

these constructs and functions under the heading "Extending Pig to Include

Branching, Looping, and Functions", but because map-reduce paradigm is

inherently functional, I would rather think that staying functional would be

a better way to approach this improvement. So the minimal amount of

additional features needed is to implement functions and branching and we

would have loops as a side-effect of those improvements.

In order for the optimizations to be available to PigLatin interpreter, the

functions and branching *must* be implemented within the Pig system. If it

is externalized, or implemented as UDL of some other language, then

opportunities for optimization of the execution vanishes.

Anyways, a couple of cents on a rainy day.

On Wed, Mar 10, 2010 at 10:15 AM, hc busy wrote:

An additional thought... we can define udf's like

ADD(bag{(int,int)}), DIVIDE(bag{(int,int)}), MULTIPLY(bag{(int,int)}),

SQRT(bag{(float)})..

basically vectorize most of the common arithmetic operations, but then the

language has to support it by converting

bag.a + bag.b

to

ADD(bag.(a,b))

I guess there are some difficulties, for instance:

SQRT(bag.a)+bag.b

How would this work? because sqrt(bag.a) returns a bag, how would we

convert it to the correct per tuple operation? It's almost like we want to

convert an expression

SUM(SQRT(bag.a),bag.b)

into a function F such that

SUM(SQRT(bag.a),bag.b) = F(bag.a,bag.b)

and then the F is computed by iterating through on each tuple of the bag.

FOREACH ... GENERATE ..., F(bag.(a,b));

An additional thought... we can define udf's like

ADD(bag{(int,int)}), DIVIDE(bag{(int,int)}), MULTIPLY(bag{(int,int)}),

SQRT(bag{(float)})..

basically vectorize most of the common arithmetic operations, but then the

language has to support it by converting

bag.a + bag.b

to

ADD(bag.(a,b))

I guess there are some difficulties, for instance:

SQRT(bag.a)+bag.b

How would this work? because sqrt(bag.a) returns a bag, how would we

convert it to the correct per tuple operation? It's almost like we want to

convert an expression

SUM(SQRT(bag.a),bag.b)

into a function F such that

SUM(SQRT(bag.a),bag.b) = F(bag.a,bag.b)

and then the F is computed by iterating through on each tuple of the bag.

FOREACH ... GENERATE ..., F(bag.(a,b));

On Wed, Mar 10, 2010 at 9:31 AM, hc busy wrote:

So, pig team, what is the right way to accomplish this?

On Tue, Mar 9, 2010 at 10:50 PM, Mridul Muralidharan <

mridulm@yahoo-inc.com> wrote:

TABLE.number1 actually gives you the bag of number1 values found in TABLE

- but I am never really sure of the semantics in these situations since I am

slightly nervous that it is impl dependent ... my understanding is, what you

are attempting should not work, but I could be wrong.

I do know that TABLE.(number1, number2) will consistently project and

pair up the fields : so to 'fix' this, you can write your own DIVIDE_SUM

which does something like this :

grunt> CALCULATE= FOREACH TABLE_group GENERATE group,

DIVIDE_SUM(TABLE.(number1 , number2));

And DIVIDE_SUM udf impl takes in a bag with tuples containing schema

(numerator, denominator) : and returns :

result == sum ( foreach tuple ( tuple.numerator / tuple.denominator ) );

Obviously, this is not as 'elegant' as your initial code and is

definitely more cumbersome ... so clarifying this behavior with someone from

pig team will definitely be better before you attempt this.

Regards,

Mridul

So, pig team, what is the right way to accomplish this?

On Tue, Mar 9, 2010 at 10:50 PM, Mridul Muralidharan <

mridulm@yahoo-inc.com> wrote:

On Tuesday 09 March 2010 04:13 AM, hc busy wrote:

okay. Here's the bag that I have:

{group: (a: int,b: chararray,c: chararray,d: int), TABLE: {number1:

int,

number2:int}}

and I want to do this

grunt> CALCULATE= FOREACH TABLE_group GENERATE group, SUM(TABLE.number1

/

TABLE.number2);

okay. Here's the bag that I have:

{group: (a: int,b: chararray,c: chararray,d: int), TABLE: {number1:

int,

number2:int}}

and I want to do this

grunt> CALCULATE= FOREACH TABLE_group GENERATE group, SUM(TABLE.number1

/

TABLE.number2);

TABLE.number1 actually gives you the bag of number1 values found in TABLE

- but I am never really sure of the semantics in these situations since I am

slightly nervous that it is impl dependent ... my understanding is, what you

are attempting should not work, but I could be wrong.

I do know that TABLE.(number1, number2) will consistently project and

pair up the fields : so to 'fix' this, you can write your own DIVIDE_SUM

which does something like this :

grunt> CALCULATE= FOREACH TABLE_group GENERATE group,

DIVIDE_SUM(TABLE.(number1 , number2));

And DIVIDE_SUM udf impl takes in a bag with tuples containing schema

(numerator, denominator) : and returns :

result == sum ( foreach tuple ( tuple.numerator / tuple.denominator ) );

Obviously, this is not as 'elegant' as your initial code and is

definitely more cumbersome ... so clarifying this behavior with someone from

pig team will definitely be better before you attempt this.

Regards,

Mridul

grunt> DUMP CALCULATE;

2010-03-08 14:02:41,055 [main] ERROR org.apache.pig.tools.grunt.Grunt -

ERROR 1039: Incompatible types in Multiplication Operator left hand

side:bag

right hand side:bag

This seems useful that I may want to calculate an agg. of some

arithmetic

operations on member of a bag. Any suggestions?

... Looking at the documentation it looks like I want to do something

like

SUM(TABLE.(number1 / number2))

but that doesn't work either :-(

2010-03-08 14:02:41,055 [main] ERROR org.apache.pig.tools.grunt.Grunt -

ERROR 1039: Incompatible types in Multiplication Operator left hand

side:bag

right hand side:bag

This seems useful that I may want to calculate an agg. of some

arithmetic

operations on member of a bag. Any suggestions?

... Looking at the documentation it looks like I want to do something

like

SUM(TABLE.(number1 / number2))

but that doesn't work either :-(