[Pig-user] Working on multiple rows

Can you give the multiple rows an id and use that ? In your example , can
you assign a user-group id for each type of user (or maybe a map with
attributes if a user can belong to multiple groups), and then process using
that attribute or id ?
(I might not have understood the problem correctly, example of input and
output data might help)
-Thejas

Taking a guess, you could group things based on your criterion and
condition.

Something simple like :

a) group by usergroup (might be too expensive ? number of records across
timestamps for users in a group might be large !).

b) group by (usergroup, timestamp / window) [this will loose accuracy
near the time window, see below] : manageable, but less accurate.

Other more sensible variations based on your input, etc !


Something like :

-- This means that if the users clicked at 9th and 11th minute, we
bucket it into two different buckets and miss out on data : so
typically, adjust accordingly for error, or replicate input or something
more complicated than this simple snippet :-)

%default WINDOW '60 * 10'

A = $MY_INPUT AS (user:chararray, user_grp:chararray, timestamp:long);

-- B = GROUP A by user_grp PARALLEL $PARALLELISM;
B = GROUP A by (user_grp, timestamp / $WINDOW) PARALLEL $PARALLELISM;
C = FILTER B by COUNT(A) > $THRESHOLD;




Ofcourse, I hope I am not misunderstanding your query entirely !


Regards,
Mridul