I have a dataset where each tupple is a term. I then do two filter
operations, to find all terms that have numbers, then all terms that dont
Oddly, there are some terms that dont fit into either group (not really sure
how). So at this point I have 3 bags, all terms, tems with numbers, and
terms without numbers.
What I'm trying to find out is what terms are in the list of all terms, but
are not in either of the two filtered bags. I thought I'd use the DIFF
function, but it only operates on different tuples in the same bag. So
somehow I think I need to crate a new relation, that has three tuples at the
same level (row?). Then I could use the DIFF function.
The script I have so far is shown below...
terms = LOAD 'terms' AS (term:chararray, min:float, max:float,
terms = FOREACH terms GENERATE term;
--bag of all terms
allTerms = GROUP terms ALL;
--bag of terms without numbers
nonNumbers = FILTER terms BY NOT (term MATCHES '^.*[0-9].*$');
nonNumbers = GROUP nonNumbers ALL;
--bag of terms with numbers
withNumbers = FILTER terms BY (term MATCHES '^.*[0-9].*$');
withNumbers = GROUP withNumbers ALL;