Pig > user > September 2008

I've come across a very basic problem—unions simply do not work in Hadoop mode. data files: $ cat ~/tmp/data 1 1 2 1 3 10 $ cat ~/tmp/data-2 4 20 5 20 pig script: data = load '/Users/arthur/tmp/data' ...

A = (a, b, c) I just want add a column 0 into A, and the A will be like this: A = (a, b, c, 0) How can I? I use cross, but when I use PARALLEL BIGGER( =300), it occurs that: ERROR ...

Hi, How are the GROUP and COGROUP functions implemented? What's its efficiency? Thanks, Chuck

I would like to parse a standard access log and get named variables back. Thinking I need to read in all the lines, then send them through my parsing function. Perhaps the two steps can be combined, ...

Is it possible to implement these using an object composed of List, Commons bag, Integer, Double, String, Boolean and Byte? (I will also have type information, so I don't need to endure the cost of ...

Hi, If I ran the query below (and this is based on actual user query): -- Note that data1 has more than 1 column but as only declares a single one A = load 'data1' as (x); B = load 'data2' as (x, y, ...

Hi, If I want to see what's happening on reduce from running a pig script, what's the best place to look at to start debugging? Thanks, Chuck

The next Hadoop User Group (Bay Area) meeting is scheduled for Wednesday, Sept 17th from 6 - 7:30 pm at Yahoo! Mission College, Santa Clara, CA, Building 2, Training Rooms 3&4. Agenda: Cloud ...

Hello, I would like to write a Load Function to load data in Pig directly from tables in a database. The definition of the function would be LoadFromDb(selectQuery, connectionParameters). In order to ...

Hi All Am a student trying to integrate PIG and Hadoop technologies to build a custom application as a part of my MS project. Am trying out a simple scenario where I have setup a single node hadoop ...

I use the script like this: querys = GROUP clear_log ALL PARALLEL 4; TOTAL = FOREACH querys GENERATE FLATTEN(clear_log.($1, $2)), COUNT($1); STORE TOTAL INTO 'total'; AND I see the monitor page in ...

howdy, Just starting to dive into pig, and have had a hard time finding examples. I would like to put up some examples (on the wiki?) of what I hope to be simple scripts that could help find the ...

Pretty sure that the svn pig link on this page http://wiki.apache.org/pig/PigDeveloperCookbook should be http://svn.apache.org/repos/asf/incubator/pig/ But there are no trunk/pig or src-gen ...

Hello, I am having a problem when giving parameters to a load function that loads data from a database. It receives a SQL query as parameter. Something like that: A = LOAD 'dbparams.cfg' USING ...

I have a log file I'm writing explicitly to process in pig. The data for each line is a set of key-value pairs, which seems pretty much perfect as a fit for Pig, however I'm having some trouble. It's ...

I split a relation to two relations based on a condition, and then join these two relations. How can I access the fields in two relations when I output join results? My code is like: A = Load ... AS ...

Versions: pig from trunk (v 694861), hadoop v 17.2 java -cp pig.jar:$HADOOPSITEPATH org.apache.pig.Main works fine; it gives me a grunt shell connected to hdfs://localhost:9000/ bin/pig doesn't seem ...

Hi, Pig release 0.1.0 is now available. This is the first Pig release from the incubator! For release details and downloads, visit: http://incubator.apache.org/pig/releases.html ...

Hi All, I seem to be seeing a problem with the DISTINCT operator. I have a script that looks like this: raw_tran_hdr = load 'tran_hdr/tran_header' using PigStorage( '|' ) as ( ... many fields ... ); ...

Hi All, Finally have the opportunity to use Pig in my day job. Things are going well; doing some pretty real stuff! Quick type massage question, a minor issue really. I have an expression that looks ...

howdy, I am hoping to parse referers to find the refering search engine and terms. Wondering if there is an analog to http://search.cpan.org/~sden/URI-ParseSearchString-2.5/ for java / pig. Tonight I ...

group a by $0, b by $0, c by $0 VS AA = group a by $0, b by $0 d = FOREACH AA generate $0, flatten($1) group d by $0, c by $0; Does them have the same efficiency, which is faster?

I use a hadoop 0.15, and it didn't have the hod contribution, I didn't wanna change my hadoop distribution. If the hod is not necessary, which files would be modified? If the hod should be used, i ...

I use GROUP can manage the data that has the same "value" into one bag. For example: character, num (b, 1) (a, 1) (b, 2) group (a, {(a, 2)}) {(a, 1)} = { (a, 1) } (a, 2) (b, {(b, 1)}) (a, 1) (b, 2) ...

I can use LOAD or FOREACH to create a data type as a Tuple, or use GROUP/COGROUP to create a data type bag. But when does the map type occurs? Or it may be used in UDFs?

The next Hadoop User Group (Bay Area) meeting is scheduled for Wednesday, Sept 17th from 6 - 7:30 pm at Yahoo! Mission College, Santa Clara, CA, Building 1, Training Rooms 3&4. Agenda: Update on ...

pig-*-core.jar the classes from the libraries (junit-4.1.jar, ${hadoop.jarfile}, jsch-0.1.33.jar). As was discussed in a previous mail ...