Grokbase Groups Pig user April 2011
FAQ
Hi,

I'm trying to pass a STORE target directory and file
into a script, providing a default. My problem
is with the parameter substitution in the STORE
function. Below are three versions of the STORE
command. Each generates a different fault.

What's the right way to do this?

Thanks!

Andreas


%default TARGET_DIR '/foo/bar';
%default URL_INDEX_FILE '${CRAWL_SOURCE}URL';

URLMap = ...;

-- The following creates directory ${TARGET_DIR} in HDFS, which is very
difficult
-- to remove, and is not what I want:
STORE URLMap INTO '${TARGET_DIR}/${URL_INDEX_FILE}'
USING PigStorage;

-- This version unsurprisingly generates error: mismatched input '$'
expecting QUOTEDSTRING
STORE URLMap INTO ${TARGET_DIR}/${URL_INDEX_FILE}
USING PigStorage

-- This one generates an Illegal group reference error (trace below).
-- Some Web research suggests this error might occur during the use
-- of the Java matcher facility:
STORE URLMap INTO '$TARGET_DIR/$URL_INDEX_FILE'
USING PigStorage;

Stack trace of last version:

2011-04-08 18:58:24,517 [main] ERROR org.apache.pig.Main - ERROR 2999:
Unexpected internal error. Illegal group reference
2011-04-08 18:58:24,517 [main] ERROR org.apache.pig.Main -
java.lang.IllegalArgumentException: Illegal group reference
at java.util.regex.Matcher.appendReplacement(Matcher.java:713)
at java.util.regex.Matcher.replaceFirst(Matcher.java:861)
at java.lang.String.replaceFirst(String.java:2146)
at
org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:236)
at
org.apache.pig.tools.parameters.PigFileParser.input(PigFileParser.java:65)
at
org.apache.pig.tools.parameters.PigFileParser.Parse(PigFileParser.java:43)
at
org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:105)
at
org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:98)
at org.apache.pig.Main.runParamPreprocessor(Main.java:692)
at org.apache.pig.Main.run(Main.java:519)
at org.apache.pig.Main.main(Main.java:108)

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedApr 9, '11 at 2:12a
activeApr 9, '11 at 2:12a
posts1
users1
websitepig.apache.org

1 user in discussion

Andreas Paepcke: 1 post

People

Translate

site design / logo © 2021 Grokbase