Grokbase › Groups › Pig › dev › June 2011
FAQ
Parameter Substitution doesn't work in the Grunt shell
------------------------------------------------------

Key: PIG-2122
URL: https://issues.apache.org/jira/browse/PIG-2122
Project: Pig
Issue Type: Bug
Affects Versions: 0.8.1, 0.8.0
Reporter: Grant Ingersoll
Priority: Minor


Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.

#Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
{quote}
foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
Y = foreach foo generate *, '$time';
dump Y;
{quote}
Output:
{quote}
2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
(1 2 3,,,$time)
(4 5 6,,,$time)
{quote}

Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
{quote}
2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
(1 2 3,,,FOO)
(4 5 6,,,FOO)
{quote}

Also, things like don't work (nor does %declare):
{quote}
grunt> %default DATE '20090101';
2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
Was expecting one of:
<EOF>
"cat" ...
"fs" ...
"sh" ...
"cd" ...
"cp" ...
"copyFromLocal" ...
"copyToLocal" ...
"dump" ...
"describe" ...
"aliases" ...
"explain" ...
"help" ...
"kill" ...
"ls" ...
"mv" ...
"mkdir" ...
"pwd" ...
"quit" ...
"register" ...
"rm" ...
"rmf" ...
"set" ...
"illustrate" ...
"run" ...
"exec" ...
"scriptDone" ...
"" ...
<EOL> ...
";" ...

Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
{quote}



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Search Discussions

  • Olga Natkovich (JIRA) at Jun 15, 2011 at 3:54 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049829#comment-13049829 ]

    Olga Natkovich commented on PIG-2122:
    -------------------------------------

    This is by design. We did not see any need for parameter substitution in the interactive mode. The current implementation is based on a preprocessor applied to the entire script.

    We would need a compelling case to reconsider.
    Parameter Substitution doesn't work in the Grunt shell
    ------------------------------------------------------

    Key: PIG-2122
    URL: https://issues.apache.org/jira/browse/PIG-2122
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0, 0.8.1
    Reporter: Grant Ingersoll
    Priority: Minor

    Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
    #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
    {quote}
    foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
    Y = foreach foo generate *, '$time';
    dump Y;
    {quote}
    Output:
    {quote}
    2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,$time)
    (4 5 6,,,$time)
    {quote}
    Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
    {quote}
    2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,FOO)
    (4 5 6,,,FOO)
    {quote}
    Also, things like don't work (nor does %declare):
    {quote}
    grunt> %default DATE '20090101';
    2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
    Was expecting one of:
    <EOF>
    "cat" ...
    "fs" ...
    "sh" ...
    "cd" ...
    "cp" ...
    "copyFromLocal" ...
    "copyToLocal" ...
    "dump" ...
    "describe" ...
    "aliases" ...
    "explain" ...
    "help" ...
    "kill" ...
    "ls" ...
    "mv" ...
    "mkdir" ...
    "pwd" ...
    "quit" ...
    "register" ...
    "rm" ...
    "rmf" ...
    "set" ...
    "illustrate" ...
    "run" ...
    "exec" ...
    "scriptDone" ...
    "" ...
    <EOL> ...
    ";" ...

    Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
    {quote}
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Grant Ingersoll (JIRA) at Jun 15, 2011 at 4:32 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049847#comment-13049847 ]

    Grant Ingersoll commented on PIG-2122:
    --------------------------------------

    Seems to me one of the major use cases of the Grunt shell is to test your scripts line by line. If you don't have parameter substitution, then you have to futz around with hand editing out the substitutions to run it in Grunt, and then add it back in before you submit it to run in file mode.

    The other use case is, the documentation doesn't work! I would bet that many people simply try out what is on the website as examples, so when things like declare, etc. as copy and pasted from the website don't work, they lose confidence in the tool.
    Parameter Substitution doesn't work in the Grunt shell
    ------------------------------------------------------

    Key: PIG-2122
    URL: https://issues.apache.org/jira/browse/PIG-2122
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0, 0.8.1
    Reporter: Grant Ingersoll
    Priority: Minor

    Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
    #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
    {quote}
    foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
    Y = foreach foo generate *, '$time';
    dump Y;
    {quote}
    Output:
    {quote}
    2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,$time)
    (4 5 6,,,$time)
    {quote}
    Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
    {quote}
    2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,FOO)
    (4 5 6,,,FOO)
    {quote}
    Also, things like don't work (nor does %declare):
    {quote}
    grunt> %default DATE '20090101';
    2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
    Was expecting one of:
    <EOF>
    "cat" ...
    "fs" ...
    "sh" ...
    "cd" ...
    "cp" ...
    "copyFromLocal" ...
    "copyToLocal" ...
    "dump" ...
    "describe" ...
    "aliases" ...
    "explain" ...
    "help" ...
    "kill" ...
    "ls" ...
    "mv" ...
    "mkdir" ...
    "pwd" ...
    "quit" ...
    "register" ...
    "rm" ...
    "rmf" ...
    "set" ...
    "illustrate" ...
    "run" ...
    "exec" ...
    "scriptDone" ...
    "" ...
    <EOL> ...
    ";" ...

    Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
    {quote}
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Dmitriy V. Ryaboy (JIRA) at Jun 15, 2011 at 5:40 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049910#comment-13049910 ]

    Dmitriy V. Ryaboy commented on PIG-2122:
    ----------------------------------------

    This really does diminish usefulness of Grunt significantly.
    There isn't an easy way to apply the existing preprocessor here, which is a shame, but it's an issue that comes up often. It would be great if someone took a crack at fixing this.. theoretically having a in-memory variable map and doing line-by-line substitution before handing off to the parser should work, right?
    Parameter Substitution doesn't work in the Grunt shell
    ------------------------------------------------------

    Key: PIG-2122
    URL: https://issues.apache.org/jira/browse/PIG-2122
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0, 0.8.1
    Reporter: Grant Ingersoll
    Priority: Minor

    Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
    #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
    {quote}
    foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
    Y = foreach foo generate *, '$time';
    dump Y;
    {quote}
    Output:
    {quote}
    2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,$time)
    (4 5 6,,,$time)
    {quote}
    Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
    {quote}
    2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,FOO)
    (4 5 6,,,FOO)
    {quote}
    Also, things like don't work (nor does %declare):
    {quote}
    grunt> %default DATE '20090101';
    2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
    Was expecting one of:
    <EOF>
    "cat" ...
    "fs" ...
    "sh" ...
    "cd" ...
    "cp" ...
    "copyFromLocal" ...
    "copyToLocal" ...
    "dump" ...
    "describe" ...
    "aliases" ...
    "explain" ...
    "help" ...
    "kill" ...
    "ls" ...
    "mv" ...
    "mkdir" ...
    "pwd" ...
    "quit" ...
    "register" ...
    "rm" ...
    "rmf" ...
    "set" ...
    "illustrate" ...
    "run" ...
    "exec" ...
    "scriptDone" ...
    "" ...
    <EOL> ...
    ";" ...

    Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
    {quote}
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Olga Natkovich (JIRA) at Jun 15, 2011 at 5:52 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049920#comment-13049920 ]

    Olga Natkovich commented on PIG-2122:
    -------------------------------------

    I think the trick would be to maintain backward compatibility.
    Parameter Substitution doesn't work in the Grunt shell
    ------------------------------------------------------

    Key: PIG-2122
    URL: https://issues.apache.org/jira/browse/PIG-2122
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0, 0.8.1
    Reporter: Grant Ingersoll
    Priority: Minor

    Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
    #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
    {quote}
    foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
    Y = foreach foo generate *, '$time';
    dump Y;
    {quote}
    Output:
    {quote}
    2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,$time)
    (4 5 6,,,$time)
    {quote}
    Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
    {quote}
    2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,FOO)
    (4 5 6,,,FOO)
    {quote}
    Also, things like don't work (nor does %declare):
    {quote}
    grunt> %default DATE '20090101';
    2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
    Was expecting one of:
    <EOF>
    "cat" ...
    "fs" ...
    "sh" ...
    "cd" ...
    "cp" ...
    "copyFromLocal" ...
    "copyToLocal" ...
    "dump" ...
    "describe" ...
    "aliases" ...
    "explain" ...
    "help" ...
    "kill" ...
    "ls" ...
    "mv" ...
    "mkdir" ...
    "pwd" ...
    "quit" ...
    "register" ...
    "rm" ...
    "rmf" ...
    "set" ...
    "illustrate" ...
    "run" ...
    "exec" ...
    "scriptDone" ...
    "" ...
    <EOL> ...
    ";" ...

    Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
    {quote}
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Grant Ingersoll (JIRA) at Jun 15, 2011 at 5:56 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049922#comment-13049922 ]

    Grant Ingersoll commented on PIG-2122:
    --------------------------------------

    I know it would be nice, but is back compat really a requirement for a pre 1.0 release?
    Parameter Substitution doesn't work in the Grunt shell
    ------------------------------------------------------

    Key: PIG-2122
    URL: https://issues.apache.org/jira/browse/PIG-2122
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0, 0.8.1
    Reporter: Grant Ingersoll
    Priority: Minor

    Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
    #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
    {quote}
    foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
    Y = foreach foo generate *, '$time';
    dump Y;
    {quote}
    Output:
    {quote}
    2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,$time)
    (4 5 6,,,$time)
    {quote}
    Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
    {quote}
    2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,FOO)
    (4 5 6,,,FOO)
    {quote}
    Also, things like don't work (nor does %declare):
    {quote}
    grunt> %default DATE '20090101';
    2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
    Was expecting one of:
    <EOF>
    "cat" ...
    "fs" ...
    "sh" ...
    "cd" ...
    "cp" ...
    "copyFromLocal" ...
    "copyToLocal" ...
    "dump" ...
    "describe" ...
    "aliases" ...
    "explain" ...
    "help" ...
    "kill" ...
    "ls" ...
    "mv" ...
    "mkdir" ...
    "pwd" ...
    "quit" ...
    "register" ...
    "rm" ...
    "rmf" ...
    "set" ...
    "illustrate" ...
    "run" ...
    "exec" ...
    "scriptDone" ...
    "" ...
    <EOL> ...
    ";" ...

    Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
    {quote}
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Olga Natkovich (JIRA) at Jun 15, 2011 at 6:05 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049926#comment-13049926 ]

    Olga Natkovich commented on PIG-2122:
    -------------------------------------

    Backward compatibility is a requirement because we have many customers and many of them uset this feature especially in production environment.

    In my opinion design/implementation that does not provide backward comptaibility for batch processing is a non-starter.

    Parameter Substitution doesn't work in the Grunt shell
    ------------------------------------------------------

    Key: PIG-2122
    URL: https://issues.apache.org/jira/browse/PIG-2122
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0, 0.8.1
    Reporter: Grant Ingersoll
    Priority: Minor

    Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
    #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
    {quote}
    foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
    Y = foreach foo generate *, '$time';
    dump Y;
    {quote}
    Output:
    {quote}
    2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,$time)
    (4 5 6,,,$time)
    {quote}
    Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
    {quote}
    2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,FOO)
    (4 5 6,,,FOO)
    {quote}
    Also, things like don't work (nor does %declare):
    {quote}
    grunt> %default DATE '20090101';
    2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
    Was expecting one of:
    <EOF>
    "cat" ...
    "fs" ...
    "sh" ...
    "cd" ...
    "cp" ...
    "copyFromLocal" ...
    "copyToLocal" ...
    "dump" ...
    "describe" ...
    "aliases" ...
    "explain" ...
    "help" ...
    "kill" ...
    "ls" ...
    "mv" ...
    "mkdir" ...
    "pwd" ...
    "quit" ...
    "register" ...
    "rm" ...
    "rmf" ...
    "set" ...
    "illustrate" ...
    "run" ...
    "exec" ...
    "scriptDone" ...
    "" ...
    <EOL> ...
    ";" ...

    Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
    {quote}
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Jeff Wartes (JIRA) at Sep 9, 2011 at 4:09 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101302#comment-13101302 ]

    Jeff Wartes commented on PIG-2122:
    ----------------------------------

    I found this JIRA issue because:
    1) I was copying a script into the shell for testing and was confused why it was wasn't working.
    2) Reading the documentation and attempting alternative methods to set parameters didn't work either.

    I mention this because I hit *both* of cases previously presented as reasons to fix this issue. The second, because of the first.

    If this issue cannot be fixed, I'd suggest the grunt shell throw a more helpful error when it encounters preprocessor directives, and warn when it sees likely attempts at dollar-sign interpolation.
    Parameter Substitution doesn't work in the Grunt shell
    ------------------------------------------------------

    Key: PIG-2122
    URL: https://issues.apache.org/jira/browse/PIG-2122
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0, 0.8.1
    Reporter: Grant Ingersoll
    Priority: Minor

    Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
    #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
    {quote}
    foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
    Y = foreach foo generate *, '$time';
    dump Y;
    {quote}
    Output:
    {quote}
    2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,$time)
    (4 5 6,,,$time)
    {quote}
    Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
    {quote}
    2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,FOO)
    (4 5 6,,,FOO)
    {quote}
    Also, things like don't work (nor does %declare):
    {quote}
    grunt> %default DATE '20090101';
    2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
    Was expecting one of:
    <EOF>
    "cat" ...
    "fs" ...
    "sh" ...
    "cd" ...
    "cp" ...
    "copyFromLocal" ...
    "copyToLocal" ...
    "dump" ...
    "describe" ...
    "aliases" ...
    "explain" ...
    "help" ...
    "kill" ...
    "ls" ...
    "mv" ...
    "mkdir" ...
    "pwd" ...
    "quit" ...
    "register" ...
    "rm" ...
    "rmf" ...
    "set" ...
    "illustrate" ...
    "run" ...
    "exec" ...
    "scriptDone" ...
    "" ...
    <EOL> ...
    ";" ...

    Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
    {quote}
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Janne Jalkanen (JIRA) at Apr 23, 2012 at 1:56 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259606#comment-13259606 ]

    Janne Jalkanen commented on PIG-2122:
    -------------------------------------

    Upvoting; this makes debugging Pig scripts *really* hard using Grunt. I mostly run a part of the script simply by cut-n-pasting from the editor into a Grunt shell, then stop to examine the intermediate values. Having to do manual search-replace for variables makes them useless.
    Parameter Substitution doesn't work in the Grunt shell
    ------------------------------------------------------

    Key: PIG-2122
    URL: https://issues.apache.org/jira/browse/PIG-2122
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0, 0.8.1
    Reporter: Grant Ingersoll
    Priority: Minor

    Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
    #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
    {quote}
    foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
    Y = foreach foo generate *, '$time';
    dump Y;
    {quote}
    Output:
    {quote}
    2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,$time)
    (4 5 6,,,$time)
    {quote}
    Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
    {quote}
    2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,FOO)
    (4 5 6,,,FOO)
    {quote}
    Also, things like don't work (nor does %declare):
    {quote}
    grunt> %default DATE '20090101';
    2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
    Was expecting one of:
    <EOF>
    "cat" ...
    "fs" ...
    "sh" ...
    "cd" ...
    "cp" ...
    "copyFromLocal" ...
    "copyToLocal" ...
    "dump" ...
    "describe" ...
    "aliases" ...
    "explain" ...
    "help" ...
    "kill" ...
    "ls" ...
    "mv" ...
    "mkdir" ...
    "pwd" ...
    "quit" ...
    "register" ...
    "rm" ...
    "rmf" ...
    "set" ...
    "illustrate" ...
    "run" ...
    "exec" ...
    "scriptDone" ...
    "" ...
    <EOL> ...
    ";" ...

    Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
    {quote}
    --
    This message is automatically generated by JIRA.
    If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Gianmarco De Francisci Morales (JIRA) at Apr 23, 2012 at 6:02 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259781#comment-13259781 ]

    Gianmarco De Francisci Morales commented on PIG-2122:
    -----------------------------------------------------

    Hi,
    I can agree that the feature is useful.

    However to debug a Pig script you can use the -debug and -dryrun options:

    {code}
    –dryrun
    Flag. With this option, the script is not run and a fully
    substituted Pig script produced in the current working
    directory named original_script_name.substituted
    {code}

    This should help you avoid manual search/replace.

    Cheers,
    Parameter Substitution doesn't work in the Grunt shell
    ------------------------------------------------------

    Key: PIG-2122
    URL: https://issues.apache.org/jira/browse/PIG-2122
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0, 0.8.1
    Reporter: Grant Ingersoll
    Priority: Minor

    Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
    #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
    {quote}
    foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
    Y = foreach foo generate *, '$time';
    dump Y;
    {quote}
    Output:
    {quote}
    2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,$time)
    (4 5 6,,,$time)
    {quote}
    Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
    {quote}
    2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,FOO)
    (4 5 6,,,FOO)
    {quote}
    Also, things like don't work (nor does %declare):
    {quote}
    grunt> %default DATE '20090101';
    2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
    Was expecting one of:
    <EOF>
    "cat" ...
    "fs" ...
    "sh" ...
    "cd" ...
    "cp" ...
    "copyFromLocal" ...
    "copyToLocal" ...
    "dump" ...
    "describe" ...
    "aliases" ...
    "explain" ...
    "help" ...
    "kill" ...
    "ls" ...
    "mv" ...
    "mkdir" ...
    "pwd" ...
    "quit" ...
    "register" ...
    "rm" ...
    "rmf" ...
    "set" ...
    "illustrate" ...
    "run" ...
    "exec" ...
    "scriptDone" ...
    "" ...
    <EOL> ...
    ";" ...

    Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
    {quote}
    --
    This message is automatically generated by JIRA.
    If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Gianmarco De Francisci Morales (JIRA) at Apr 23, 2012 at 6:06 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259783#comment-13259783 ]

    Gianmarco De Francisci Morales commented on PIG-2122:
    -----------------------------------------------------

    On the backward compatibility issue, I must say that I think modifying Grunt and interactive mode is not as bad as modifying the batch mode.
    I don't feel changing Grunt should be a big issue as most of the production job that I know are batch scripts which would be unaffected.
    Parameter Substitution doesn't work in the Grunt shell
    ------------------------------------------------------

    Key: PIG-2122
    URL: https://issues.apache.org/jira/browse/PIG-2122
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0, 0.8.1
    Reporter: Grant Ingersoll
    Priority: Minor

    Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
    #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
    {quote}
    foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
    Y = foreach foo generate *, '$time';
    dump Y;
    {quote}
    Output:
    {quote}
    2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,$time)
    (4 5 6,,,$time)
    {quote}
    Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
    {quote}
    2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,FOO)
    (4 5 6,,,FOO)
    {quote}
    Also, things like don't work (nor does %declare):
    {quote}
    grunt> %default DATE '20090101';
    2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
    Was expecting one of:
    <EOF>
    "cat" ...
    "fs" ...
    "sh" ...
    "cd" ...
    "cp" ...
    "copyFromLocal" ...
    "copyToLocal" ...
    "dump" ...
    "describe" ...
    "aliases" ...
    "explain" ...
    "help" ...
    "kill" ...
    "ls" ...
    "mv" ...
    "mkdir" ...
    "pwd" ...
    "quit" ...
    "register" ...
    "rm" ...
    "rmf" ...
    "set" ...
    "illustrate" ...
    "run" ...
    "exec" ...
    "scriptDone" ...
    "" ...
    <EOL> ...
    ";" ...

    Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
    {quote}
    --
    This message is automatically generated by JIRA.
    If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Joshua Hartman (JIRA) at Sep 27, 2012 at 12:37 am ⇧
    [ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464326#comment-13464326 ]

    Joshua Hartman commented on PIG-2122:
    -------------------------------------

    This irritates me on a weekly basis. I'm often running ad-hoc analysis in grunt with snippets taken from some heavier scripts I've written and I get bitten ALL the time.
    Parameter Substitution doesn't work in the Grunt shell
    ------------------------------------------------------

    Key: PIG-2122
    URL: https://issues.apache.org/jira/browse/PIG-2122
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0, 0.8.1
    Reporter: Grant Ingersoll
    Priority: Minor

    Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
    #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
    {quote}
    foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
    Y = foreach foo generate *, '$time';
    dump Y;
    {quote}
    Output:
    {quote}
    2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,$time)
    (4 5 6,,,$time)
    {quote}
    Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
    {quote}
    2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,FOO)
    (4 5 6,,,FOO)
    {quote}
    Also, things like don't work (nor does %declare):
    {quote}
    grunt> %default DATE '20090101';
    2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
    Was expecting one of:
    <EOF>
    "cat" ...
    "fs" ...
    "sh" ...
    "cd" ...
    "cp" ...
    "copyFromLocal" ...
    "copyToLocal" ...
    "dump" ...
    "describe" ...
    "aliases" ...
    "explain" ...
    "help" ...
    "kill" ...
    "ls" ...
    "mv" ...
    "mkdir" ...
    "pwd" ...
    "quit" ...
    "register" ...
    "rm" ...
    "rmf" ...
    "set" ...
    "illustrate" ...
    "run" ...
    "exec" ...
    "scriptDone" ...
    "" ...
    <EOL> ...
    ";" ...

    Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
    {quote}
    --
    This message is automatically generated by JIRA.
    If you think it was sent incorrectly, please contact your JIRA administrators
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Aniket Mokashi (JIRA) at Oct 1, 2012 at 3:28 am ⇧
    [ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466619#comment-13466619 ]

    Aniket Mokashi commented on PIG-2122:
    -------------------------------------

    There is a workaround to this that you can use-

    * Store your script line in a file (with $params included).
    * Start grunt interactively
    * type- run -param a=b -param c=d myscript.pig

    This should do the trick.
    Parameter Substitution doesn't work in the Grunt shell
    ------------------------------------------------------

    Key: PIG-2122
    URL: https://issues.apache.org/jira/browse/PIG-2122
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0, 0.8.1
    Reporter: Grant Ingersoll
    Priority: Minor

    Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
    #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
    {quote}
    foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
    Y = foreach foo generate *, '$time';
    dump Y;
    {quote}
    Output:
    {quote}
    2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,$time)
    (4 5 6,,,$time)
    {quote}
    Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
    {quote}
    2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,FOO)
    (4 5 6,,,FOO)
    {quote}
    Also, things like don't work (nor does %declare):
    {quote}
    grunt> %default DATE '20090101';
    2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
    Was expecting one of:
    <EOF>
    "cat" ...
    "fs" ...
    "sh" ...
    "cd" ...
    "cp" ...
    "copyFromLocal" ...
    "copyToLocal" ...
    "dump" ...
    "describe" ...
    "aliases" ...
    "explain" ...
    "help" ...
    "kill" ...
    "ls" ...
    "mv" ...
    "mkdir" ...
    "pwd" ...
    "quit" ...
    "register" ...
    "rm" ...
    "rmf" ...
    "set" ...
    "illustrate" ...
    "run" ...
    "exec" ...
    "scriptDone" ...
    "" ...
    <EOL> ...
    ";" ...

    Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
    {quote}
    --
    This message is automatically generated by JIRA.
    If you think it was sent incorrectly, please contact your JIRA administrators
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • SnowCrash (JIRA) at Oct 5, 2012 at 7:04 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470566#comment-13470566 ]

    SnowCrash commented on PIG-2122:
    --------------------------------

    [~aniket486]

    Your workaround is not really much help if you're trying to debug a script (i.e. line by line, DUMPing or ILLUSTRATing as you go. We want to be able to use the interactive shell with the existing script (including params) but execute it line by line.
    Parameter Substitution doesn't work in the Grunt shell
    ------------------------------------------------------

    Key: PIG-2122
    URL: https://issues.apache.org/jira/browse/PIG-2122
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0, 0.8.1
    Reporter: Grant Ingersoll
    Priority: Minor

    Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
    #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
    {quote}
    foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
    Y = foreach foo generate *, '$time';
    dump Y;
    {quote}
    Output:
    {quote}
    2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,$time)
    (4 5 6,,,$time)
    {quote}
    Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
    {quote}
    2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
    (1 2 3,,,FOO)
    (4 5 6,,,FOO)
    {quote}
    Also, things like don't work (nor does %declare):
    {quote}
    grunt> %default DATE '20090101';
    2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
    Was expecting one of:
    <EOF>
    "cat" ...
    "fs" ...
    "sh" ...
    "cd" ...
    "cp" ...
    "copyFromLocal" ...
    "copyToLocal" ...
    "dump" ...
    "describe" ...
    "aliases" ...
    "explain" ...
    "help" ...
    "kill" ...
    "ls" ...
    "mv" ...
    "mkdir" ...
    "pwd" ...
    "quit" ...
    "register" ...
    "rm" ...
    "rmf" ...
    "set" ...
    "illustrate" ...
    "run" ...
    "exec" ...
    "scriptDone" ...
    "" ...
    <EOL> ...
    ";" ...

    Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
    {quote}
    --
    This message is automatically generated by JIRA.
    If you think it was sent incorrectly, please contact your JIRA administrators
    For more information on JIRA, see: http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriespig, hadoop
postedJun 14, '11 at 12:27a
activeOct 5, '12 at 7:04p
posts14
users1
websitepig.apache.org

1 user in discussion

SnowCrash (JIRA): 14 posts

Content

People

Support

Translate

site design / logo © 2023 Grokbase