Grokbase Groups Pig dev December 2008
FAQ
run command for grunt
---------------------

Key: PIG-574
URL: https://issues.apache.org/jira/browse/PIG-574
Project: Pig
Issue Type: New Feature
Components: grunt
Reporter: David Ciemiewicz
Priority: Minor


This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.

One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).

Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.

I'm used to doing similar things with Oracle, MySQL, and R.



--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Gunther Hagleitner (JIRA) at Feb 10, 2009 at 9:07 am
    [ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Gunther Hagleitner updated PIG-574:
    -----------------------------------

    Attachment: run_command.patch

    Introduces run and exec command
    run command for grunt
    ---------------------

    Key: PIG-574
    URL: https://issues.apache.org/jira/browse/PIG-574
    Project: Pig
    Issue Type: New Feature
    Components: grunt
    Reporter: David Ciemiewicz
    Priority: Minor
    Attachments: run_command.patch


    This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
    One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
    Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
    I'm used to doing similar things with Oracle, MySQL, and R.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Gunther Hagleitner (JIRA) at Feb 10, 2009 at 9:13 am
    [ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Gunther Hagleitner updated PIG-574:
    -----------------------------------

    Status: Patch Available (was: Open)
    run command for grunt
    ---------------------

    Key: PIG-574
    URL: https://issues.apache.org/jira/browse/PIG-574
    Project: Pig
    Issue Type: New Feature
    Components: grunt
    Reporter: David Ciemiewicz
    Priority: Minor
    Attachments: run_command.patch


    This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
    One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
    Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
    I'm used to doing similar things with Oracle, MySQL, and R.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Feb 10, 2009 at 6:59 pm
    [ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672365#action_12672365 ]

    Olga Natkovich commented on PIG-574:
    ------------------------------------

    I will be reviewing this patch
    run command for grunt
    ---------------------

    Key: PIG-574
    URL: https://issues.apache.org/jira/browse/PIG-574
    Project: Pig
    Issue Type: New Feature
    Components: grunt
    Reporter: David Ciemiewicz
    Priority: Minor
    Attachments: run_command.patch


    This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
    One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
    Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
    I'm used to doing similar things with Oracle, MySQL, and R.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Feb 10, 2009 at 10:08 pm
    [ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672425#action_12672425 ]

    Olga Natkovich commented on PIG-574:
    ------------------------------------

    I reviewed the patch and it looks good. I also ran unit tests and they all passed.

    In addition, I ran some manual tests and have a couple of comments:

    (1) As implemented now, scripts that ran from within grunt would not be able to take advantage of the parameter substitution as it is not available in the interactive mode. I think this is ok for now and we can revisit it later if users ask for it.
    (2) When using run, I could integrate commands in my script with the interactive commands in the shell which was really nice; however, the commands from the script did not show in the command history. If it is reasonably easy to integrate, it would be nice to have that functionality.
    (3) Very minor thing: after I execute ran command, I see double prompt from grant:

    grunt> grunt>


    run command for grunt
    ---------------------

    Key: PIG-574
    URL: https://issues.apache.org/jira/browse/PIG-574
    Project: Pig
    Issue Type: New Feature
    Components: grunt
    Reporter: David Ciemiewicz
    Priority: Minor
    Attachments: run_command.patch


    This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
    One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
    Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
    I'm used to doing similar things with Oracle, MySQL, and R.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Gunther Hagleitner (JIRA) at Feb 11, 2009 at 9:53 am
    [ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Gunther Hagleitner updated PIG-574:
    -----------------------------------

    Attachment: run_command_params.patch

    Thanks for reviewing the patch!

    I tried to address the 3 issues you pointed out:

    1) You can now specify parameters and param files in both the exec and run command

    grunt> run myscript.pig using param_file myparams.ppf
    or:
    grunt> run myscript.pig using param LIMIT=5 param_file myparams.ppf

    The syntax mimics what you can do on the command line when executing a script without the "-"s.

    2) The script lines are now added to the command history in interactive mode

    3) The double grunt... That's actually harder to fix than it thought, but I added a newline, so it won't say:

    grunt> grunt>

    but:

    grunt>
    grunt>

    Let's just tell everyone that that's because they have extra newlines in their scripts. Maybe they won't find out. ;-)
    run command for grunt
    ---------------------

    Key: PIG-574
    URL: https://issues.apache.org/jira/browse/PIG-574
    Project: Pig
    Issue Type: New Feature
    Components: grunt
    Reporter: David Ciemiewicz
    Priority: Minor
    Attachments: run_command.patch, run_command_params.patch


    This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
    One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
    Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
    I'm used to doing similar things with Oracle, MySQL, and R.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Gunther Hagleitner (JIRA) at Feb 11, 2009 at 9:55 am
    [ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672570#action_12672570 ]

    Gunther Hagleitner commented on PIG-574:
    ----------------------------------------

    Oh, I also ran the unit tests. They pass.
    run command for grunt
    ---------------------

    Key: PIG-574
    URL: https://issues.apache.org/jira/browse/PIG-574
    Project: Pig
    Issue Type: New Feature
    Components: grunt
    Reporter: David Ciemiewicz
    Priority: Minor
    Attachments: run_command.patch, run_command_params.patch


    This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
    One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
    Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
    I'm used to doing similar things with Oracle, MySQL, and R.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • David Ciemiewicz (JIRA) at Feb 11, 2009 at 3:27 pm
    [ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672655#action_12672655 ]

    David Ciemiewicz commented on PIG-574:
    --------------------------------------

    Thanks!

    This will make so iterative development faster and less painful than preallocating a HOD subcluster and then forgetting to delete it.
    run command for grunt
    ---------------------

    Key: PIG-574
    URL: https://issues.apache.org/jira/browse/PIG-574
    Project: Pig
    Issue Type: New Feature
    Components: grunt
    Reporter: David Ciemiewicz
    Priority: Minor
    Attachments: run_command.patch, run_command_params.patch


    This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
    One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
    Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
    I'm used to doing similar things with Oracle, MySQL, and R.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Feb 11, 2009 at 5:59 pm
    [ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672699#action_12672699 ]

    Olga Natkovich commented on PIG-574:
    ------------------------------------

    Gunther, thanks for quick turnaround on the patch.

    Is there a reason why we can't use exactly the same syntax on run/exec command for parameter substituion as we do on Pig command line. I think that might be easier for users to remember and provide a more consistent interface.
    run command for grunt
    ---------------------

    Key: PIG-574
    URL: https://issues.apache.org/jira/browse/PIG-574
    Project: Pig
    Issue Type: New Feature
    Components: grunt
    Reporter: David Ciemiewicz
    Priority: Minor
    Attachments: run_command.patch, run_command_params.patch


    This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
    One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
    Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
    I'm used to doing similar things with Oracle, MySQL, and R.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Gunther Hagleitner (JIRA) at Feb 11, 2009 at 9:32 pm
    [ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Gunther Hagleitner updated PIG-574:
    -----------------------------------

    Attachment: run_command_params_021109.patch

    Good point. I felt it was a little strange to specify "-param" on the grunt shell, but it is easier to remember if your using it outside the shell already.

    So, this patch does the same as the last one, but the syntax is:

    run myscript.pig -param LIMIT=5 -param FILE=/foo/bar.txt -param_file myparams.ppf
    run command for grunt
    ---------------------

    Key: PIG-574
    URL: https://issues.apache.org/jira/browse/PIG-574
    Project: Pig
    Issue Type: New Feature
    Components: grunt
    Reporter: David Ciemiewicz
    Priority: Minor
    Attachments: run_command.patch, run_command_params.patch, run_command_params_021109.patch


    This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
    One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
    Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
    I'm used to doing similar things with Oracle, MySQL, and R.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Feb 12, 2009 at 10:25 pm
    [ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich updated PIG-574:
    -------------------------------

    Attachment: PIG-574.patch
    run command for grunt
    ---------------------

    Key: PIG-574
    URL: https://issues.apache.org/jira/browse/PIG-574
    Project: Pig
    Issue Type: New Feature
    Components: grunt
    Reporter: David Ciemiewicz
    Priority: Minor
    Attachments: PIG-574.patch, run_command.patch, run_command_params.patch, run_command_params_021109.patch


    This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
    One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
    Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
    I'm used to doing similar things with Oracle, MySQL, and R.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Feb 12, 2009 at 10:27 pm
    [ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12673074#action_12673074 ]

    Olga Natkovich commented on PIG-574:
    ------------------------------------

    I tested patch and all is good. I have made one small change - moving parameters in front of the pig script to be consistent with pig cmd syntax. New patch is attached
    run command for grunt
    ---------------------

    Key: PIG-574
    URL: https://issues.apache.org/jira/browse/PIG-574
    Project: Pig
    Issue Type: New Feature
    Components: grunt
    Reporter: David Ciemiewicz
    Priority: Minor
    Attachments: PIG-574.patch, run_command.patch, run_command_params.patch, run_command_params_021109.patch


    This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
    One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
    Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
    I'm used to doing similar things with Oracle, MySQL, and R.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Feb 12, 2009 at 10:29 pm
    [ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich updated PIG-574:
    -------------------------------

    Resolution: Fixed
    Status: Resolved (was: Patch Available)

    Patch committed. Thanks Gunther for contributing!
    run command for grunt
    ---------------------

    Key: PIG-574
    URL: https://issues.apache.org/jira/browse/PIG-574
    Project: Pig
    Issue Type: New Feature
    Components: grunt
    Reporter: David Ciemiewicz
    Priority: Minor
    Attachments: PIG-574.patch, run_command.patch, run_command_params.patch, run_command_params_021109.patch


    This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
    One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
    Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
    I'm used to doing similar things with Oracle, MySQL, and R.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Alan Gates (JIRA) at Nov 12, 2009 at 2:01 am
    [ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Alan Gates reassigned PIG-574:
    ------------------------------

    Assignee: Olga Natkovich
    run command for grunt
    ---------------------

    Key: PIG-574
    URL: https://issues.apache.org/jira/browse/PIG-574
    Project: Pig
    Issue Type: New Feature
    Components: grunt
    Reporter: David Ciemiewicz
    Assignee: Olga Natkovich
    Priority: Minor
    Attachments: PIG-574.patch, run_command.patch, run_command_params.patch, run_command_params_021109.patch


    This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
    One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
    Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
    I'm used to doing similar things with Oracle, MySQL, and R.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriespig, hadoop
postedDec 22, '08 at 6:29p
activeNov 12, '09 at 2:01a
posts14
users1
websitepig.apache.org

1 user in discussion

Alan Gates (JIRA): 14 posts

People

Translate

site design / logo © 2022 Grokbase