Grokbase Groups Pig dev July 2010
FAQ
support jars and scripts in dfs
-------------------------------

Key: PIG-1505
URL: https://issues.apache.org/jira/browse/PIG-1505
Project: Pig
Issue Type: Improvement
Reporter: Andrew Hitchcock


Pig can't operate on files stored in Amazon S3.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Andrew Hitchcock (JIRA) at Jul 16, 2010 at 11:34 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Andrew Hitchcock updated PIG-1505:
    ----------------------------------

    Attachment: pig-jars-and-scripts-from-dfs-trunk.patch

    This patch lets Pig operate on jars and scripts that are hosted within a DFS such as Amazon S3.
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Andrew Hitchcock (JIRA) at Jul 16, 2010 at 11:34 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Andrew Hitchcock updated PIG-1505:
    ----------------------------------

    Status: Patch Available (was: Open)
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Jul 16, 2010 at 11:55 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889421#action_12889421 ]

    Hadoop QA commented on PIG-1505:
    --------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12449736/pig-jars-and-scripts-from-dfs-trunk.patch
    against trunk revision 964182.

    +1 @author. The patch does not contain any @author tags.

    -1 tests included. The patch doesn't appear to include any new or modified tests.
    Please justify why no tests are needed for this patch.

    -1 patch. The patch command could not apply the patch.

    Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/345/console

    This message is automatically generated.
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Andrew Hitchcock (JIRA) at Jul 17, 2010 at 12:02 am
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Andrew Hitchcock updated PIG-1505:
    ----------------------------------

    Attachment: pig-jars-and-scripts-from-dfs-trunk-1.patch

    Fixed the patch to not have prefixes. Should apply now correctly now.
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Alan Gates (JIRA) at Jul 19, 2010 at 7:12 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Alan Gates updated PIG-1505:
    ----------------------------

    Status: Patch Available (was: Open)
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Alan Gates (JIRA) at Jul 19, 2010 at 7:19 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Alan Gates updated PIG-1505:
    ----------------------------

    Status: Open (was: Patch Available)

    Canceling and resubmitting patch so Hudson will run the tests.
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Jul 20, 2010 at 1:17 am
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890118#action_12890118 ]

    Hadoop QA commented on PIG-1505:
    --------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12449741/pig-jars-and-scripts-from-dfs-trunk-1.patch
    against trunk revision 965559.

    +1 @author. The patch does not contain any @author tags.

    -1 tests included. The patch doesn't appear to include any new or modified tests.
    Please justify why no tests are needed for this patch.

    +1 javadoc. The javadoc tool did not generate any warning messages.

    +1 javac. The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs. The patch appears to introduce 3 new Findbugs warnings.

    +1 release audit. The applied patch does not increase the total number of release audit warnings.

    -1 core tests. The patch failed core unit tests.

    -1 contrib tests. The patch failed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/372/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/372/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/372/console

    This message is automatically generated.
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Alan Gates (JIRA) at Jul 21, 2010 at 6:00 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Alan Gates reassigned PIG-1505:
    -------------------------------

    Assignee: Andrew Hitchcock
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Andrew Hitchcock (JIRA) at Jul 21, 2010 at 9:40 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Andrew Hitchcock updated PIG-1505:
    ----------------------------------

    Attachment: pig-jars-and-scripts-from-dfs-trunk-2.patch

    This should fix the findbugs errors.
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Andrew Hitchcock (JIRA) at Jul 21, 2010 at 9:40 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Andrew Hitchcock updated PIG-1505:
    ----------------------------------

    Status: Open (was: Patch Available)
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Andrew Hitchcock (JIRA) at Jul 21, 2010 at 9:47 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Andrew Hitchcock updated PIG-1505:
    ----------------------------------

    Status: Patch Available (was: Open)
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Andrew Hitchcock (JIRA) at Jul 21, 2010 at 9:50 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890913#action_12890913 ]

    Andrew Hitchcock commented on PIG-1505:
    ---------------------------------------

    I think the test failures were unrelated. Can you confirm Alan?
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Richard Ding (JIRA) at Jul 22, 2010 at 12:04 am
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890954#action_12890954 ]

    Richard Ding commented on PIG-1505:
    -----------------------------------

    The patch looks good.

    A couple of comments:

    * The property "pig.initial.fs.name" is set, but I don't see where it's used.
    * Pig is prepared to work in the multi-name-node environment (HDFS-1052). It seems that FileLocalizer.fetchFile method doesn't need to use the default dfs.

    I also looked into the failed tests and they were unrelated to the patch.
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Andrew Hitchcock (JIRA) at Jul 22, 2010 at 12:24 am
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890964#action_12890964 ]

    Andrew Hitchcock commented on PIG-1505:
    ---------------------------------------

    Thanks Richard. Regarding your points.

    * You are correct, that snuck in from another patch I am working on.
    * I'm not sure I understand what you mean on this one. What should I change?
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Richard Ding (JIRA) at Jul 22, 2010 at 1:04 am
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890977#action_12890977 ]

    Richard Ding commented on PIG-1505:
    -----------------------------------


    The line

    {code}
    if (dfs != null){
    src = new Path(dfs.asElement(filePath).toString());
    }
    {code}

    seems unnecessary.

    Do you allow user to register a jar with relative path?
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Andrew Hitchcock (JIRA) at Jul 22, 2010 at 2:11 am
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Andrew Hitchcock updated PIG-1505:
    ----------------------------------

    Status: Patch Available (was: Open)
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-3.patch, pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Andrew Hitchcock (JIRA) at Jul 22, 2010 at 2:11 am
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Andrew Hitchcock updated PIG-1505:
    ----------------------------------

    Status: Open (was: Patch Available)
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-3.patch, pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Andrew Hitchcock (JIRA) at Jul 22, 2010 at 2:14 am
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890985#action_12890985 ]

    Andrew Hitchcock commented on PIG-1505:
    ---------------------------------------

    By default, the relative path is relative to your local file system directory. If you set "pig.jars.relative.to.dfs" to true then you can register jars that are relative to your current grunt working directory, even within HDFS.

    I'll post a patch in a couple of minutes that address your issues.
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Andrew Hitchcock (JIRA) at Jul 22, 2010 at 2:18 am
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Andrew Hitchcock updated PIG-1505:
    ----------------------------------

    Attachment: pig-jars-and-scripts-from-dfs-3.patch

    Removed unnecessary code and code that was intended for a different patch.
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-3.patch, pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Alan Gates (JIRA) at Jul 22, 2010 at 5:41 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891258#action_12891258 ]

    Alan Gates commented on PIG-1505:
    ---------------------------------

    I ran core and contrib tests manually and they both pass. Richard will be reviewing the patch.
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-3.patch, pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Richard Ding (JIRA) at Jul 22, 2010 at 9:24 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891347#action_12891347 ]

    Richard Ding commented on PIG-1505:
    -----------------------------------

    Thank you for the update. A few more comments:

    * According to Pig Latin manual, user can also register additional files (to use with user's Pig script) via the command line using the -Dpig.additional.jars option (in addition to the REGISTER statement inside a Pig script). I suggest you call FileLocalizer.fetchFile from the shared method PigServer.registerJar so both cases will be covered.

    * Can you change the method signature to

    {code}
    public static FetchFileRet fetchFile(Properties properties, String filePath) throws IOException
    {code}

    The reason is that we have deprecated all other public methods on FileLocalizer which has DataStorage as a parameter (so we can deprecate DataStorage in the future). I think this is safe since the condition in the method

    {code}
    ((fileUri.getScheme() == null) && (dfs == null))
    {code}

    is not used in the patch.

    * You need to add a unit test in the patch (by first copying a Pig script to the mini-cluster).

    * Finally, since this is a new feature, can you add a release note (On jira, there is a Release Note field) so that it will be incorporated in the next Pig release notes.


    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-3.patch, pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Andrew Hitchcock (JIRA) at Jul 23, 2010 at 12:57 am
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891436#action_12891436 ]

    Andrew Hitchcock commented on PIG-1505:
    ---------------------------------------

    Thanks Richard. Is there a unit test you recommend that I can model mine after? Something that uses the mini-cluster.
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-3.patch, pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Richard Ding (JIRA) at Jul 23, 2010 at 5:51 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891685#action_12891685 ]

    Richard Ding commented on PIG-1505:
    -----------------------------------


    You can take a look at the test cases in TestPigRunner where local Pig scripts are passed to the PigRunner.run method.

    You can first copy a local Pig script to the mini-cluster using

    {code}
    Util.copyFromLocalToCluster(cluster, <localScriptFileName>, <scriptFileNameOnCluster>);
    {code}

    and then invoke run method with argument

    {code}
    String[] args = { "-f", "hdfs://<scriptFileNameOnCluste>" };
    PigRunner.run(args, null);
    {code}
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-3.patch, pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Jul 27, 2010 at 12:27 am
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892564#action_12892564 ]

    Hadoop QA commented on PIG-1505:
    --------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12450123/pig-jars-and-scripts-from-dfs-3.patch
    against trunk revision 979362.

    +1 @author. The patch does not contain any @author tags.

    -1 tests included. The patch doesn't appear to include any new or modified tests.
    Please justify why no tests are needed for this patch.

    +1 javadoc. The javadoc tool did not generate any warning messages.

    +1 javac. The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs. The patch does not introduce any new Findbugs warnings.

    +1 release audit. The applied patch does not increase the total number of release audit warnings.

    -1 core tests. The patch failed core unit tests.

    -1 contrib tests. The patch failed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/379/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/379/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/379/console

    This message is automatically generated.
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: pig-jars-and-scripts-from-dfs-3.patch, pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Andrew Hitchcock (JIRA) at Aug 19, 2010 at 11:21 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Andrew Hitchcock updated PIG-1505:
    ----------------------------------

    Status: Open (was: Patch Available)
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: PIG-1505-4.patch, pig-jars-and-scripts-from-dfs-3.patch, pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Andrew Hitchcock (JIRA) at Aug 19, 2010 at 11:21 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Andrew Hitchcock updated PIG-1505:
    ----------------------------------

    Attachment: PIG-1505-4.patch

    Rebased to trunk and added unit test. I think this should be everything.
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: PIG-1505-4.patch, pig-jars-and-scripts-from-dfs-3.patch, pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Andrew Hitchcock (JIRA) at Aug 19, 2010 at 11:25 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Andrew Hitchcock updated PIG-1505:
    ----------------------------------

    Status: Patch Available (was: Open)
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: PIG-1505-4.patch, pig-jars-and-scripts-from-dfs-3.patch, pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Andrew Hitchcock (JIRA) at Aug 19, 2010 at 11:27 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Andrew Hitchcock updated PIG-1505:
    ----------------------------------

    Release Note: Pig now supports running scripts and registering jars that are stored in HDFS, Amazon S3, or other distributed file systems. Also added a -R parameter which allows users to specify properties in key=value form on the command line.

    Added release notes.
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: PIG-1505-4.patch, pig-jars-and-scripts-from-dfs-3.patch, pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Richard Ding (JIRA) at Aug 20, 2010 at 6:18 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900811#action_12900811 ]

    Richard Ding commented on PIG-1505:
    -----------------------------------


    The results of test-patch:

    {code}
    [exec] +1 overall.
    [exec]
    [exec] +1 @author. The patch does not contain any @author tags.
    [exec]
    [exec] +1 tests included. The patch appears to include 3 new or modified tests.
    [exec]
    [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
    [exec]
    [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
    [exec]
    [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
    [exec]
    [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
    {code}

    I'll commit the patch after running unit tests.



    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Attachments: PIG-1505-4.patch, pig-jars-and-scripts-from-dfs-3.patch, pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Richard Ding (JIRA) at Aug 20, 2010 at 6:27 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Richard Ding updated PIG-1505:
    ------------------------------

    Fix Version/s: 0.8.0
    Affects Version/s: 0.7.0
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Affects Versions: 0.7.0
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Fix For: 0.8.0

    Attachments: PIG-1505-4.patch, pig-jars-and-scripts-from-dfs-3.patch, pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Richard Ding (JIRA) at Aug 21, 2010 at 12:43 am
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Richard Ding updated PIG-1505:
    ------------------------------

    Status: Resolved (was: Patch Available)
    Hadoop Flags: [Reviewed]
    Resolution: Fixed

    All core tests passed. The patch is committed to the trunk.

    Thanks Andrew for contributing this feature!
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Affects Versions: 0.7.0
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Fix For: 0.8.0

    Attachments: PIG-1505-4.patch, pig-jars-and-scripts-from-dfs-3.patch, pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Richard Ding (JIRA) at Aug 23, 2010 at 8:24 pm
    [ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Richard Ding updated PIG-1505:
    ------------------------------

    Release Note: Pig now supports running scripts and registering jars that are stored in HDFS, Amazon S3, or other distributed file systems. (was: Pig now supports running scripts and registering jars that are stored in HDFS, Amazon S3, or other distributed file systems. Also added a -R parameter which allows users to specify properties in key=value form on the command line.)

    Remove -R option. In 0.8 Pig supports generic parameters such as -Dkey=value.
    support jars and scripts in dfs
    -------------------------------

    Key: PIG-1505
    URL: https://issues.apache.org/jira/browse/PIG-1505
    Project: Pig
    Issue Type: Improvement
    Affects Versions: 0.7.0
    Reporter: Andrew Hitchcock
    Assignee: Andrew Hitchcock
    Fix For: 0.8.0

    Attachments: PIG-1505-4.patch, pig-jars-and-scripts-from-dfs-3.patch, pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch


    Pig can't operate on files stored in Amazon S3.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriespig, hadoop
postedJul 16, '10 at 11:32p
activeAug 23, '10 at 8:24p
posts33
users1
websitepig.apache.org

1 user in discussion

Richard Ding (JIRA): 33 posts

People

Translate

site design / logo © 2021 Grokbase