Grokbase Groups Pig user June 2009
FAQ
Hi pig users,
I tried to copyToLocal my stored result from pig queries to my local
workspace. My lines of code in Java are:

........
pigServer.store("B","output");
*pigServer.registerQuery("copyToLocal output WorkingDir output ");*

And I know Pig Latin statements will execute only at "store" and "dump", so
I think the last line of code won't get executed.
so I tried to add another line:
pigServer.registerQuery( "quit;" );
*
*
This will work in Grunt, but not here. So what's the best practice to copy
the file to a local working directory for reading or further processing?
Should I use the interface in Hadoop?
Thank you.

George

Search Discussions

  • Kevin Weil at Jun 16, 2009 at 4:51 pm
    If you're already writing Pig from within Java, your best bet is to probably
    go through the standard HDFS interfaces. In particular, see
    http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/fs/FileUtil.htmlfor
    a utility class that exposes copy method from HDFS to the local file
    system.

    Kevin
    On Thu, Jun 11, 2009 at 11:46 AM, George Pang wrote:

    Hi pig users,
    I tried to copyToLocal my stored result from pig queries to my local
    workspace. My lines of code in Java are:

    ........
    pigServer.store("B","output");
    *pigServer.registerQuery("copyToLocal output WorkingDir output ");*

    And I know Pig Latin statements will execute only at "store" and "dump", so
    I think the last line of code won't get executed.
    so I tried to add another line:
    pigServer.registerQuery( "quit;" );
    *
    *
    This will work in Grunt, but not here. So what's the best practice to copy
    the file to a local working directory for reading or further processing?
    Should I use the interface in Hadoop?
    Thank you.

    George
  • George Pang at Jun 16, 2009 at 5:04 pm
    Thank you Kevin, this is one option. But my question to the Pig guru is, is
    there API for file IO between HDFS and local system, or will be there in the
    future?
    George

    2009/6/16 Kevin Weil <kevinweil@gmail.com>
    If you're already writing Pig from within Java, your best bet is to
    probably
    go through the standard HDFS interfaces. In particular, see

    http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/fs/FileUtil.htmlfor
    a utility class that exposes copy method from HDFS to the local file
    system.

    Kevin
    On Thu, Jun 11, 2009 at 11:46 AM, George Pang wrote:

    Hi pig users,
    I tried to copyToLocal my stored result from pig queries to my local
    workspace. My lines of code in Java are:

    ........
    pigServer.store("B","output");
    *pigServer.registerQuery("copyToLocal output WorkingDir output ");*

    And I know Pig Latin statements will execute only at "store" and "dump", so
    I think the last line of code won't get executed.
    so I tried to add another line:
    pigServer.registerQuery( "quit;" );
    *
    *
    This will work in Grunt, but not here. So what's the best practice to copy
    the file to a local working directory for reading or further processing?
    Should I use the interface in Hadoop?
    Thank you.

    George
  • Alan Gates at Jun 16, 2009 at 5:27 pm
    It is odd that PigServer supports some fs operations (mkdirs,
    deleteFile) and not others (copyToLocal). Perhaps some of the
    original designers of this class could chime in on the thinking here.
    I do not know of any immediate plans to alter this interface. Kevin's
    suggestion of using the hadoop classes directly is good.

    Alan.
    On Jun 16, 2009, at 10:04 AM, George Pang wrote:

    Thank you Kevin, this is one option. But my question to the Pig
    guru is, is
    there API for file IO between HDFS and local system, or will be
    there in the
    future?
    George

    2009/6/16 Kevin Weil <kevinweil@gmail.com>
    If you're already writing Pig from within Java, your best bet is to
    probably
    go through the standard HDFS interfaces. In particular, see

    http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/fs/FileUtil.htmlfor
    a utility class that exposes copy method from HDFS to the local file
    system.

    Kevin

    On Thu, Jun 11, 2009 at 11:46 AM, George Pang <p0941p@gmail.com>
    wrote:
    Hi pig users,
    I tried to copyToLocal my stored result from pig queries to my local
    workspace. My lines of code in Java are:

    ........
    pigServer.store("B","output");
    *pigServer.registerQuery("copyToLocal output WorkingDir output ");*

    And I know Pig Latin statements will execute only at "store" and
    "dump", so
    I think the last line of code won't get executed.
    so I tried to add another line:
    pigServer.registerQuery( "quit;" );
    *
    *
    This will work in Grunt, but not here. So what's the best
    practice to copy
    the file to a local working directory for reading or further
    processing?
    Should I use the interface in Hadoop?
    Thank you.

    George

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedJun 11, '09 at 6:46p
activeJun 16, '09 at 5:27p
posts4
users3
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase