Grokbase Groups Pig user May 2009
FAQ
Hi,

Yesterday we committed support for multiquery into Pig trunk. This
functionality allows to share scans and other computations across
multiple stores within the same pig script. The details are outlined in
http://wiki.apache.org/pig/PigMultiQueryPerformanceSpecification. This
functionality will go into Pig 0.3.0 to be release in late May or early
June but please give it a try and see whether it improves performance of
your queries. We also made some changes to commands like describe and
explain to work with multiquery - the changes are outlined in the same
document.

Olga

Search Discussions

  • Tamir Kamara at May 5, 2009 at 5:41 pm
    Hi Olga,

    Recent changes cause the build to fail. ant output is below.

    Thanks,
    Tamir

    Buildfile: build.xml

    init:
    [mkdir] Created dir:
    /app/home/tkamara/pig1/src-gen/org/apache/pig/impl/logicalLayer/parser
    [mkdir] Created dir:
    /app/home/tkamara/pig1/src-gen/org/apache/pig/tools/pigscript/parser
    [mkdir] Created dir:
    /app/home/tkamara/pig1/src-gen/org/apache/pig/tools/parameters
    [mkdir] Created dir: /app/home/tkamara/pig1/build/pig-0.3.0-dev
    [mkdir] Created dir: /app/home/tkamara/pig1/build/classes
    [mkdir] Created dir: /app/home/tkamara/pig1/build/test/classes
    [mkdir] Created dir:
    /app/home/tkamara/pig1/test/org/apache/pig/test/utils/dotGraph/parser
    [mkdir] Created dir:
    /app/home/tkamara/pig1/src-gen/org/apache/pig/data/parser

    ivy-download:
    [get] Getting:
    http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar
    [get] To: /app/home/tkamara/pig1/ivy/ivy-2.0.0-rc2.jar

    ivy-init-dirs:
    [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy
    [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy/lib
    [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy/report
    [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy/maven

    ivy-probe-antlib:

    BUILD FAILED
    /app/home/tkamara/pig1/build.xml:727: Class
    org.apache.tools.ant.taskdefs.ConditionTask doesn't support the nested
    "typefound" element.


    On Tue, May 5, 2009 at 7:12 PM, Olga Natkovich wrote:

    Hi,

    Yesterday we committed support for multiquery into Pig trunk. This
    functionality allows to share scans and other computations across
    multiple stores within the same pig script. The details are outlined in
    http://wiki.apache.org/pig/PigMultiQueryPerformanceSpecification. This
    functionality will go into Pig 0.3.0 to be release in late May or early
    June but please give it a try and see whether it improves performance of
    your queries. We also made some changes to commands like describe and
    explain to work with multiquery - the changes are outlined in the same
    document.

    Olga
  • Olga Natkovich at May 5, 2009 at 7:35 pm
    What version of ant do you have on your system? I don't have problems
    building and have ant 1.7.0 and 1.7.1 installed on my machines.

    Also, this is not related to multiquery commit but looks like
    integration with ivy that was also done recently.

    Olga
    -----Original Message-----
    From: Tamir Kamara
    Sent: Tuesday, May 05, 2009 10:40 AM
    To: pig-user@hadoop.apache.org
    Subject: Re: Multiquery support

    Hi Olga,

    Recent changes cause the build to fail. ant output is below.

    Thanks,
    Tamir

    Buildfile: build.xml

    init:
    [mkdir] Created dir:
    /app/home/tkamara/pig1/src-gen/org/apache/pig/impl/logicalLayer/parser
    [mkdir] Created dir:
    /app/home/tkamara/pig1/src-gen/org/apache/pig/tools/pigscript/parser
    [mkdir] Created dir:
    /app/home/tkamara/pig1/src-gen/org/apache/pig/tools/parameters
    [mkdir] Created dir: /app/home/tkamara/pig1/build/pig-0.3.0-dev
    [mkdir] Created dir: /app/home/tkamara/pig1/build/classes
    [mkdir] Created dir: /app/home/tkamara/pig1/build/test/classes
    [mkdir] Created dir:
    /app/home/tkamara/pig1/test/org/apache/pig/test/utils/dotGraph/parser
    [mkdir] Created dir:
    /app/home/tkamara/pig1/src-gen/org/apache/pig/data/parser

    ivy-download:
    [get] Getting:
    http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy
    -2.0.0-rc2.jar
    [get] To: /app/home/tkamara/pig1/ivy/ivy-2.0.0-rc2.jar

    ivy-init-dirs:
    [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy
    [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy/lib
    [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy/report
    [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy/maven

    ivy-probe-antlib:

    BUILD FAILED
    /app/home/tkamara/pig1/build.xml:727: Class
    org.apache.tools.ant.taskdefs.ConditionTask doesn't support
    the nested "typefound" element.



    On Tue, May 5, 2009 at 7:12 PM, Olga Natkovich
    wrote:
    Hi,

    Yesterday we committed support for multiquery into Pig trunk. This
    functionality allows to share scans and other computations across
    multiple stores within the same pig script. The details are outlined
    in
    http://wiki.apache.org/pig/PigMultiQueryPerformanceSpecification.
    This functionality will go into Pig 0.3.0 to be release in
    late May or
    early June but please give it a try and see whether it improves
    performance of your queries. We also made some changes to commands
    like describe and explain to work with multiquery - the changes are
    outlined in the same document.

    Olga
  • Tamir Kamara at May 6, 2009 at 6:16 am
    You're right - with ant 1.6.5 the build fails but with 1.7.1 it's builds
    just fine.

    Thanks.

    On Tue, May 5, 2009 at 10:33 PM, Olga Natkovich wrote:

    What version of ant do you have on your system? I don't have problems
    building and have ant 1.7.0 and 1.7.1 installed on my machines.

    Also, this is not related to multiquery commit but looks like
    integration with ivy that was also done recently.

    Olga
    -----Original Message-----
    From: Tamir Kamara
    Sent: Tuesday, May 05, 2009 10:40 AM
    To: pig-user@hadoop.apache.org
    Subject: Re: Multiquery support

    Hi Olga,

    Recent changes cause the build to fail. ant output is below.

    Thanks,
    Tamir

    Buildfile: build.xml

    init:
    [mkdir] Created dir:
    /app/home/tkamara/pig1/src-gen/org/apache/pig/impl/logicalLayer/parser
    [mkdir] Created dir:
    /app/home/tkamara/pig1/src-gen/org/apache/pig/tools/pigscript/parser
    [mkdir] Created dir:
    /app/home/tkamara/pig1/src-gen/org/apache/pig/tools/parameters
    [mkdir] Created dir: /app/home/tkamara/pig1/build/pig-0.3.0-dev
    [mkdir] Created dir: /app/home/tkamara/pig1/build/classes
    [mkdir] Created dir: /app/home/tkamara/pig1/build/test/classes
    [mkdir] Created dir:
    /app/home/tkamara/pig1/test/org/apache/pig/test/utils/dotGraph/parser
    [mkdir] Created dir:
    /app/home/tkamara/pig1/src-gen/org/apache/pig/data/parser

    ivy-download:
    [get] Getting:
    http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy
    -2.0.0-rc2.jar
    [get] To: /app/home/tkamara/pig1/ivy/ivy-2.0.0-rc2.jar

    ivy-init-dirs:
    [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy
    [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy/lib
    [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy/report
    [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy/maven

    ivy-probe-antlib:

    BUILD FAILED
    /app/home/tkamara/pig1/build.xml:727: Class
    org.apache.tools.ant.taskdefs.ConditionTask doesn't support
    the nested "typefound" element.



    On Tue, May 5, 2009 at 7:12 PM, Olga Natkovich
    wrote:
    Hi,

    Yesterday we committed support for multiquery into Pig trunk. This
    functionality allows to share scans and other computations across
    multiple stores within the same pig script. The details are outlined
    in
    http://wiki.apache.org/pig/PigMultiQueryPerformanceSpecification.
    This functionality will go into Pig 0.3.0 to be release in
    late May or
    early June but please give it a try and see whether it improves
    performance of your queries. We also made some changes to commands
    like describe and explain to work with multiquery - the changes are
    outlined in the same document.

    Olga

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedMay 5, '09 at 4:15p
activeMay 6, '09 at 6:16a
posts4
users2
websitepig.apache.org

2 users in discussion

Olga Natkovich: 2 posts Tamir Kamara: 2 posts

People

Translate

site design / logo © 2021 Grokbase