Grokbase Groups Pig user March 2011
FAQ
Hi,
I tried to use exec to stop multiquery optimizer from combining too
many actions together, which will result in heap space problem. But it
seems multiquery just ignores exec, and still combines actions before
and after exec together. Disable multiquery works for me, just
wondering whether it is a bug? or my misunderstanding of exec?

Regards,
Shawn

Search Discussions

  • Richard Ding at Mar 17, 2011 at 6:21 pm
    What you want is the 'run' command. With the run command, every store triggers execution.

    Thanks
    -- Richard



    On 3/17/11 10:48 AM, "Xiaomeng Wan" wrote:

    Hi,
    I tried to use exec to stop multiquery optimizer from combining too
    many actions together, which will result in heap space problem. But it
    seems multiquery just ignores exec, and still combines actions before
    and after exec together. Disable multiquery works for me, just
    wondering whether it is a bug? or my misunderstanding of exec?

    Regards,
    Shawn
  • Xiaomeng Wan at Mar 17, 2011 at 7:32 pm
    Hi Richard,

    In my case, there is no store or dump involved. My code is like:

    a = ...
    b = ...
    c = ...
    EXEC;
    d = ...
    e = ...
    ...

    By adding the "EXEC;", i try to stop multiquery optimizer from
    combining all a, b, c, d, e into a single MR plan. In other words, do
    a, b, c in one MR and d, e in another. I guess it is not doable right
    now except disable multiquery or add a "store c into ..." before EXEC.

    Shawn
    On Thu, Mar 17, 2011 at 12:20 PM, Richard Ding wrote:
    What you want is the 'run' command. With the run command, every store triggers execution.

    Thanks
    -- Richard



    On 3/17/11 10:48 AM, "Xiaomeng Wan" wrote:

    Hi,
    I tried to use exec to stop multiquery optimizer from combining too
    many actions together, which will result in heap space problem. But it
    seems multiquery just ignores exec, and still combines actions before
    and after exec together. Disable multiquery works for me, just
    wondering whether it is a bug? or my misunderstanding of exec?

    Regards,
    Shawn
  • Richard Ding at Mar 17, 2011 at 8:59 pm
    Hi Shawn,

    Inserting "exec" in a script forces the Pig statements before "exec" to run. But if there is no store before the "exec" statement, it becomes a no-op. If you want to disable multiquery with "exec" statement, just add "exec" after each store. This way Pig will independently execute each store.

    Thanks
    -- Richard





    On 3/17/11 12:32 PM, "Xiaomeng Wan" wrote:

    Hi Richard,

    In my case, there is no store or dump involved. My code is like:

    a = ...
    b = ...
    c = ...
    EXEC;
    d = ...
    e = ...
    ...

    By adding the "EXEC;", i try to stop multiquery optimizer from
    combining all a, b, c, d, e into a single MR plan. In other words, do
    a, b, c in one MR and d, e in another. I guess it is not doable right
    now except disable multiquery or add a "store c into ..." before EXEC.

    Shawn
    On Thu, Mar 17, 2011 at 12:20 PM, Richard Ding wrote:
    What you want is the 'run' command. With the run command, every store triggers execution.

    Thanks
    -- Richard



    On 3/17/11 10:48 AM, "Xiaomeng Wan" wrote:

    Hi,
    I tried to use exec to stop multiquery optimizer from combining too
    many actions together, which will result in heap space problem. But it
    seems multiquery just ignores exec, and still combines actions before
    and after exec together. Disable multiquery works for me, just
    wondering whether it is a bug? or my misunderstanding of exec?

    Regards,
    Shawn

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedMar 17, '11 at 5:48p
activeMar 17, '11 at 8:59p
posts4
users2
websitepig.apache.org

2 users in discussion

Xiaomeng Wan: 2 posts Richard Ding: 2 posts

People

Translate

site design / logo © 2021 Grokbase