Grokbase Groups Pig dev August 2010
FAQ
Hey everyone, I was wondering if anybody has any references or suggestion on
how to learn about Pig's optimizer besides the source code or Pig's paper.
Thanks in advance.


Renato M.

Search Discussions

  • Daniel Dai at Aug 23, 2010 at 10:29 pm
    Hi, Renato,
    There is a description of optimization rule in Pig Latin reference menu:
    http://hadoop.apache.org/pig/docs/r0.7.0/piglatin_ref1.html#Optimization+Rules.
    Is that enough?

    Daniel

    Renato Marroquín Mogrovejo wrote:
    Hey everyone, I was wondering if anybody has any references or suggestion on
    how to learn about Pig's optimizer besides the source code or Pig's paper.
    Thanks in advance.


    Renato M.
  • Renato Marroquín Mogrovejo at Aug 25, 2010 at 2:30 am
    Hi Daniel,

    Thanks, but that was not what I was actually looking. What I want to know is
    for example, how the optimizer work when the bags' logical plans are
    combined, or if all commands are reduced at the end to CO-GROUP commands,
    how is this handled? I know from Pig's paper that the ORDER, and LOAD,
    commands generate new MapReduce jobs, are there any optimizations for the
    physical plans?
    Thanks in advanced.


    Renato M.

    2010/8/23 Daniel Dai <jianyong@yahoo-inc.com>
    Hi, Renato,
    There is a description of optimization rule in Pig Latin reference menu:
    http://hadoop.apache.org/pig/docs/r0.7.0/piglatin_ref1.html#Optimization+Rules.
    Is that enough?

    Daniel


    Renato Marroquín Mogrovejo wrote:
    Hey everyone, I was wondering if anybody has any references or suggestion
    on
    how to learn about Pig's optimizer besides the source code or Pig's paper.
    Thanks in advance.


    Renato M.
  • Renato Marroquín Mogrovejo at Aug 26, 2010 at 3:49 pm
    Anyone, please?

    Renato M.

    2010/8/24 Renato Marroquín Mogrovejo <renatoj.marroquin@gmail.com>
    Hi Daniel,

    Thanks, but that was not what I was actually looking. What I want to know
    is for example, how the optimizer work when the bags' logical plans are
    combined, or if all commands are reduced at the end to CO-GROUP commands,
    how is this handled? I know from Pig's paper that the ORDER, and LOAD,
    commands generate new MapReduce jobs, are there any optimizations for the
    physical plans?
    Thanks in advanced.


    Renato M.

    2010/8/23 Daniel Dai <jianyong@yahoo-inc.com>

    Hi, Renato,
    There is a description of optimization rule in Pig Latin reference menu:
    http://hadoop.apache.org/pig/docs/r0.7.0/piglatin_ref1.html#Optimization+Rules.
    Is that enough?

    Daniel


    Renato Marroquín Mogrovejo wrote:
    Hey everyone, I was wondering if anybody has any references or suggestion
    on
    how to learn about Pig's optimizer besides the source code or Pig's
    paper.
    Thanks in advance.


    Renato M.
  • Daniel Dai at Aug 26, 2010 at 6:13 pm
    Hi, Renato,
    I think you are talking about how we organize different operators into
    map-reduce jobs. Unfortunately there is no document currently. Basically
    we will put as much operators into one map-reduce job as possible.
    Co-group/Group, Join, Order, Distinct, Cross, Stream will create a
    map-reduce boundary; Most others we will put into existing jobs. The
    main logic is inside MRCompiler.java.

    Daniel

    Renato Marroquín Mogrovejo wrote:
    Anyone, please?

    Renato M.

    2010/8/24 Renato Marroquín Mogrovejo <renatoj.marroquin@gmail.com>

    Hi Daniel,

    Thanks, but that was not what I was actually looking. What I want to know
    is for example, how the optimizer work when the bags' logical plans are
    combined, or if all commands are reduced at the end to CO-GROUP commands,
    how is this handled? I know from Pig's paper that the ORDER, and LOAD,
    commands generate new MapReduce jobs, are there any optimizations for the
    physical plans?
    Thanks in advanced.


    Renato M.

    2010/8/23 Daniel Dai <jianyong@yahoo-inc.com>

    Hi, Renato,
    There is a description of optimization rule in Pig Latin reference menu:
    http://hadoop.apache.org/pig/docs/r0.7.0/piglatin_ref1.html#Optimization+Rules.
    Is that enough?

    Daniel


    Renato Marroquín Mogrovejo wrote:

    Hey everyone, I was wondering if anybody has any references or suggestion
    on
    how to learn about Pig's optimizer besides the source code or Pig's
    paper.
    Thanks in advance.


    Renato M.

  • Renato Marroquín Mogrovejo at Aug 28, 2010 at 7:49 pm
    Hi Daniel,

    Yeah that is, but like there are two types of optimizations right? I mean
    physical and logical optimizations. The physical ones are the ones on how
    the operators are distributed along mapreduce jobs and the logical ones are
    the ones similar to relational algebra right?
    Do you have any tips on how to get a quick grasp on pig logical
    optimizations?
    Thanks again.


    Renato M.


    2010/8/26 Daniel Dai <jianyong@yahoo-inc.com>
    Hi, Renato,
    I think you are talking about how we organize different operators into
    map-reduce jobs. Unfortunately there is no document currently. Basically we
    will put as much operators into one map-reduce job as possible.
    Co-group/Group, Join, Order, Distinct, Cross, Stream will create a
    map-reduce boundary; Most others we will put into existing jobs. The main
    logic is inside MRCompiler.java.


    Daniel

    Renato Marroquín Mogrovejo wrote:
    Anyone, please?

    Renato M.

    2010/8/24 Renato Marroquín Mogrovejo <renatoj.marroquin@gmail.com>


    Hi Daniel,

    Thanks, but that was not what I was actually looking. What I want to know
    is for example, how the optimizer work when the bags' logical plans are
    combined, or if all commands are reduced at the end to CO-GROUP commands,
    how is this handled? I know from Pig's paper that the ORDER, and LOAD,
    commands generate new MapReduce jobs, are there any optimizations for the
    physical plans?
    Thanks in advanced.


    Renato M.

    2010/8/23 Daniel Dai <jianyong@yahoo-inc.com>

    Hi, Renato,

    There is a description of optimization rule in Pig Latin reference menu:

    http://hadoop.apache.org/pig/docs/r0.7.0/piglatin_ref1.html#Optimization+Rules
    .
    Is that enough?

    Daniel


    Renato Marroquín Mogrovejo wrote:


    Hey everyone, I was wondering if anybody has any references or
    suggestion
    on
    how to learn about Pig's optimizer besides the source code or Pig's
    paper.
    Thanks in advance.


    Renato M.


Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriespig, hadoop
postedAug 23, '10 at 9:59p
activeAug 28, '10 at 7:49p
posts6
users2
websitepig.apache.org

People

Translate

site design / logo © 2022 Grokbase