FAQ
Hi all ,

We are trying to work on the Optimizer part of hive . Can anyone point me to
a document/link containing how Hive designs it's query plans ,
what meta data it uses , how it optimizes the plans (pushing down selects
etc) ... Iam facing difficulties is reading the source code , so it would
be nice if any one point me to the link containing the above data or any
document containing the same.

Any help is highly appreciated !

Thanks !

Search Discussions

  • Ning Zhang at Oct 13, 2009 at 1:41 pm
    The design doc in Hive's wiki page provides some high level
    descriptions. See http://wiki.apache.org/hadoop/Hive/Design
    the "compilation" and "optimization" sections.

    Ning
    On Oct 13, 2009, at 4:09 AM, bharath vissapragada wrote:

    Hi all ,

    We are trying to work on the Optimizer part of hive . Can anyone
    point me to a document/link containing how Hive designs it's query
    plans ,
    what meta data it uses , how it optimizes the plans (pushing down
    selects etc) ... Iam facing difficulties is reading the source
    code , so it would
    be nice if any one point me to the link containing the above data or
    any document containing the same.

    Any help is highly appreciated !

    Thanks !
  • Bharath v at Oct 13, 2009 at 5:02 pm
    Thanks for your reply Ning ,

    I saw that page , but we need some detailed description .. such as how
    optimizations are carried out .. how ASTs are built etc ..
    Do u (or any others) have any idea abt them .. Kindly reply

    Thanks !
    On Tue, Oct 13, 2009 at 7:10 PM, Ning Zhang wrote:

    The design doc in Hive's wiki page provides some high level
    descriptions. See http://wiki.apache.org/hadoop/Hive/Design
    the "compilation" and "optimization" sections.

    Ning
    On Oct 13, 2009, at 4:09 AM, bharath vissapragada wrote:

    Hi all ,

    We are trying to work on the Optimizer part of hive . Can anyone
    point me to a document/link containing how Hive designs it's query
    plans ,
    what meta data it uses , how it optimizes the plans (pushing down
    selects etc) ... Iam facing difficulties is reading the source
    code , so it would
    be nice if any one point me to the link containing the above data or
    any document containing the same.

    Any help is highly appreciated !

    Thanks !
  • Zheng Shao at Oct 13, 2009 at 5:42 pm
    AST are built by Hive.g (Using antlr).
    AST -> OperatorTree is done by SemanticAnalyzer.java.
    Optimizations are done by Transformer.java and its sub classes.
    Hope this help you get started.

    Zheng
    On Tue, Oct 13, 2009 at 10:01 AM, bharath v wrote:

    Thanks for your reply Ning ,

    I saw that page , but we need some detailed description .. such as how
    optimizations are carried out .. how ASTs are built etc ..
    Do u (or any others) have any idea abt them .. Kindly reply

    Thanks !

    On Tue, Oct 13, 2009 at 7:10 PM, Ning Zhang wrote:

    The design doc in Hive's wiki page provides some high level
    descriptions. See http://wiki.apache.org/hadoop/Hive/Design
    the "compilation" and "optimization" sections.

    Ning
    On Oct 13, 2009, at 4:09 AM, bharath vissapragada wrote:

    Hi all ,

    We are trying to work on the Optimizer part of hive . Can anyone
    point me to a document/link containing how Hive designs it's query
    plans ,
    what meta data it uses , how it optimizes the plans (pushing down
    selects etc) ... Iam facing difficulties is reading the source
    code , so it would
    be nice if any one point me to the link containing the above data or
    any document containing the same.

    Any help is highly appreciated !

    Thanks !

    --
    Yours,
    Zheng
  • Ashish Thusoo at Oct 13, 2009 at 6:07 pm
    We will also add some more information on that page about the tree walker interface and how it is used to do the optimizations.

    Ashish

    ________________________________
    From: Zheng Shao
    Sent: Tuesday, October 13, 2009 10:42 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: Current Hive Optimizer

    AST are built by Hive.g (Using antlr).
    AST -> OperatorTree is done by SemanticAnalyzer.java.
    Optimizations are done by Transformer.java and its sub classes.
    Hope this help you get started.

    Zheng

    On Tue, Oct 13, 2009 at 10:01 AM, bharath v wrote:
    Thanks for your reply Ning ,

    I saw that page , but we need some detailed description .. such as how optimizations are carried out .. how ASTs are built etc ..
    Do u (or any others) have any idea abt them .. Kindly reply

    Thanks !


    On Tue, Oct 13, 2009 at 7:10 PM, Ning Zhang wrote:
    The design doc in Hive's wiki page provides some high level
    descriptions. See http://wiki.apache.org/hadoop/Hive/Design
    the "compilation" and "optimization" sections.

    Ning
    On Oct 13, 2009, at 4:09 AM, bharath vissapragada wrote:

    Hi all ,

    We are trying to work on the Optimizer part of hive . Can anyone
    point me to a document/link containing how Hive designs it's query
    plans ,
    what meta data it uses , how it optimizes the plans (pushing down
    selects etc) ... Iam facing difficulties is reading the source
    code , so it would
    be nice if any one point me to the link containing the above data or
    any document containing the same.

    Any help is highly appreciated !

    Thanks !




    --
    Yours,
    Zheng
  • Bharath v at Oct 13, 2009 at 7:21 pm
    Kindly inform if on the mailing list , once you update it .. Also do u
    suggest any external links/docs specifying the same ?

    I wanted to no in detail about the way the hive optimizer works and how MR
    Job plan is created .. Thanks for ur help !

    Thanks!

    On Tue, Oct 13, 2009 at 11:23 PM, Ashish Thusoo wrote:

    We will also add some more information on that page about the tree walker
    interface and how it is used to do the optimizations.

    Ashish

    ------------------------------
    *From:* Zheng Shao
    *Sent:* Tuesday, October 13, 2009 10:42 AM
    *To:* hive-user@hadoop.apache.org
    *Subject:* Re: Current Hive Optimizer

    AST are built by Hive.g (Using antlr).
    AST -> OperatorTree is done by SemanticAnalyzer.java.
    Optimizations are done by Transformer.java and its sub classes.
    Hope this help you get started.

    Zheng

    On Tue, Oct 13, 2009 at 10:01 AM, bharath v <
    bharathvissapragada1990@gmail.com> wrote:
    Thanks for your reply Ning ,

    I saw that page , but we need some detailed description .. such as how
    optimizations are carried out .. how ASTs are built etc ..
    Do u (or any others) have any idea abt them .. Kindly reply

    Thanks !

    On Tue, Oct 13, 2009 at 7:10 PM, Ning Zhang wrote:

    The design doc in Hive's wiki page provides some high level
    descriptions. See http://wiki.apache.org/hadoop/Hive/Design
    the "compilation" and "optimization" sections.

    Ning
    On Oct 13, 2009, at 4:09 AM, bharath vissapragada wrote:

    Hi all ,

    We are trying to work on the Optimizer part of hive . Can anyone
    point me to a document/link containing how Hive designs it's query
    plans ,
    what meta data it uses , how it optimizes the plans (pushing down
    selects etc) ... Iam facing difficulties is reading the source
    code , so it would
    be nice if any one point me to the link containing the above data or
    any document containing the same.

    Any help is highly appreciated !

    Thanks !

    --
    Yours,
    Zheng
  • Bharath v at Oct 13, 2009 at 7:25 pm
    Thanks for ur reply zheng ,

    I am facing difficulties in understanding their code ..because i am not yet
    acquainted to that style of coding ..
    Thats the reason why i was asking about the documentation and stuff ..
    Anyway i'll try u understand the code ..

    Thanks!
    On Tue, Oct 13, 2009 at 11:12 PM, Zheng Shao wrote:

    AST are built by Hive.g (Using antlr).
    AST -> OperatorTree is done by SemanticAnalyzer.java.
    Optimizations are done by Transformer.java and its sub classes.
    Hope this help you get started.

    Zheng


    On Tue, Oct 13, 2009 at 10:01 AM, bharath v <
    bharathvissapragada1990@gmail.com> wrote:
    Thanks for your reply Ning ,

    I saw that page , but we need some detailed description .. such as how
    optimizations are carried out .. how ASTs are built etc ..
    Do u (or any others) have any idea abt them .. Kindly reply

    Thanks !

    On Tue, Oct 13, 2009 at 7:10 PM, Ning Zhang wrote:

    The design doc in Hive's wiki page provides some high level
    descriptions. See http://wiki.apache.org/hadoop/Hive/Design
    the "compilation" and "optimization" sections.

    Ning
    On Oct 13, 2009, at 4:09 AM, bharath vissapragada wrote:

    Hi all ,

    We are trying to work on the Optimizer part of hive . Can anyone
    point me to a document/link containing how Hive designs it's query
    plans ,
    what meta data it uses , how it optimizes the plans (pushing down
    selects etc) ... Iam facing difficulties is reading the source
    code , so it would
    be nice if any one point me to the link containing the above data or
    any document containing the same.

    Any help is highly appreciated !

    Thanks !

    --
    Yours,
    Zheng

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedOct 13, '09 at 11:10a
activeOct 13, '09 at 7:25p
posts7
users5
websitehive.apache.org

People

Translate

site design / logo © 2022 Grokbase