Grokbase Groups Pig dev August 2008
FAQ
SplitIntroducer does not handle some cases correctly
----------------------------------------------------

Key: PIG-401
URL: https://issues.apache.org/jira/browse/PIG-401
Project: Pig
Issue Type: Bug
Affects Versions: types_branch
Reporter: Pradeep Kamath
Fix For: types_branch
Attachments: PIG-401-partial.patch

The splitIntroducer does not handle the following cases correctly:
* As soon as SplitIntroducer sees an operator which does NOT have multiple outputs it stops - it should still recursively check the successors:
Current code which needs to change:
{code}
List<LogicalOperator> sucs = mPlan.getSuccessors(root);
if(sucs==null) return;
int size = sucs.size();
if(size==0 || size==1) return; --> THIS should change to recursively process the succesor if it exists(size == 1 case)
{code}

* If there are more than one operators in the plan which need implicit splits to be introduced
* If the new SPLIT operator is introduced before a cogroup, the cogroup's inner map for it's inputs and GroupBy expressions needs to be updated
* The kind of changes need for cogroup maybe needed for some other operators

I started looking at it some and have a patch for the first two issues - *UNTESTED* which I am attaching to this issue

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Pradeep Kamath (JIRA) at Aug 27, 2008 at 11:02 pm
    [ https://issues.apache.org/jira/browse/PIG-401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Pradeep Kamath updated PIG-401:
    -------------------------------

    Attachment: PIG-401-partial.patch
    SplitIntroducer does not handle some cases correctly
    ----------------------------------------------------

    Key: PIG-401
    URL: https://issues.apache.org/jira/browse/PIG-401
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Pradeep Kamath
    Fix For: types_branch

    Attachments: PIG-401-partial.patch


    The splitIntroducer does not handle the following cases correctly:
    * As soon as SplitIntroducer sees an operator which does NOT have multiple outputs it stops - it should still recursively check the successors:
    Current code which needs to change:
    {code}
    List<LogicalOperator> sucs = mPlan.getSuccessors(root);
    if(sucs==null) return;
    int size = sucs.size();
    if(size==0 || size==1) return; --> THIS should change to recursively process the succesor if it exists(size == 1 case)
    {code}
    * If there are more than one operators in the plan which need implicit splits to be introduced
    * If the new SPLIT operator is introduced before a cogroup, the cogroup's inner map for it's inputs and GroupBy expressions needs to be updated
    * The kind of changes need for cogroup maybe needed for some other operators
    I started looking at it some and have a patch for the first two issues - *UNTESTED* which I am attaching to this issue
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Sep 4, 2008 at 12:38 am
    [ https://issues.apache.org/jira/browse/PIG-401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich reassigned PIG-401:
    ----------------------------------

    Assignee: Alan Gates
    SplitIntroducer does not handle some cases correctly
    ----------------------------------------------------

    Key: PIG-401
    URL: https://issues.apache.org/jira/browse/PIG-401
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Pradeep Kamath
    Assignee: Alan Gates
    Fix For: types_branch

    Attachments: PIG-401-partial.patch


    The splitIntroducer does not handle the following cases correctly:
    * As soon as SplitIntroducer sees an operator which does NOT have multiple outputs it stops - it should still recursively check the successors:
    Current code which needs to change:
    {code}
    List<LogicalOperator> sucs = mPlan.getSuccessors(root);
    if(sucs==null) return;
    int size = sucs.size();
    if(size==0 || size==1) return; --> THIS should change to recursively process the succesor if it exists(size == 1 case)
    {code}
    * If there are more than one operators in the plan which need implicit splits to be introduced
    * If the new SPLIT operator is introduced before a cogroup, the cogroup's inner map for it's inputs and GroupBy expressions needs to be updated
    * The kind of changes need for cogroup maybe needed for some other operators
    I started looking at it some and have a patch for the first two issues - *UNTESTED* which I am attaching to this issue
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Sep 4, 2008 at 11:54 pm
    [ https://issues.apache.org/jira/browse/PIG-401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich updated PIG-401:
    -------------------------------

    Priority: Critical (was: Major)
    SplitIntroducer does not handle some cases correctly
    ----------------------------------------------------

    Key: PIG-401
    URL: https://issues.apache.org/jira/browse/PIG-401
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Pradeep Kamath
    Assignee: Alan Gates
    Priority: Critical
    Fix For: types_branch

    Attachments: PIG-401-partial.patch


    The splitIntroducer does not handle the following cases correctly:
    * As soon as SplitIntroducer sees an operator which does NOT have multiple outputs it stops - it should still recursively check the successors:
    Current code which needs to change:
    {code}
    List<LogicalOperator> sucs = mPlan.getSuccessors(root);
    if(sucs==null) return;
    int size = sucs.size();
    if(size==0 || size==1) return; --> THIS should change to recursively process the succesor if it exists(size == 1 case)
    {code}
    * If there are more than one operators in the plan which need implicit splits to be introduced
    * If the new SPLIT operator is introduced before a cogroup, the cogroup's inner map for it's inputs and GroupBy expressions needs to be updated
    * The kind of changes need for cogroup maybe needed for some other operators
    I started looking at it some and have a patch for the first two issues - *UNTESTED* which I am attaching to this issue
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Alan Gates (JIRA) at Sep 5, 2008 at 10:51 pm
    [ https://issues.apache.org/jira/browse/PIG-401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Alan Gates updated PIG-401:
    ---------------------------

    Status: Patch Available (was: Open)
    SplitIntroducer does not handle some cases correctly
    ----------------------------------------------------

    Key: PIG-401
    URL: https://issues.apache.org/jira/browse/PIG-401
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Pradeep Kamath
    Assignee: Alan Gates
    Priority: Critical
    Fix For: types_branch

    Attachments: PIG-401-partial.patch, PIG401.patch


    The splitIntroducer does not handle the following cases correctly:
    * As soon as SplitIntroducer sees an operator which does NOT have multiple outputs it stops - it should still recursively check the successors:
    Current code which needs to change:
    {code}
    List<LogicalOperator> sucs = mPlan.getSuccessors(root);
    if(sucs==null) return;
    int size = sucs.size();
    if(size==0 || size==1) return; --> THIS should change to recursively process the succesor if it exists(size == 1 case)
    {code}
    * If there are more than one operators in the plan which need implicit splits to be introduced
    * If the new SPLIT operator is introduced before a cogroup, the cogroup's inner map for it's inputs and GroupBy expressions needs to be updated
    * The kind of changes need for cogroup maybe needed for some other operators
    I started looking at it some and have a patch for the first two issues - *UNTESTED* which I am attaching to this issue
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Alan Gates (JIRA) at Sep 5, 2008 at 10:51 pm
    [ https://issues.apache.org/jira/browse/PIG-401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Alan Gates updated PIG-401:
    ---------------------------

    Attachment: PIG401.patch

    This patch moves the SplitIntroducer into the optimizer (and renames it as ImplicitSplitInserter for whatever reason). This addresses two issues:

    1) catching splits past the initial node in the graph
    2) patching up schemas and inner plans of other operators after a split is introduced.
    SplitIntroducer does not handle some cases correctly
    ----------------------------------------------------

    Key: PIG-401
    URL: https://issues.apache.org/jira/browse/PIG-401
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Pradeep Kamath
    Assignee: Alan Gates
    Priority: Critical
    Fix For: types_branch

    Attachments: PIG-401-partial.patch, PIG401.patch


    The splitIntroducer does not handle the following cases correctly:
    * As soon as SplitIntroducer sees an operator which does NOT have multiple outputs it stops - it should still recursively check the successors:
    Current code which needs to change:
    {code}
    List<LogicalOperator> sucs = mPlan.getSuccessors(root);
    if(sucs==null) return;
    int size = sucs.size();
    if(size==0 || size==1) return; --> THIS should change to recursively process the succesor if it exists(size == 1 case)
    {code}
    * If there are more than one operators in the plan which need implicit splits to be introduced
    * If the new SPLIT operator is introduced before a cogroup, the cogroup's inner map for it's inputs and GroupBy expressions needs to be updated
    * The kind of changes need for cogroup maybe needed for some other operators
    I started looking at it some and have a patch for the first two issues - *UNTESTED* which I am attaching to this issue
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Alan Gates (JIRA) at Sep 5, 2008 at 11:35 pm
    [ https://issues.apache.org/jira/browse/PIG-401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Alan Gates updated PIG-401:
    ---------------------------

    Resolution: Fixed
    Status: Resolved (was: Patch Available)

    PIG401.patch checked in.
    SplitIntroducer does not handle some cases correctly
    ----------------------------------------------------

    Key: PIG-401
    URL: https://issues.apache.org/jira/browse/PIG-401
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Pradeep Kamath
    Assignee: Alan Gates
    Priority: Critical
    Fix For: types_branch

    Attachments: PIG-401-partial.patch, PIG401.patch


    The splitIntroducer does not handle the following cases correctly:
    * As soon as SplitIntroducer sees an operator which does NOT have multiple outputs it stops - it should still recursively check the successors:
    Current code which needs to change:
    {code}
    List<LogicalOperator> sucs = mPlan.getSuccessors(root);
    if(sucs==null) return;
    int size = sucs.size();
    if(size==0 || size==1) return; --> THIS should change to recursively process the succesor if it exists(size == 1 case)
    {code}
    * If there are more than one operators in the plan which need implicit splits to be introduced
    * If the new SPLIT operator is introduced before a cogroup, the cogroup's inner map for it's inputs and GroupBy expressions needs to be updated
    * The kind of changes need for cogroup maybe needed for some other operators
    I started looking at it some and have a patch for the first two issues - *UNTESTED* which I am attaching to this issue
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriespig, hadoop
postedAug 27, '08 at 11:02p
activeSep 5, '08 at 11:35p
posts7
users1
websitepig.apache.org

1 user in discussion

Alan Gates (JIRA): 7 posts

People

Translate

site design / logo © 2022 Grokbase