Grokbase Groups Pig dev July 2010
FAQ
Failure planning nested FOREACH with DISTINCT, POLoad cannot be cast to POLocalRearrange
----------------------------------------------------------------------------------------

Key: PIG-1491
URL: https://issues.apache.org/jira/browse/PIG-1491
Project: Pig
Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Scott Carey


I have a failure that occurs during planning while using DISTINCT in a nested FOREACH.

Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizer.visitMROp(SecondaryKeyOptimizer.java:352)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:218)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:40)
at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Ashutosh Chauhan (JIRA) at Jul 9, 2010 at 9:49 pm
    [ https://issues.apache.org/jira/browse/PIG-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886906#action_12886906 ]

    Ashutosh Chauhan commented on PIG-1491:
    ---------------------------------------

    Scott,

    It will be useful if you can also paste the Pig script which produced this exception.
    Failure planning nested FOREACH with DISTINCT, POLoad cannot be cast to POLocalRearrange
    ----------------------------------------------------------------------------------------

    Key: PIG-1491
    URL: https://issues.apache.org/jira/browse/PIG-1491
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.7.0
    Reporter: Scott Carey

    I have a failure that occurs during planning while using DISTINCT in a nested FOREACH.
    Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizer.visitMROp(SecondaryKeyOptimizer.java:352)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:218)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:40)
    at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Scott Carey (JIRA) at Jul 9, 2010 at 9:53 pm
    [ https://issues.apache.org/jira/browse/PIG-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886908#action_12886908 ]

    Scott Carey commented on PIG-1491:
    ----------------------------------

    more readable:

    {code}
    DATA_G = COGROUP A by (a, b, c, d) OUTER, B by (a, b, c, d) OUTER;
    DATA = FOREACH DATA_G {
    a_items = DISTINCT A.x;
    b_items = DISTINCT B.x;
    GENERATE
    FLATTEN(group) as (a,b,c,d),
    SUM(A.m) as m, SUM(A.n) as n,
    COUNT(a_items) as a_item_count,
    (long)(SUM(B.u) + (double)0.5) as u,
    (long)(SUM(B.v) + (double)0.5) as v,
    COUNT(b_items) as b_item_count;
    }
    {code}
    Failure planning nested FOREACH with DISTINCT, POLoad cannot be cast to POLocalRearrange
    ----------------------------------------------------------------------------------------

    Key: PIG-1491
    URL: https://issues.apache.org/jira/browse/PIG-1491
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.7.0
    Reporter: Scott Carey

    I have a failure that occurs during planning while using DISTINCT in a nested FOREACH.
    Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizer.visitMROp(SecondaryKeyOptimizer.java:352)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:218)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:40)
    at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Scott Carey (JIRA) at Jul 9, 2010 at 9:56 pm
    [ https://issues.apache.org/jira/browse/PIG-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886907#action_12886907 ]

    Scott Carey commented on PIG-1491:
    ----------------------------------

    Full stack trace at the end of this comment.

    The pig script used is a couple hundred lines long. But the individual chunk that I can change to trigger the issue is the following:

    DATA_G = COGROUP A by (a, b, c, d) OUTER, B by (a, b, c, d) OUTER;
    DATA = FOREACH DATA_G {
    a_items = DISTINCT A.x;
    b_items = DISTINCT B.x;
    GENERATE
    FLATTEN(group) as (a,b,c,d),
    SUM(A.m) as m,
    SUM(A.n) as n,
    COUNT(a_items) as a_item_count,
    (long)(SUM(B.u) + (double)0.5) as u,
    (long)(SUM(B.v) + (double)0.5) as v,
    COUNT(b_items) as b_item_count;
    }

    Removing both of the DISTINCT temporary aliases and not generating those counts works fine. Adding either one of them causes it to fail.





    ERROR 2043: Unexpected error during execution.

    org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected error during execution.
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:318)
    at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1016)
    at org.apache.pig.PigServer.execute(PigServer.java:1009)
    at org.apache.pig.PigServer.access$100(PigServer.java:114)
    at org.apache.pig.PigServer$Graph.execute(PigServer.java:1261)
    at org.apache.pig.PigServer.executeBatch(PigServer.java:326)
    at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:110)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:167)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:139)
    at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
    at org.apache.pig.Main.main(Main.java:335)
    Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizer.visitMROp(SecondaryKeyOptimizer.java:352)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:218)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:40)
    at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
    at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
    at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
    at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
    at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
    at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
    at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
    at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:446)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:108)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:294)
    ... 10 more

    Failure planning nested FOREACH with DISTINCT, POLoad cannot be cast to POLocalRearrange
    ----------------------------------------------------------------------------------------

    Key: PIG-1491
    URL: https://issues.apache.org/jira/browse/PIG-1491
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.7.0
    Reporter: Scott Carey

    I have a failure that occurs during planning while using DISTINCT in a nested FOREACH.
    Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizer.visitMROp(SecondaryKeyOptimizer.java:352)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:218)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:40)
    at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Alan Gates (JIRA) at Sep 21, 2010 at 6:16 pm
    [ https://issues.apache.org/jira/browse/PIG-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Alan Gates updated PIG-1491:
    ----------------------------

    Fix Version/s: 0.9.0
    Failure planning nested FOREACH with DISTINCT, POLoad cannot be cast to POLocalRearrange
    ----------------------------------------------------------------------------------------

    Key: PIG-1491
    URL: https://issues.apache.org/jira/browse/PIG-1491
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.7.0
    Reporter: Scott Carey
    Fix For: 0.9.0


    I have a failure that occurs during planning while using DISTINCT in a nested FOREACH.
    Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizer.visitMROp(SecondaryKeyOptimizer.java:352)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:218)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:40)
    at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriespig, hadoop
postedJul 9, '10 at 9:40p
activeSep 21, '10 at 6:16p
posts5
users1
websitepig.apache.org

1 user in discussion

Alan Gates (JIRA): 5 posts

People

Translate

site design / logo © 2022 Grokbase