FAQ
Illustrate for the Types branch
-------------------------------

Key: PIG-443
URL: https://issues.apache.org/jira/browse/PIG-443
Project: Pig
Issue Type: New Feature
Affects Versions: types_branch
Reporter: Shubham Chopra
Assignee: Shubham Chopra


This implementation of Illustrate uses visitors to traverse the plans.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Shubham Chopra (JIRA) at Sep 22, 2008 at 10:11 am
    [ https://issues.apache.org/jira/browse/PIG-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Shubham Chopra updated PIG-443:
    -------------------------------

    Attachment: pig_types_22_09.patch

    Illustrate for types branch with testcases. This also contains the local implementations of Cogroup and Split operators and testcases.
    Illustrate for the Types branch
    -------------------------------

    Key: PIG-443
    URL: https://issues.apache.org/jira/browse/PIG-443
    Project: Pig
    Issue Type: New Feature
    Affects Versions: types_branch
    Reporter: Shubham Chopra
    Assignee: Shubham Chopra
    Attachments: pig_types_22_09.patch


    This implementation of Illustrate uses visitors to traverse the plans.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Sep 22, 2008 at 9:35 pm
    [ https://issues.apache.org/jira/browse/PIG-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich updated PIG-443:
    -------------------------------

    Fix Version/s: types_branch
    Illustrate for the Types branch
    -------------------------------

    Key: PIG-443
    URL: https://issues.apache.org/jira/browse/PIG-443
    Project: Pig
    Issue Type: New Feature
    Affects Versions: types_branch
    Reporter: Shubham Chopra
    Assignee: Shubham Chopra
    Fix For: types_branch

    Attachments: pig_types_22_09.patch


    This implementation of Illustrate uses visitors to traverse the plans.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Pradeep Kamath (JIRA) at Sep 22, 2008 at 11:36 pm
    [ https://issues.apache.org/jira/browse/PIG-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12633562#action_12633562 ]

    Pradeep Kamath commented on PIG-443:
    ------------------------------------

    I haven't completely gone through the code - but initial comments:

    - since PigServer.getExamples(alias) works on the optimized plan it may have some changes to original operators expected by the user script - for example additional foreach for casting to schema types. moving of filters etc. (an explain before the illustrate would result in optimization of the plan, if there is no explain, then it may not be optimized)

    - currently I am getting NullPointerException after applying the patch to a fresh svn checkout and running the following script:
    a = load 'somefile';
    illustrate a;

    - Cosmetic: There are "tabs" instead of 4 spaces in some of the code

    - Has streaming been tested with illustrate?

    - Apache header is missing from new files

    - Class level comments for new classes would be helpful


    Illustrate for the Types branch
    -------------------------------

    Key: PIG-443
    URL: https://issues.apache.org/jira/browse/PIG-443
    Project: Pig
    Issue Type: New Feature
    Affects Versions: types_branch
    Reporter: Shubham Chopra
    Assignee: Shubham Chopra
    Fix For: types_branch

    Attachments: pig_types_22_09.patch


    This implementation of Illustrate uses visitors to traverse the plans.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shubham Chopra (JIRA) at Sep 23, 2008 at 7:53 am
    [ https://issues.apache.org/jira/browse/PIG-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Shubham Chopra updated PIG-443:
    -------------------------------

    Attachment: pig_types_23_09.patch

    I have modified the code to address these issues.

    * I now use the clonePlan method in PigServer to generate the logical plan. I already had a functional optimiser that puts in the necessary operators to make the plan functional (Cast and Split operators). Using clonePlan also addresses the optimisations done by explain since it starts with the initial script.

    * I checked with the latest checkout. You get a RuntimeException that says you need to specify a schema while loading files. I need the schema to be able to generate synthetic data, since without schema I have no idea how many columns exist in the data. Can you pls confirm if its a null pointer exception?

    * I haven't tested streaming with illustrate. I am just looking at supporting the basic operators right now. I was hoping to get illustrate into the beta drop.

    Illustrate for the Types branch
    -------------------------------

    Key: PIG-443
    URL: https://issues.apache.org/jira/browse/PIG-443
    Project: Pig
    Issue Type: New Feature
    Affects Versions: types_branch
    Reporter: Shubham Chopra
    Assignee: Shubham Chopra
    Fix For: types_branch

    Attachments: pig_types_22_09.patch, pig_types_23_09.patch


    This implementation of Illustrate uses visitors to traverse the plans.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Pradeep Kamath (JIRA) at Sep 23, 2008 at 11:02 pm
    [ https://issues.apache.org/jira/browse/PIG-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12633958#action_12633958 ]

    Pradeep Kamath commented on PIG-443:
    ------------------------------------

    - new patch is broken - does not contain the files under src/org/apache/pig/backend/local

    - The patch gives a Null POinter Exception when used with a hadoop cluster. I wonder if this is breaking backward compatibility with illustrate in 1.4 - is it worthwhile supporting reading data from dfs and using for illustrate. At the least, we need to give an error message if this is not supported.
    {noformat}
    java -cp pig.jar:<location of hadoopsite.xml> org.apache.pig.Main
    grunt> a = load 'st10k' as (name, age, gpa);
    grunt> illustrate a;
    2008-09-23 15:01:21,948 [main] ERROR org.apache.pig.pen.ExampleGenerator - Error reading data
    2008-09-23 15:01:21,949 [main] ERROR org.apache.pig.tools.grunt.GruntParser - java.lang.NullPointerException
    at org.apache.pig.pen.EquivalenceClasses.GetEquivalenceClasses(EquivalenceClasses.java:73)
    at org.apache.pig.pen.DerivedDataVisitor.visit(DerivedDataVisitor.java:242)
    at org.apache.pig.impl.logicalLayer.LOLoad.visit(LOLoad.java:145)
    at org.apache.pig.impl.logicalLayer.LOLoad.visit(LOLoad.java:34)
    at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
    at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
    at org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:98)
    at org.apache.pig.pen.LineageTrimmingVisitor.(ExampleGenerator.java:100)
    at org.apache.pig.PigServer.getExamples(PigServer.java:627)
    at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:278)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:183)
    at org.apache.pig.tools.grunt.GruntParser.parseContOnError(GruntParser.java:93)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:58)
    at org.apache.pig.Main.main(Main.java:282)

    2008-09-23 15:01:21,949 [main] ERROR org.apache.pig.tools.grunt.GruntParser -
    2008-09-23 15:01:21,949 [main] ERROR org.apache.pig.tools.grunt.GruntParser - java.lang.NullPointerException
    {noformat}

    - If schema is not supplied like you said, it gives an exception asking for a schema. Isn't this too restrictive? Is it possible to sample and generate data based on real user input without considering the schema?

    - In local mode, cogroup seems like cause an issue:
    {noformat}
    [/tmp/types]java -Xmx1024m -cp pig.jar org.apache.pig.Main
    2008-09-23 15:29:38,477 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
    2008-09-23 15:29:38,689 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=
    grunt> a = load '/home/pradeepk/work/pig/simpletest';
    grunt> b = load '/home/pradeepk/work/pig/simpletest';
    grunt> c = group a by $0, b by $0;
    grunt> illustrate c;
    2008-09-23 15:29:52,829 [main] ERROR org.apache.pig.pen.AugmentBaseDataVisitor - Error visiting Cogroup during Augmentation phase of Example Generator! null
    2008-09-23 15:29:52,833 [main] ERROR org.apache.pig.pen.ExampleGenerator - Visitor exception while creating example data Error visiting Cogroup during Augmentation phase of Example Generator! null
    java.lang.NullPointerException
    at org.apache.pig.pen.util.DisplayExamples.MakeArray(DisplayExamples.java:184)
    at org.apache.pig.pen.util.DisplayExamples.PrintTabular(DisplayExamples.java:86)
    at org.apache.pig.pen.util.DisplayExamples.PrintTabular(DisplayExamples.java:80)
    at org.apache.pig.pen.util.DisplayExamples.PrintTabular(DisplayExamples.java:69)
    at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:138)
    at org.apache.pig.PigServer.getExamples(PigServer.java:627)
    at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:278)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:183)
    at org.apache.pig.tools.grunt.GruntParser.parseContOnError(GruntParser.java:93)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:58)
    at org.apache.pig.Main.main(Main.java:282)
    java.lang.NullPointerException
    at org.apache.pig.pen.util.DisplayExamples.MakeArray(DisplayExamples.java:184)
    at org.apache.pig.pen.util.DisplayExamples.PrintTabular(DisplayExamples.java:86)
    at org.apache.pig.pen.util.DisplayExamples.PrintTabular(DisplayExamples.java:80)
    at org.apache.pig.pen.util.DisplayExamples.PrintTabular(DisplayExamples.java:69)
    at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:138)
    at org.apache.pig.PigServer.getExamples(PigServer.java:627)
    at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:278)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:183)
    at org.apache.pig.tools.grunt.GruntParser.parseContOnError(GruntParser.java:93)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:58)
    at org.apache.pig.Main.main(Main.java:282)
    ----------------------------------------------------
    c | group: bytearray | a: bag | b: bag |
    ----------------------------------------------------
    a | {(a, 12)} | {(a, 12)} |
    ----------------------------------------------------

    grunt>
    {noformat}
    Illustrate for the Types branch
    -------------------------------

    Key: PIG-443
    URL: https://issues.apache.org/jira/browse/PIG-443
    Project: Pig
    Issue Type: New Feature
    Affects Versions: types_branch
    Reporter: Shubham Chopra
    Assignee: Shubham Chopra
    Fix For: types_branch

    Attachments: pig_types_22_09.patch, pig_types_23_09.patch


    This implementation of Illustrate uses visitors to traverse the plans.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shubham Chopra (JIRA) at Sep 24, 2008 at 5:08 am
    [ https://issues.apache.org/jira/browse/PIG-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Shubham Chopra updated PIG-443:
    -------------------------------

    Attachment: pig_types_24_09.patch

    * I have fixed the connection issue. I think I had misunderstood the way PigContext works. Using a proper pigContext fixed the connection problem.

    * Deducing a schema from sample data is a bit of a problem. Consider the following case:
    a = load 'somefile' as (x, y, z);
    b = filter a by some-condition;
    c = foreach b generate x;
    illustrate c;

    If 'b' is an empty set, 'c' would also be an empty set. I wouldn't be able to depend on sample data alone to deduce the schemas. I might end up using the same logic as the getSchema methods to deduce the schema for 'c' in this case.

    Also, schemas are used only while generating synthetic data and displaying the final set of examples.

    * That issue with cogroup I guess is again because of a no-schema load. Like pointed out earlier, the augmentation phase where I try to generate synthetic data used schemas.

    I have done a modification in the code. I now check for schema while reading the data and throw an exception if I don't find one asking users to provide a schema.

    Illustrate for the Types branch
    -------------------------------

    Key: PIG-443
    URL: https://issues.apache.org/jira/browse/PIG-443
    Project: Pig
    Issue Type: New Feature
    Affects Versions: types_branch
    Reporter: Shubham Chopra
    Assignee: Shubham Chopra
    Fix For: types_branch

    Attachments: pig_types_22_09.patch, pig_types_23_09.patch, pig_types_24_09.patch


    This implementation of Illustrate uses visitors to traverse the plans.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Pradeep Kamath (JIRA) at Sep 25, 2008 at 12:28 am
    [ https://issues.apache.org/jira/browse/PIG-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634354#action_12634354 ]

    Pradeep Kamath commented on PIG-443:
    ------------------------------------

    - A couple of unit tests - TestLogicalPlanBuilder and TestLogToPhyCompiler fail with the patch.

    - The following script fails at the last illustrate.
    {code}
    a = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
    b = load '/user/pig/tests/data/singlefile/votertab10k' as (name, age, registration, contributions);
    c = filter a by age < 50;
    illustrate c;
    d = filter b by age < 50;
    illustrate d;
    e = cogroup c by (name, age), d by (name, age) ;
    illustrate e;
    f = foreach e generate flatten(c), flatten(d);
    illustrate f;
    g = group f by registration;
    illustrate g;
    h = foreach g generate group, SUM(f.d::contributions);
    illustrate h;
    i = order h by $1;
    illustrate i;

    {code}

    Error:
    {noformat}
    ...
    --------------------------------------
    h | group: bytearray | double |
    --------------------------------------
    libertarian | 1697.08 |
    republican | 1655.32 |
    --------------------------------------

    10118 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://wilbur20.labs.corp.sp1.yahoo.com:802013
    10143 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: wilbur20.labs.corp.sp1.yahoo.com:50020
    java.lang.NullPointerException
    at org.apache.pig.pen.LineageTrimmingVisitor.PruneBaseDataConstrainedCoverage(LineageTrimmingVisitor.java:290)
    at org.apache.pig.pen.LineageTrimmingVisitor.checkCompleteness(LineageTrimmingVisitor.java:458)
    at org.apache.pig.pen.LineageTrimmingVisitor.processOperator(LineageTrimmingVisitor.java:416)
    at org.apache.pig.pen.LineageTrimmingVisitor.visit(LineageTrimmingVisitor.java:251)
    at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:105)
    at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:40)
    at org.apache.pig.pen.util.PreOrderDepthFirstWalker.depthFirst(PreOrderDepthFirstWalker.java:69)
    at org.apache.pig.pen.util.PreOrderDepthFirstWalker.depthFirst(PreOrderDepthFirstWalker.java:71)
    at org.apache.pig.pen.util.PreOrderDepthFirstWalker.walk(PreOrderDepthFirstWalker.java:55)
    at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
    at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:108)
    at org.apache.pig.PigServer.getExamples(PigServer.java:627)
    at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:278)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:183)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:83)
    at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
    at org.apache.pig.Main.main(Main.java:306)

    {noformat}
    Illustrate for the Types branch
    -------------------------------

    Key: PIG-443
    URL: https://issues.apache.org/jira/browse/PIG-443
    Project: Pig
    Issue Type: New Feature
    Affects Versions: types_branch
    Reporter: Shubham Chopra
    Assignee: Shubham Chopra
    Fix For: types_branch

    Attachments: pig_types_22_09.patch, pig_types_23_09.patch, pig_types_24_09.patch


    This implementation of Illustrate uses visitors to traverse the plans.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Pradeep Kamath (JIRA) at Sep 25, 2008 at 12:30 am
    [ https://issues.apache.org/jira/browse/PIG-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634354#action_12634354 ]

    pkamath edited comment on PIG-443 at 9/24/08 5:29 PM:
    -------------------------------------------------------------

    - A couple of unit tests - TestLogicalPlanBuilder and TestLogToPhyCompiler fail with the patch.

    - The following script fails at the last illustrate.
    {code}
    a = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
    b = load '/user/pig/tests/data/singlefile/votertab10k' as (name, age, registration, contributions);
    c = filter a by age < 50;
    illustrate c;
    d = filter b by age < 50;
    illustrate d;
    e = cogroup c by (name, age), d by (name, age) ;
    illustrate e;
    f = foreach e generate flatten(c), flatten(d);
    illustrate f;
    g = group f by registration;
    illustrate g;
    h = foreach g generate group, SUM(f.d::contributions);
    illustrate h;
    i = order h by $1;
    illustrate i;

    {code}

    Error:
    {noformat}
    ...
    --------------------------------------
    h | group: bytearray | double |
    --------------------------------------
    libertarian | 1697.08 |
    republican | 1655.32 |
    --------------------------------------

    10118 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://clusterhost
    10143 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: clusterhost
    java.lang.NullPointerException
    at org.apache.pig.pen.LineageTrimmingVisitor.PruneBaseDataConstrainedCoverage(LineageTrimmingVisitor.java:290)
    at org.apache.pig.pen.LineageTrimmingVisitor.checkCompleteness(LineageTrimmingVisitor.java:458)
    at org.apache.pig.pen.LineageTrimmingVisitor.processOperator(LineageTrimmingVisitor.java:416)
    at org.apache.pig.pen.LineageTrimmingVisitor.visit(LineageTrimmingVisitor.java:251)
    at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:105)
    at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:40)
    at org.apache.pig.pen.util.PreOrderDepthFirstWalker.depthFirst(PreOrderDepthFirstWalker.java:69)
    at org.apache.pig.pen.util.PreOrderDepthFirstWalker.depthFirst(PreOrderDepthFirstWalker.java:71)
    at org.apache.pig.pen.util.PreOrderDepthFirstWalker.walk(PreOrderDepthFirstWalker.java:55)
    at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
    at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:108)
    at org.apache.pig.PigServer.getExamples(PigServer.java:627)
    at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:278)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:183)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:83)
    at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
    at org.apache.pig.Main.main(Main.java:306)

    {noformat}

    was (Author: pkamath):
    - A couple of unit tests - TestLogicalPlanBuilder and TestLogToPhyCompiler fail with the patch.

    - The following script fails at the last illustrate.
    {code}
    a = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
    b = load '/user/pig/tests/data/singlefile/votertab10k' as (name, age, registration, contributions);
    c = filter a by age < 50;
    illustrate c;
    d = filter b by age < 50;
    illustrate d;
    e = cogroup c by (name, age), d by (name, age) ;
    illustrate e;
    f = foreach e generate flatten(c), flatten(d);
    illustrate f;
    g = group f by registration;
    illustrate g;
    h = foreach g generate group, SUM(f.d::contributions);
    illustrate h;
    i = order h by $1;
    illustrate i;

    {code}

    Error:
    {noformat}
    ...
    --------------------------------------
    h | group: bytearray | double |
    --------------------------------------
    libertarian | 1697.08 |
    republican | 1655.32 |
    --------------------------------------

    10118 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://wilbur20.labs.corp.sp1.yahoo.com:802013
    10143 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: wilbur20.labs.corp.sp1.yahoo.com:50020
    java.lang.NullPointerException
    at org.apache.pig.pen.LineageTrimmingVisitor.PruneBaseDataConstrainedCoverage(LineageTrimmingVisitor.java:290)
    at org.apache.pig.pen.LineageTrimmingVisitor.checkCompleteness(LineageTrimmingVisitor.java:458)
    at org.apache.pig.pen.LineageTrimmingVisitor.processOperator(LineageTrimmingVisitor.java:416)
    at org.apache.pig.pen.LineageTrimmingVisitor.visit(LineageTrimmingVisitor.java:251)
    at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:105)
    at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:40)
    at org.apache.pig.pen.util.PreOrderDepthFirstWalker.depthFirst(PreOrderDepthFirstWalker.java:69)
    at org.apache.pig.pen.util.PreOrderDepthFirstWalker.depthFirst(PreOrderDepthFirstWalker.java:71)
    at org.apache.pig.pen.util.PreOrderDepthFirstWalker.walk(PreOrderDepthFirstWalker.java:55)
    at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
    at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:108)
    at org.apache.pig.PigServer.getExamples(PigServer.java:627)
    at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:278)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:183)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:83)
    at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
    at org.apache.pig.Main.main(Main.java:306)

    {noformat}
    Illustrate for the Types branch
    -------------------------------

    Key: PIG-443
    URL: https://issues.apache.org/jira/browse/PIG-443
    Project: Pig
    Issue Type: New Feature
    Affects Versions: types_branch
    Reporter: Shubham Chopra
    Assignee: Shubham Chopra
    Fix For: types_branch

    Attachments: pig_types_22_09.patch, pig_types_23_09.patch, pig_types_24_09.patch


    This implementation of Illustrate uses visitors to traverse the plans.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shubham Chopra (JIRA) at Sep 25, 2008 at 11:46 am
    [ https://issues.apache.org/jira/browse/PIG-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Shubham Chopra updated PIG-443:
    -------------------------------

    Attachment: pig_types_25_09.patch

    * I had put in a toString() method in LogicalPlan, that used explain. I was using it for debugging and have removed it now. But, shouldn't the explain just give out the plan instead of modifying it in any way?

    * I wasn't supporting Orderby in the sense that the equivalence class model doesn't hold good for OrderBy. So the output that you would get from it might not make sense (You would get just a single tuple in the output in the worst case).
    Illustrate for the Types branch
    -------------------------------

    Key: PIG-443
    URL: https://issues.apache.org/jira/browse/PIG-443
    Project: Pig
    Issue Type: New Feature
    Affects Versions: types_branch
    Reporter: Shubham Chopra
    Assignee: Shubham Chopra
    Fix For: types_branch

    Attachments: pig_types_22_09.patch, pig_types_23_09.patch, pig_types_24_09.patch, pig_types_25_09.patch


    This implementation of Illustrate uses visitors to traverse the plans.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Pradeep Kamath (JIRA) at Sep 25, 2008 at 9:26 pm
    [ https://issues.apache.org/jira/browse/PIG-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Pradeep Kamath updated PIG-443:
    -------------------------------

    Attachment: pig_types_25_09-v2.patch

    - I mostly tested the new patch and glanced a little over the code. The TestLocalPOSplit unit test case was failing - I checked the code and found the minor issue in it - it assumes an order for the leaf operators which may not always be true since the plan consists of HashMaps which do not guarantee order. I modified the test case to test without assuming any order in the leaf operators - please take a look at the diff between your previous patch and my attachment.

    All unit test cases pass, I also tested with the script above which used to cause a failure and it no longer does.
    Illustrate for the Types branch
    -------------------------------

    Key: PIG-443
    URL: https://issues.apache.org/jira/browse/PIG-443
    Project: Pig
    Issue Type: New Feature
    Affects Versions: types_branch
    Reporter: Shubham Chopra
    Assignee: Shubham Chopra
    Fix For: types_branch

    Attachments: pig_types_22_09.patch, pig_types_23_09.patch, pig_types_24_09.patch, pig_types_25_09-v2.patch, pig_types_25_09.patch


    This implementation of Illustrate uses visitors to traverse the plans.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shubham Chopra (JIRA) at Sep 26, 2008 at 2:10 pm
    [ https://issues.apache.org/jira/browse/PIG-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634874#action_12634874 ]

    Shubham Chopra commented on PIG-443:
    ------------------------------------

    Yes, I think I had assumed the order of Splits assuming that they are in some kind of ordered list since getLeaves() returns a linked list. Thanks a lot for correcting it. I have reviewed it and it looks good.
    Illustrate for the Types branch
    -------------------------------

    Key: PIG-443
    URL: https://issues.apache.org/jira/browse/PIG-443
    Project: Pig
    Issue Type: New Feature
    Affects Versions: types_branch
    Reporter: Shubham Chopra
    Assignee: Shubham Chopra
    Fix For: types_branch

    Attachments: pig_types_22_09.patch, pig_types_23_09.patch, pig_types_24_09.patch, pig_types_25_09-v2.patch, pig_types_25_09.patch


    This implementation of Illustrate uses visitors to traverse the plans.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Sep 26, 2008 at 9:22 pm
    [ https://issues.apache.org/jira/browse/PIG-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich resolved PIG-443.
    --------------------------------

    Resolution: Fixed

    patch committed. Thanks Shubham and Pradeep!
    Illustrate for the Types branch
    -------------------------------

    Key: PIG-443
    URL: https://issues.apache.org/jira/browse/PIG-443
    Project: Pig
    Issue Type: New Feature
    Affects Versions: types_branch
    Reporter: Shubham Chopra
    Assignee: Shubham Chopra
    Fix For: types_branch

    Attachments: pig_types_22_09.patch, pig_types_23_09.patch, pig_types_24_09.patch, pig_types_25_09-v2.patch, pig_types_25_09.patch


    This implementation of Illustrate uses visitors to traverse the plans.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriespig, hadoop
postedSep 22, '08 at 10:09a
activeSep 26, '08 at 9:22p
posts13
users1
websitepig.apache.org

1 user in discussion

Olga Natkovich (JIRA): 13 posts

People

Translate

site design / logo © 2022 Grokbase