[Pig-dev] [jira] Created: (PIG-421) error with complex nested plan

[ https://issues.apache.org/jira/browse/PIG-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629325#action_12629325 ]

Santhosh Srinivasan commented on PIG-421:
-----------------------------------------

With Pig-398, the issue with LOVisitor not walking all inner plans correctly was resolved. That has uncovered other issues. One of which is the following:

During type checking, schemas are computed for a collection of operators. Later, the optimizer, resets the schemas and calls the getSchema() method on the operators. Instead of returning the schemas computed by the type checker (which are now lost), the schemas computed by the individual operator are returned. In order to resolve this, we need to move the schema computation out of the type checker and into a common place (getSchema() method of the operators).

The query in the bug report generates the following stack trace:

{code}
384826 [main] ERROR org.apache.pig.tools.grunt.GruntParser - java.io.IOException: Unable to explain alias d [Can't overwrite cause]
at java.lang.Throwable.initCause(Throwable.java:320)
at org.apache.pig.impl.logicalLayer.validators.TypeCheckingVisitor.resolveLOProjectType(TypeCheckingVisitor.java:212)
at org.apache.pig.impl.logicalLayer.validators.TypeCheckingVisitor.checkInnerPlan(TypeCheckingVisitor.java:2298)
at org.apache.pig.impl.logicalLayer.validators.TypeCheckingVisitor.visit(TypeCheckingVisitor.java:1848)
at org.apache.pig.impl.logicalLayer.LOFilter.visit(LOFilter.java:102)
{code}

[ https://issues.apache.org/jira/browse/PIG-421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Santhosh Srinivasan updated PIG-421:
------------------------------------

Attachment: PIG-421.patch

The attached patch (PIG-421.patch) does the following:

1. Moved all the schema computation that existed within the type checker to the appropriate operators.
2. The getType() method for all expressions will now call getFieldSchema() and return the type appropriately
3. Renamed the resetSchema methods to unsetSchema which was an existing method in LogicalOperator

Unit tests that still fail are:

[junit] Running org.apache.pig.test.TestMRCompiler
[junit] Tests run: 16, Failures: 1, Errors: 0, Time elapsed: 0.34 sec
[junit] Test org.apache.pig.test.TestMRCompiler FAILED

[junit] Running org.apache.pig.test.TestSplitStore
[junit] Tests run: 9, Failures: 0, Errors: 2, Time elapsed: 216.145 sec
[junit] Test org.apache.pig.test.TestSplitStore FAILED

TestSplitStore has two test cases failing test6 and test7. These tests fail due to the issue reported in PIG-407.

[ https://issues.apache.org/jira/browse/PIG-421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Santhosh Srinivasan updated PIG-421:
------------------------------------

Attachment: PIG-421_1.patch

Attaching a new patch (PIG-421_1.patch). In addition to the previous patch, this patch fixes:

1. Fixes the 2 failing unit test cases. Removed the 2 gold files for the Logical to Physical translation
2. Fixed a bug in LOVisitor where LOSplit was visiting the LOSplitOutputs. The walker was also visiting LOSplitOutputs

All unit test cases pass.

[ https://issues.apache.org/jira/browse/PIG-421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Olga Natkovich resolved PIG-421.
--------------------------------

Resolution: Fixed

patch committed. thanks, santhosh!