[
https://issues.apache.org/jira/browse/PIG-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895410#action_12895410 ]
Thejas M Nair commented on PIG-1536:
------------------------------------
The way 'default union' deals with columns of different but compatible types in same position is not right. It creates a merged schema choosing a merged type, but there is not cast that happens to convert the rows to this type.
eg -
{code}
grunt> l1 = load '/tmp/f1' as (a : chararray, t (a : int, c : long) );
grunt> l2 = load '/tmp/f1' as (a : chararray, t (a : int, b : int) );
grunt> u = union l1, l2;
grunt> describe u;
u: {a: chararray,t: (a: int,c: long)}
-- the result of u, only the rows originating from l1 will correspond to schema shown in describe.
MapReduce node 1-206
Map Plan
u: Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-203
---u: Union[bag] - 1-202
---l1: New For Each(false,false)[bag] - 1-195
Cast[chararray] - 1-192
---Project[bytearray][0] - 1-191
Cast[tuple:(int,long)] - 1-194
---Project[bytearray][1] - 1-193
---l1: Load(/tmp/f1:org.apache.pig.builtin.PigStorage) - 1-190
---l2: New For Each(false,false)[bag] - 1-201
Cast[chararray] - 1-198
---Project[bytearray][0] - 1-197
Cast[tuple:(int,int)] - 1-200
---Project[bytearray][1] - 1-199
---l2: Load(/tmp/f1:org.apache.pig.builtin.PigStorage) - 1-196--------
Global sort: false
----------------
{code}
use same logic for merging inner schemas in "default union" and "union onschema"
--------------------------------------------------------------------------------
Key: PIG-1536
URL:
https://issues.apache.org/jira/browse/PIG-1536Project: Pig
Issue Type: Task
Reporter: Thejas M Nair
Fix For: 0.9.0
We should consider using logic for merging inner schema in case of the two different types of union.
In case of 'default union', it merges the two inner schema of bags/tuples by position if the number of fields are same and the corresponding types are compatible.
In case of 'union onschema', it considers tuple/bag with different innerschema to be incompatible types.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
