FAQ
TypeCastInserter does not replace projects in inner plans correctly
-------------------------------------------------------------------

Key: PIG-428
URL: https://issues.apache.org/jira/browse/PIG-428
Project: Pig
Issue Type: Bug
Affects Versions: types_branch
Reporter: Pradeep Kamath
Fix For: types_branch


The TypeCastInserter tries to replace the Project's input operator in inner plans with the new foreach operator it adds. However it should replace only those Projects' input where the new Foreach has been added after the operator which was earlier the input to Project.

Here is a query which fails due to this:
{code}
a = load 'st10k' as (name:chararray,age:int, gpa:double);
another = load 'st10k';
c = foreach another generate $0, $1+ 10, $2 + 10;
d = join a by $0, c by $0;
dump d;

{code}

Here is the error:
{noformat}
2008-09-11 23:34:28,169 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) tip_200809051428_0045_m_000000java.io.IOException: Type mismatch in key from map: expected org.apache.pig.impl.io.NullableText, recieved org.apache.pig.impl.io.NullableBytesWritable
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:419)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:83)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:172)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:158)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:75)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)

{noformat}

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Pradeep Kamath (JIRA) at Sep 12, 2008 at 5:26 pm
    [ https://issues.apache.org/jira/browse/PIG-428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Pradeep Kamath updated PIG-428:
    -------------------------------

    Status: Patch Available (was: Open)

    Attached patch for the issue
    TypeCastInserter does not replace projects in inner plans correctly
    -------------------------------------------------------------------

    Key: PIG-428
    URL: https://issues.apache.org/jira/browse/PIG-428
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Pradeep Kamath
    Fix For: types_branch

    Attachments: PIG-428.patch


    The TypeCastInserter tries to replace the Project's input operator in inner plans with the new foreach operator it adds. However it should replace only those Projects' input where the new Foreach has been added after the operator which was earlier the input to Project.
    Here is a query which fails due to this:
    {code}
    a = load 'st10k' as (name:chararray,age:int, gpa:double);
    another = load 'st10k';
    c = foreach another generate $0, $1+ 10, $2 + 10;
    d = join a by $0, c by $0;
    dump d;
    {code}
    Here is the error:
    {noformat}
    2008-09-11 23:34:28,169 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) tip_200809051428_0045_m_000000java.io.IOException: Type mismatch in key from map: expected org.apache.pig.impl.io.NullableText, recieved org.apache.pig.impl.io.NullableBytesWritable
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:419)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:83)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:172)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:158)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:75)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219)
    at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
    {noformat}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Pradeep Kamath (JIRA) at Sep 12, 2008 at 5:26 pm
    [ https://issues.apache.org/jira/browse/PIG-428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Pradeep Kamath updated PIG-428:
    -------------------------------

    Attachment: PIG-428.patch
    TypeCastInserter does not replace projects in inner plans correctly
    -------------------------------------------------------------------

    Key: PIG-428
    URL: https://issues.apache.org/jira/browse/PIG-428
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Pradeep Kamath
    Fix For: types_branch

    Attachments: PIG-428.patch


    The TypeCastInserter tries to replace the Project's input operator in inner plans with the new foreach operator it adds. However it should replace only those Projects' input where the new Foreach has been added after the operator which was earlier the input to Project.
    Here is a query which fails due to this:
    {code}
    a = load 'st10k' as (name:chararray,age:int, gpa:double);
    another = load 'st10k';
    c = foreach another generate $0, $1+ 10, $2 + 10;
    d = join a by $0, c by $0;
    dump d;
    {code}
    Here is the error:
    {noformat}
    2008-09-11 23:34:28,169 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) tip_200809051428_0045_m_000000java.io.IOException: Type mismatch in key from map: expected org.apache.pig.impl.io.NullableText, recieved org.apache.pig.impl.io.NullableBytesWritable
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:419)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:83)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:172)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:158)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:75)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219)
    at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
    {noformat}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Sep 12, 2008 at 7:31 pm
    [ https://issues.apache.org/jira/browse/PIG-428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich updated PIG-428:
    -------------------------------

    Resolution: Fixed
    Status: Resolved (was: Patch Available)

    patch committed. thanks, pradeep
    TypeCastInserter does not replace projects in inner plans correctly
    -------------------------------------------------------------------

    Key: PIG-428
    URL: https://issues.apache.org/jira/browse/PIG-428
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Pradeep Kamath
    Fix For: types_branch

    Attachments: PIG-428.patch


    The TypeCastInserter tries to replace the Project's input operator in inner plans with the new foreach operator it adds. However it should replace only those Projects' input where the new Foreach has been added after the operator which was earlier the input to Project.
    Here is a query which fails due to this:
    {code}
    a = load 'st10k' as (name:chararray,age:int, gpa:double);
    another = load 'st10k';
    c = foreach another generate $0, $1+ 10, $2 + 10;
    d = join a by $0, c by $0;
    dump d;
    {code}
    Here is the error:
    {noformat}
    2008-09-11 23:34:28,169 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) tip_200809051428_0045_m_000000java.io.IOException: Type mismatch in key from map: expected org.apache.pig.impl.io.NullableText, recieved org.apache.pig.impl.io.NullableBytesWritable
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:419)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:83)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:172)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:158)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:75)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219)
    at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
    {noformat}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriespig, hadoop
postedSep 12, '08 at 6:42a
activeSep 12, '08 at 7:31p
posts4
users1
websitepig.apache.org

1 user in discussion

Olga Natkovich (JIRA): 4 posts

People

Translate

site design / logo © 2022 Grokbase