FAQ
group by followed by group ALL causes error in reduce
-----------------------------------------------------

Key: PIG-454
URL: https://issues.apache.org/jira/browse/PIG-454
Project: Pig
Issue Type: Bug
Affects Versions: types_branch
Reporter: Pradeep Kamath
Fix For: types_branch


Script:
{code}
a = load 'st10k' as (name, age, gpa);
b = group a by name;
c = foreach b generate flatten(group), COUNT(a) as cnt;
d = group c all;
e = foreach d generate AVG(c.cnt);
dump e;
{code}

Error:
{noformat}
2008-09-23 17:58:12,002 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Job failed!
2008-09-23 17:58:12,004 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) tip_200809051428_0117_m_000000java.io.IOException: wrong key class: org.apache.pig.impl.io.NullableTuple is not class org.apache.pig.impl.io.NullableText
at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:995)
at org.apache.hadoop.mapred.MapTask$CombineOutputCollector.collect(MapTask.java:1079)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:155)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:56)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask.java:872)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:779)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:691)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)

...
{noformat}

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Olga Natkovich (JIRA) at Sep 24, 2008 at 1:10 am
    [ https://issues.apache.org/jira/browse/PIG-454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich reassigned PIG-454:
    ----------------------------------

    Assignee: Alan Gates
    group by followed by group ALL causes error in reduce
    -----------------------------------------------------

    Key: PIG-454
    URL: https://issues.apache.org/jira/browse/PIG-454
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Pradeep Kamath
    Assignee: Alan Gates
    Fix For: types_branch


    Script:
    {code}
    a = load 'st10k' as (name, age, gpa);
    b = group a by name;
    c = foreach b generate flatten(group), COUNT(a) as cnt;
    d = group c all;
    e = foreach d generate AVG(c.cnt);
    dump e;
    {code}
    Error:
    {noformat}
    2008-09-23 17:58:12,002 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Job failed!
    2008-09-23 17:58:12,004 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) tip_200809051428_0117_m_000000java.io.IOException: wrong key class: org.apache.pig.impl.io.NullableTuple is not class org.apache.pig.impl.io.NullableText
    at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:995)
    at org.apache.hadoop.mapred.MapTask$CombineOutputCollector.collect(MapTask.java:1079)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:155)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:56)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask.java:872)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:779)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:691)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
    at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
    ...
    {noformat}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Alan Gates (JIRA) at Sep 25, 2008 at 12:20 am
    [ https://issues.apache.org/jira/browse/PIG-454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Alan Gates updated PIG-454:
    ---------------------------

    Attachment: PIG-454.patch

    CombinerOptimizer is a visitor that walks the entire plan of MapReduceOpers. It was not resetting state as it visited each operator, causing it to get confused on the key to set in the combiner in cases where there were multiple ops that could use the combiner.
    group by followed by group ALL causes error in reduce
    -----------------------------------------------------

    Key: PIG-454
    URL: https://issues.apache.org/jira/browse/PIG-454
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Pradeep Kamath
    Assignee: Alan Gates
    Fix For: types_branch

    Attachments: PIG-454.patch


    Script:
    {code}
    a = load 'st10k' as (name, age, gpa);
    b = group a by name;
    c = foreach b generate flatten(group), COUNT(a) as cnt;
    d = group c all;
    e = foreach d generate AVG(c.cnt);
    dump e;
    {code}
    Error:
    {noformat}
    2008-09-23 17:58:12,002 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Job failed!
    2008-09-23 17:58:12,004 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) tip_200809051428_0117_m_000000java.io.IOException: wrong key class: org.apache.pig.impl.io.NullableTuple is not class org.apache.pig.impl.io.NullableText
    at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:995)
    at org.apache.hadoop.mapred.MapTask$CombineOutputCollector.collect(MapTask.java:1079)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:155)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:56)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask.java:872)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:779)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:691)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
    at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
    ...
    {noformat}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Alan Gates (JIRA) at Sep 25, 2008 at 12:20 am
    [ https://issues.apache.org/jira/browse/PIG-454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Alan Gates updated PIG-454:
    ---------------------------

    Status: Patch Available (was: Open)
    group by followed by group ALL causes error in reduce
    -----------------------------------------------------

    Key: PIG-454
    URL: https://issues.apache.org/jira/browse/PIG-454
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Pradeep Kamath
    Assignee: Alan Gates
    Fix For: types_branch

    Attachments: PIG-454.patch


    Script:
    {code}
    a = load 'st10k' as (name, age, gpa);
    b = group a by name;
    c = foreach b generate flatten(group), COUNT(a) as cnt;
    d = group c all;
    e = foreach d generate AVG(c.cnt);
    dump e;
    {code}
    Error:
    {noformat}
    2008-09-23 17:58:12,002 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Job failed!
    2008-09-23 17:58:12,004 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) tip_200809051428_0117_m_000000java.io.IOException: wrong key class: org.apache.pig.impl.io.NullableTuple is not class org.apache.pig.impl.io.NullableText
    at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:995)
    at org.apache.hadoop.mapred.MapTask$CombineOutputCollector.collect(MapTask.java:1079)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:155)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:56)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask.java:872)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:779)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:691)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
    at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
    ...
    {noformat}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Sep 25, 2008 at 12:28 am
    [ https://issues.apache.org/jira/browse/PIG-454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634353#action_12634353 ]

    Olga Natkovich commented on PIG-454:
    ------------------------------------

    +1
    group by followed by group ALL causes error in reduce
    -----------------------------------------------------

    Key: PIG-454
    URL: https://issues.apache.org/jira/browse/PIG-454
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Pradeep Kamath
    Assignee: Alan Gates
    Fix For: types_branch

    Attachments: PIG-454.patch


    Script:
    {code}
    a = load 'st10k' as (name, age, gpa);
    b = group a by name;
    c = foreach b generate flatten(group), COUNT(a) as cnt;
    d = group c all;
    e = foreach d generate AVG(c.cnt);
    dump e;
    {code}
    Error:
    {noformat}
    2008-09-23 17:58:12,002 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Job failed!
    2008-09-23 17:58:12,004 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) tip_200809051428_0117_m_000000java.io.IOException: wrong key class: org.apache.pig.impl.io.NullableTuple is not class org.apache.pig.impl.io.NullableText
    at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:995)
    at org.apache.hadoop.mapred.MapTask$CombineOutputCollector.collect(MapTask.java:1079)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:155)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:56)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask.java:872)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:779)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:691)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
    at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
    ...
    {noformat}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Sep 25, 2008 at 4:22 pm
    [ https://issues.apache.org/jira/browse/PIG-454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich updated PIG-454:
    -------------------------------

    Resolution: Fixed
    Status: Resolved (was: Patch Available)

    patch is in
    group by followed by group ALL causes error in reduce
    -----------------------------------------------------

    Key: PIG-454
    URL: https://issues.apache.org/jira/browse/PIG-454
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Pradeep Kamath
    Assignee: Alan Gates
    Fix For: types_branch

    Attachments: PIG-454.patch


    Script:
    {code}
    a = load 'st10k' as (name, age, gpa);
    b = group a by name;
    c = foreach b generate flatten(group), COUNT(a) as cnt;
    d = group c all;
    e = foreach d generate AVG(c.cnt);
    dump e;
    {code}
    Error:
    {noformat}
    2008-09-23 17:58:12,002 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Job failed!
    2008-09-23 17:58:12,004 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) tip_200809051428_0117_m_000000java.io.IOException: wrong key class: org.apache.pig.impl.io.NullableTuple is not class org.apache.pig.impl.io.NullableText
    at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:995)
    at org.apache.hadoop.mapred.MapTask$CombineOutputCollector.collect(MapTask.java:1079)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:155)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:56)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask.java:872)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:779)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:691)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
    at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
    ...
    {noformat}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriespig, hadoop
postedSep 24, '08 at 1:02a
activeSep 25, '08 at 4:22p
posts6
users1
websitepig.apache.org

1 user in discussion

Olga Natkovich (JIRA): 6 posts

People

Translate

site design / logo © 2022 Grokbase