FAQ
Order by desc does not work with more than one reducer
------------------------------------------------------

Key: PIG-441
URL: https://issues.apache.org/jira/browse/PIG-441
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: types_branch
Reporter: Alan Gates
Assignee: Alan Gates
Fix For: types_branch


Order by with desc works if you use a single reducer. But with more than one reducer the partitions are not correctly constructed. The SortParitioner uses the object comparators, not the raw the comparators. The object comparators have no concept of order by descending. So they build the partitions for ascending order. The end result is data sorted descending within the partition but not across partitions.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Alan Gates (JIRA) at Sep 22, 2008 at 10:09 pm
    [ https://issues.apache.org/jira/browse/PIG-441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Alan Gates updated PIG-441:
    ---------------------------

    Attachment: PIG-441.patch

    This patch does a couple of things:

    1) add object comparators to the PigXRawComparator classes so that in places where hadoop uses the object instead of null comparators the desc behavior can still be handled.

    2) Change FindQuantiles UDF to take in its constructor an array that indicates ascending vs descending order so that the quantiles can be correclty computed.
    Order by desc does not work with more than one reducer
    ------------------------------------------------------

    Key: PIG-441
    URL: https://issues.apache.org/jira/browse/PIG-441
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: types_branch
    Reporter: Alan Gates
    Assignee: Alan Gates
    Fix For: types_branch

    Attachments: PIG-441.patch


    Order by with desc works if you use a single reducer. But with more than one reducer the partitions are not correctly constructed. The SortParitioner uses the object comparators, not the raw the comparators. The object comparators have no concept of order by descending. So they build the partitions for ascending order. The end result is data sorted descending within the partition but not across partitions.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Alan Gates (JIRA) at Sep 22, 2008 at 10:09 pm
    [ https://issues.apache.org/jira/browse/PIG-441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Alan Gates updated PIG-441:
    ---------------------------

    Status: Patch Available (was: Open)
    Order by desc does not work with more than one reducer
    ------------------------------------------------------

    Key: PIG-441
    URL: https://issues.apache.org/jira/browse/PIG-441
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: types_branch
    Reporter: Alan Gates
    Assignee: Alan Gates
    Fix For: types_branch

    Attachments: PIG-441.patch


    Order by with desc works if you use a single reducer. But with more than one reducer the partitions are not correctly constructed. The SortParitioner uses the object comparators, not the raw the comparators. The object comparators have no concept of order by descending. So they build the partitions for ascending order. The end result is data sorted descending within the partition but not across partitions.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Sep 22, 2008 at 10:33 pm
    [ https://issues.apache.org/jira/browse/PIG-441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12633542#action_12633542 ]

    Olga Natkovich commented on PIG-441:
    ------------------------------------

    +1
    Order by desc does not work with more than one reducer
    ------------------------------------------------------

    Key: PIG-441
    URL: https://issues.apache.org/jira/browse/PIG-441
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: types_branch
    Reporter: Alan Gates
    Assignee: Alan Gates
    Fix For: types_branch

    Attachments: PIG-441.patch


    Order by with desc works if you use a single reducer. But with more than one reducer the partitions are not correctly constructed. The SortParitioner uses the object comparators, not the raw the comparators. The object comparators have no concept of order by descending. So they build the partitions for ascending order. The end result is data sorted descending within the partition but not across partitions.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Alan Gates (JIRA) at Sep 23, 2008 at 12:51 am
    [ https://issues.apache.org/jira/browse/PIG-441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Alan Gates updated PIG-441:
    ---------------------------

    Resolution: Fixed
    Status: Resolved (was: Patch Available)

    Patch checked in.
    Order by desc does not work with more than one reducer
    ------------------------------------------------------

    Key: PIG-441
    URL: https://issues.apache.org/jira/browse/PIG-441
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: types_branch
    Reporter: Alan Gates
    Assignee: Alan Gates
    Fix For: types_branch

    Attachments: PIG-441.patch


    Order by with desc works if you use a single reducer. But with more than one reducer the partitions are not correctly constructed. The SortParitioner uses the object comparators, not the raw the comparators. The object comparators have no concept of order by descending. So they build the partitions for ascending order. The end result is data sorted descending within the partition but not across partitions.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriespig, hadoop
postedSep 19, '08 at 7:03p
activeSep 23, '08 at 12:51a
posts5
users1
websitepig.apache.org

1 user in discussion

Alan Gates (JIRA): 5 posts

People

Translate

site design / logo © 2022 Grokbase