[Hive-user] Sorting data

There is a SORT BY.

You can do:
SELECT * FROM tableA SORT BY c1 DESC;

Then each of the partition will be sorted.

However in order to get the global 10, we will need to do LIMIT 10 on top of
that. LIMIT 10 and SORT BY do not work exactly as the user wants now.


Zheng

Ah okay, I guess I can simply just not do fetchAll() to grab the global ten
so I do not mistakenly grab too much data.

Suhail

Hey Zheng,

What is the timeline and priority for doing a total ordering for ORDER BY
support?

Thanks,
Jeff

Right now there is already a way to get total ordering. You can do a SORT BY
and specify one reducer.

raghu

Hi Jeff,

Besides the way of achieving total ordering as Raghu said (with 1 reducer),
we can also first get a partitioned ordering, and then merge the partitions
(preserving the order) when reading.
The reducer step can be much faster because it's parallelized, but the
reading is sequential so it will still take a long time to get all the data.



However most of the use cases of a total ordering is just to get the top 10.

The current work-around is:

First store the top 10 from each partition to some temp table:
INSERT OVERWRITE tableB
REDUCE a.*
USING 'head -n 10'
AS (col1, col2, col3, col4, ...)
FROM (SELECT * FROM tableA SORT BY col3 DESC, col4 ASC) a

Second, set the #reducer to 1 and get the top 10 globally.
set mapred.reduce.tasks=1;
SELECT * FROM tableB SORT BY col3 DESC, col4 ASC LIMIT 10


Zheng

Is there a way to sort by a function such as count(1)?

Suhail

Nevermind, count(1) as count works heh, thought it didn't.

Suhail