|| at Mar 26, 2009 at 9:38 pm
Besides the way of achieving total ordering as Raghu said (with 1 reducer),
we can also first get a partitioned ordering, and then merge the partitions
(preserving the order) when reading.
The reducer step can be much faster because it's parallelized, but the
reading is sequential so it will still take a long time to get all the data.
However most of the use cases of a total ordering is just to get the top 10.
The current work-around is:
First store the top 10 from each partition to some temp table:
INSERT OVERWRITE tableB
USING 'head -n 10'
AS (col1, col2, col3, col4, ...)
FROM (SELECT * FROM tableA SORT BY col3 DESC, col4 ASC) a
Second, set the #reducer to 1 and get the top 10 globally.
SELECT * FROM tableB SORT BY col3 DESC, col4 ASC LIMIT 10
On Thu, Mar 26, 2009 at 11:10 AM, Raghu Murthy wrote:
Right now there is already a way to get total ordering. You can do a SORT
and specify one reducer.
On 3/26/09 10:49 AM, "Jeff Hammerbacher" wrote:
What is the timeline and priority for doing a total ordering for ORDER BY
On Wed, Mar 25, 2009 at 9:02 PM, Suhail Doshi <[email protected]
Ah okay, I guess I can simply just not do fetchAll() to grab the global
I do not mistakenly grab too much data.
On Wed, Mar 25, 2009 at 6:43 PM, Zheng Shao wrote:
There is a SORT BY.
You can do:
SELECT * FROM tableA SORT BY c1 DESC;
Then each of the partition will be sorted.
However in order to get the global 10, we will need to do LIMIT 10 on
that. LIMIT 10 and SORT BY do not work exactly as the user wants now.
On Wed, Mar 25, 2009 at 3:23 PM, Suhail Doshi <
Since Hive does not have an ORDER BY...yet what is the solution for
the top 10 rows based on a field without having your client in thrift
getting too much data back? Seems like it is possible to actually get
much data but unfortunately you have to get all rows and sort by