Grokbase Groups Hive user July 2009
FAQ
Hi,

I have a table with 15,000,000+ rows sitting on a 4-node hadoop cluster with
dfs.replication=4. Hive seems to ignoring my settings for
mapred.reduce.tasks & hive.exec.reducers.max. Given below is a snippet of
what I'm trying. What am I doing wrong?

hive> set mapred.reduce.tasks=17;
hive> set hive.exec.reducers.max=17;
hive> select count(1) from hits;
Total MapReduce jobs = 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>

Saurabh.

Search Discussions

  • Namit Jain at Jul 15, 2009 at 5:18 am
    For a query with no grouping. hive always uses 1 reducer.
    Since map-side aggregation will reduce the number of rows, it is not a problem



    On 7/14/09 10:11 PM, "Saurabh Nanda" wrote:

    Hi,

    I have a table with 15,000,000+ rows sitting on a 4-node hadoop cluster with dfs.replication=4. Hive seems to ignoring my settings for mapred.reduce.tasks & hive.exec.reducers.max. Given below is a snippet of what I'm trying. What am I doing wrong?

    hive> set mapred.reduce.tasks=17;
    hive> set hive.exec.reducers.max=17;
    hive> select count(1) from hits;
    Total MapReduce jobs = 1
    Number of reduce tasks determined at compile time: 1
    In order to change the average load for a reducer (in bytes):
    set hive.exec.reducers.bytes.per.reducer=<number>
    In order to limit the maximum number of reducers:
    set hive.exec.reducers.max=<number>
    In order to set a constant number of reducers:
    set mapred.reduce.tasks=<number>

    Saurabh.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedJul 15, '09 at 5:12a
activeJul 15, '09 at 5:18a
posts2
users2
websitehive.apache.org

2 users in discussion

Namit Jain: 1 post Saurabh Nanda: 1 post

People

Translate

site design / logo © 2022 Grokbase