|| at Jan 19, 2011 at 5:01 pm
The wiki probably needs to be fixed :
For 32, buckets, I need to set the following flags.
set hive.merge.mapfiles = false;
... the set mapred.reduce.tasks ... is irrelevant.
The query mechanism should ideally set this automatically !!
On Wed, Jan 19, 2011 at 8:04 AM, Edward Capriolo wrote: On Wed, Jan 19, 2011 at 10:46 AM, Ajo Fod wrote:
I've 2 questions:
1) how to raise the number of reducers?
2) why are there only 2 bucket files per partition even though I
specified 32 buckets?
I've set the following and don't see an increase in the number of reducers.
Could this be because the jobs are too small?
I have a feeling that this is the cause for my having only 2 bucket
files in each partition, inspite of specifing 32 buckets.
I have never tried it you should use:
set hive.enforce.bucketing = true;
The number of reducers must equal the number of buckets. This is
described in the language manual.http://wiki.apache.org/hadoop/Hive/LanguageManual/DDL/BucketedTables