We are working with a trunk of hive hive-0.6.0-957988.
Dynamic partitions are working for us in testing, we tested with about
100 dynamic partitions.
For our production run we have about 1000 (offer_id)'s.
HQL="set hive.exec.dynamic.partition.mode=nonstrict;
set hive.exec.max.dynamic.partition.pernode=200000;
set hive.exec.max.dynamic.partitions=20000;
insert overwrite table bco PARTITION (dt=20100721,offer)
SELECT * FROM (
SELECT browser_id, country, dma_code,
offer_id from baseline_raw where
gen_date=20100721 and blt='bco' DISTRIBUTE BY offer_id
) X;
"
2010-07-22 22:36:21,499 Stage-1 map = 100%, reduce = 50%[Fatal Error]
Operator FS_6 (id=6): Number of dynamic partitions exceeded
hive.exec.max.dynamic.partitions.pernode.. Killing the job.
As you can see we always die in the reducer with the same exception.
Is it possible that hive.exec.max.dynamic.partitions is not being used
in the reducer code?
Thanks,
Edward
