FAQ
Hi,

I'm using a custom grouping comparator class to simulate a secondary
sort on values, and I set it via Job.setGroupingComparatorClass (using
Hadoop 0.20.x) inside my driver. I'm wondering if this class is also
used when grouping the records in the combiner.

Using a combiner greatly improves the performance in my case, but for
the combiners, I want to use the default comparator, not the custom
one that I use before the actual reduce.

Is there a way to just set the custom grouping comparator for the
reduce and bypass it during the combine stage?


Thanks,
Jim

Search Discussions

  • Amogh Vasekar at Jan 27, 2010 at 7:14 pm
    Hi,
    I think combiner gets only the keys sort comparator, not the grouping comparator. So I believe the default grouping is used on combiner, but custom on reducer.
    Here's a relevant snipped of code :
    {
    super(inputCounter, conf, reporter);
    combinerClass = cls;
    keyClass = (Class<K>) job.getMapOutputKeyClass();
    valueClass = (Class<V>) job.getMapOutputValueClass();
    comparator = (RawComparator<K>) job.getOutputKeyComparator();
    }

    Amogh

    On 1/26/10 12:57 PM, "Jim Twensky" wrote:

    Hi,

    I'm using a custom grouping comparator class to simulate a secondary
    sort on values, and I set it via Job.setGroupingComparatorClass (using
    Hadoop 0.20.x) inside my driver. I'm wondering if this class is also
    used when grouping the records in the combiner.

    Using a combiner greatly improves the performance in my case, but for
    the combiners, I want to use the default comparator, not the custom
    one that I use before the actual reduce.

    Is there a way to just set the custom grouping comparator for the
    reduce and bypass it during the combine stage?


    Thanks,
    Jim

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJan 26, '10 at 7:27a
activeJan 27, '10 at 7:14p
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Amogh Vasekar: 1 post Jim Twensky: 1 post

People

Translate

site design / logo © 2022 Grokbase