[Hadoop-common-user] Restricting number of records from map output

Either use an instance variable or a Combiner. The latter is correct
if you want the top-n per key from the mapper.

Ideally, mappers should be independent of other mappers. Still, you can use
counters and start skipping records when counter>some value to achieve
similar behavior. It will not be very reliable if you want very exact
results though.

Hi Rakesh, What do you mean by the top N? The first ones or you need to
sort them in memory? You can always output records in the cleanup() method
at the end of the mapper run.

Hi,
I think I understand your goal. However the question is toward (what I
think) is the wrong solution.

A mapper gets 1 record as input and only knows about that one record.
There is no way to limit there.

If you implement a simple reducer you can very easily let is stop
reading the input iterator after N records and limit the output in
that way.

Doing it in the reducer also allows you to easily add a concept of
"Top N" by using the "Secondary Sort" trick to sort the input before
it arrives at the reducer.

HTH

Niels Basjes