[MapReduce-user] What is the cost of using a counter

Hi Steve, One thing to keep in mind is that the counters in Hadoop are
passed via heartbeats, so you'll see updates only 2 seconds or so. I have
seen implementations with 1,000s of counters w/o a noticeable performance
impact (since they are passed via heartbeats and pre-aggregated in a
reducer the frequency does not matter).

Looking at the code all of the instances of Counter for the increment method just to a value += incr. They are not going to be much more expensive then keeping the counter yourself internally. What happens in that at periodic intervals the counters are then sent to the JT (pre YARN) or the ApplicationMaster (in YARN) where it keeps track of them. It should really have no impact on how often you increment them.

--Bobby Evans

On 12/8/11 12:48 PM, "Steve Lewis" wrote:

I have several counters that I maintain to allow me to keep statistics on critical operations. I have my code incrementing the counters in an inner loop
partly to make sure my job is not killed for not making progress. It would be very easy to keep an internal counter and increment the Hadoop value less frequently. Assuming
I am currently incrementing a counter several million times in a reduce task- is this costing me performance and would I be better off incrementing less frequently