[Hadoop-common-dev] Question about how Hadoop stores intermediate results

Chen,

Files are stored based on the reducer partitions, not exactly per-key.
The result is that there are far lesser files than you imagine there
ought to be. The keys are kept sorted inside the partitioned files and
thus you do not lose out on your key groups either.

See Partitioner, which is responsible for doing the partitioning of
your map outputs:
(http://hadoop.apache.org/common/docs/r0.20.2/mapred_tutorial.html#Partitioner)

There is only one file per-map. Actually two, an output file and an index file to quickly get the offset/length for a given reducer.

The index file is also cached in memory for performance.

Arun

Hi Arun and Harsh J

Thank you for your replies.

Yes, there will be two finally. But during the map running, there are more
than two.

The scenario I mentioned before will not occur with the Hadoop default
partitioner. If there is a partitioner lead to above problem. Is there any
security policy prevent this?

We all know that the unbalanced keys distribution can lead to the
differences of reduce tasks' execution time even in homogeneous environment.
It will be easier to rearrange unbalanced keys if each key occupies a file.

Thanks!

Chen

Irrespective of the partitioner used a single file stores all keys/values during a single iteration of each 'spill' after sorting records in the sort-buffer.

You could have multiple spills, but you have lots of keys/values in each spill - we never do file per record. You'd very quickly run out of inodes.

In very early days we had a file per reducer and that caused huge issues, never mind file per record.

Arun