[Hadoop-common-user] is it possible to concatenate output files under many reducers?

hi, all.

I have 60 reducers which are generating same output files.

from output-r--00001 to output-r-00059.

under this situation, I want to control the count of output files.

for example, is it possible to concatenate all output files to 10 ?

from output-r-00001 to output-r-00010.

thanks

--
Junyoung Kim (juneng603@gmail.com)

On 05/12/2011 02:17 PM, Harsh J wrote:

Short, blind answer: You could run 10 reducers.

Otherwise, you'll have to run another job that picks up a few files
each in mapper and merges them out. But having 60 files shouldn't
really be a problem if they are sufficiently large (at least 80% of a
block size perhaps -- you can tune # of reducers to achieve this).

On Thu, May 12, 2011 at 6:14 AM, Jun Young Kim<juneng603@gmail.com>
wrote: