[Hadoop-common-user] Reducing Mapper InputSplit size

Correct reduce the dfs.block.size to increase the number of mappers.

- Jagaran



________________________________
From: Mark question <markq2011@gmail.com>
To: common-user <common-user@hadoop.apache.org>
Sent: Mon, 6 June, 2011 7:31:17 PM
Subject: Reducing Mapper InputSplit size

Hi,

Does anyone have a way to reduce InputSplit size in general ?

By default, the minimum size chunk that map input should be split into is
set to 0 (ie.mapred.min.split.size). Can I change dfs.block.size or some
other configuration to reduce the split size and spawn many mappers?

Thanks,
Mark

Hi Mark,

Check: http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.html

I think that setMaxInputSplitSize(Job job,
long size)


will do what you need.

Regards,
P.A.

Great! Thanks guys :)
Mark

2011/6/6 Panayotis Antonopoulos <antonopoulospan@hotmail.com>