FAQ
Hi,

Does anyone have a way to reduce InputSplit size in general ?

By default, the minimum size chunk that map input should be split into is
set to 0 (ie.mapred.min.split.size). Can I change dfs.block.size or some
other configuration to reduce the split size and spawn many mappers?

Thanks,
Mark

Search Discussions

  • Jagaran das at Jun 7, 2011 at 3:20 am
    Correct reduce the dfs.block.size to increase the number of mappers.

    - Jagaran



    ________________________________
    From: Mark question <markq2011@gmail.com>
    To: common-user <common-user@hadoop.apache.org>
    Sent: Mon, 6 June, 2011 7:31:17 PM
    Subject: Reducing Mapper InputSplit size

    Hi,

    Does anyone have a way to reduce InputSplit size in general ?

    By default, the minimum size chunk that map input should be split into is
    set to 0 (ie.mapred.min.split.size). Can I change dfs.block.size or some
    other configuration to reduce the split size and spawn many mappers?

    Thanks,
    Mark
  • Panayotis Antonopoulos at Jun 7, 2011 at 3:29 am
    Hi Mark,

    Check: http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.html

    I think that setMaxInputSplitSize(Job job,
    long size)


    will do what you need.

    Regards,
    P.A.
    Date: Mon, 6 Jun 2011 19:31:17 -0700
    Subject: Reducing Mapper InputSplit size
    From: markq2011@gmail.com
    To: common-user@hadoop.apache.org

    Hi,

    Does anyone have a way to reduce InputSplit size in general ?

    By default, the minimum size chunk that map input should be split into is
    set to 0 (ie.mapred.min.split.size). Can I change dfs.block.size or some
    other configuration to reduce the split size and spawn many mappers?

    Thanks,
    Mark
  • Mark question at Jun 7, 2011 at 5:09 am
    Great! Thanks guys :)
    Mark

    2011/6/6 Panayotis Antonopoulos <antonopoulospan@hotmail.com>
    Hi Mark,

    Check:
    http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.html

    I think that setMaxInputSplitSize(Job job,
    long size)


    will do what you need.

    Regards,
    P.A.
    Date: Mon, 6 Jun 2011 19:31:17 -0700
    Subject: Reducing Mapper InputSplit size
    From: markq2011@gmail.com
    To: common-user@hadoop.apache.org

    Hi,

    Does anyone have a way to reduce InputSplit size in general ?

    By default, the minimum size chunk that map input should be split into is
    set to 0 (ie.mapred.min.split.size). Can I change dfs.block.size or some
    other configuration to reduce the split size and spawn many mappers?

    Thanks,
    Mark

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJun 7, '11 at 2:31a
activeJun 7, '11 at 5:09a
posts4
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase