FAQ
Can mapreduce job recursively browse through all files and select them for processing when higher level folder is set in FileInputPath?

For example,
Dir-1
___ Dir-2
____ Dir-3
If dir-1 is given in fileInput path, does it includes files from dir-2 and dir-3?

Regards
Jaydeep



DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.

Search Discussions

  • Harsh J at Nov 11, 2010 at 1:02 pm
    Hi,

    On Thu, Nov 11, 2010 at 4:25 PM, Jaydeep Ayachit
    wrote:
    Can mapreduce job recursively browse through all files and select them for processing when higher level folder is set in FileInputPath?

    For example,
    Dir-1
    ___      Dir-2
    ____   Dir-3
    If dir-1 is given in fileInput path, does it includes files from dir-2 and dir-3?
    Not directly, no. You need to implement the logic for this yourself,
    see what happens in FileInputFormat.listStatus method and override
    that functionality to recurse as you need it.

    In the next release, this will be given by FileInputFormat itself,
    controllable by a Configuration-settable property. See Zheng Shao's
    patch for that feature at MAPREDUCE-1501 :)

    --
    Harsh J
    www.harshj.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedNov 11, '10 at 10:55a
activeNov 11, '10 at 1:02p
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Jaydeep Ayachit: 1 post Harsh J: 1 post

People

Translate

site design / logo © 2022 Grokbase