FAQ
I know mappers can take files in HDFS as input. I wonder whether they
can take local files as input.
Thanks.

Gerald

Search Discussions

  • Harsh J at Nov 4, 2010 at 2:04 pm
    Hi,
    On Thu, Nov 4, 2010 at 3:12 AM, Zhenhua Guo wrote:
    I know mappers can take files in HDFS as input. I wonder whether they
    can take local files as input.
    Thanks.

    Gerald
    It can :)

    Try with fs.default.name set to "file:///" to use the local filesystem
    (and not HDFS). Or in your MR job, give the path as file://<path>
    (assuming that this is available across all working nodes).

    --
    Harsh J
    www.harshj.com
  • Owen O'Malley at Nov 4, 2010 at 4:08 pm
    Note that if you are running on a multi-node cluster, the "local" file
    system needs to be NFS or some other distributed file system. If you have a
    non-small cluster (> 10 machines), NFS will be very very busy trying to keep
    up.

    -- Owen

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedNov 3, '10 at 9:42p
activeNov 4, '10 at 4:08p
posts3
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase