FAQ
Hey Folks,

I am seeing a very weird problem in FileSystem.get(Configuration).

I want to get a FileSystem given the configuration, so I am using

Configuration conf = new Configuration();

_fs = FileSystem.get(conf);


The problem is I am getting LocalFileSystem on some machines and Distributed
on others. I am printing conf.get("fs.default.name") at all places and
It returns the right HDFS value 'hdfs://dummy:9000'

My expectation is looking at fs.default.name if it is hdfs:// it should give
me a DistributedFileSystem always.

Best
Bhupesh

Search Discussions

  • Ashutosh Chauhan at Oct 15, 2009 at 8:47 pm
    Each node reads its own conf files (mapred-site.xml, hdfs-site.xml etc.)
    Make sure your configs are consistent on all nodes across entire cluster and
    are pointing to correct fs.

    Hope it helps,
    Ashutosh
    On Thu, Oct 15, 2009 at 16:36, Bhupesh Bansal wrote:

    Hey Folks,

    I am seeing a very weird problem in FileSystem.get(Configuration).

    I want to get a FileSystem given the configuration, so I am using

    Configuration conf = new Configuration();

    _fs = FileSystem.get(conf);


    The problem is I am getting LocalFileSystem on some machines and
    Distributed
    on others. I am printing conf.get("fs.default.name") at all places and
    It returns the right HDFS value 'hdfs://dummy:9000'

    My expectation is looking at fs.default.name if it is hdfs:// it should
    give
    me a DistributedFileSystem always.

    Best
    Bhupesh




  • Bhupesh Bansal at Oct 15, 2009 at 10:07 pm
    This code is not map/reduce code and run only on single machine and
    Also each node prints the right value for "fs.default.name" so it is
    Reading the right configuration file too .. The issue looks like use of
    CACHE in filesystem and someplace my code is setting up a wrong value
    If that is possible.

    Best
    Bhupesh



    On 10/15/09 1:46 PM, "Ashutosh Chauhan" wrote:

    Each node reads its own conf files (mapred-site.xml, hdfs-site.xml etc.)
    Make sure your configs are consistent on all nodes across entire cluster and
    are pointing to correct fs.

    Hope it helps,
    Ashutosh
    On Thu, Oct 15, 2009 at 16:36, Bhupesh Bansal wrote:

    Hey Folks,

    I am seeing a very weird problem in FileSystem.get(Configuration).

    I want to get a FileSystem given the configuration, so I am using

    Configuration conf = new Configuration();

    _fs = FileSystem.get(conf);


    The problem is I am getting LocalFileSystem on some machines and
    Distributed
    on others. I am printing conf.get("fs.default.name") at all places and
    It returns the right HDFS value 'hdfs://dummy:9000'

    My expectation is looking at fs.default.name if it is hdfs:// it should
    give
    me a DistributedFileSystem always.

    Best
    Bhupesh




  • Aaron Kimball at Oct 15, 2009 at 10:51 pm
    Bhupesh: If you use FileSystem.newInstance(), does that return the correct
    object type? This sidesteps CACHE.
    - A
    On Thu, Oct 15, 2009 at 3:07 PM, Bhupesh Bansal wrote:

    This code is not map/reduce code and run only on single machine and
    Also each node prints the right value for "fs.default.name" so it is
    Reading the right configuration file too .. The issue looks like use of
    CACHE in filesystem and someplace my code is setting up a wrong value
    If that is possible.

    Best
    Bhupesh



    On 10/15/09 1:46 PM, "Ashutosh Chauhan" wrote:

    Each node reads its own conf files (mapred-site.xml, hdfs-site.xml etc.)
    Make sure your configs are consistent on all nodes across entire cluster and
    are pointing to correct fs.

    Hope it helps,
    Ashutosh
    On Thu, Oct 15, 2009 at 16:36, Bhupesh Bansal wrote:

    Hey Folks,

    I am seeing a very weird problem in FileSystem.get(Configuration).

    I want to get a FileSystem given the configuration, so I am using

    Configuration conf = new Configuration();

    _fs = FileSystem.get(conf);


    The problem is I am getting LocalFileSystem on some machines and
    Distributed
    on others. I am printing conf.get("fs.default.name") at all places and
    It returns the right HDFS value 'hdfs://dummy:9000'

    My expectation is looking at fs.default.name if it is hdfs:// it should
    give
    me a DistributedFileSystem always.

    Best
    Bhupesh




Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedOct 15, '09 at 8:35p
activeOct 15, '09 at 10:51p
posts4
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase