FAQ
Hi,

I have basic question on property "fs.default.name" is that I am not able to open NameNode URL when I set fs.default.name=file:/// .
If we define not use HDFS as our file system then how hadoop deals with File system of local?..
Please reply.

Mahesh Shinde | Systems Engineer
mahesh_shinde@persistent.co.in | Cell: +918308321501 | Tel: +91-20-30235194
Persistent Systems Ltd. | 20 Glorious Years | www.persistentsys.com<http://www.persistentsys.com/>


DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.

Search Discussions

  • Harsh J at Jul 23, 2011 at 9:39 pm
    Mahesh,

    The NameNode is a component of HDFS. It does not apply to other FSes
    Hadoop can use. You can browse file:/// (are you planning to use NFS,
    etc.?) with a regular file browser.

    Besides this, am not sure I even got your question. Why don't you try
    out how Hadoop deals with a local fs by running a few MR jobs with
    that config?

    On Fri, Jul 22, 2011 at 11:50 AM, Mahesh Shinde
    wrote:

    Hi,

    I have basic question on property "fs.default.name" is that I am not able to open NameNode URL when I set fs.default.name=file:/// .
    If we define not use HDFS as our file system then how hadoop deals with File system of local?..
    Please reply.

    Mahesh Shinde | Systems Engineer
    mahesh_shinde@persistent.co.in | Cell: +918308321501 | Tel: +91-20-30235194
    Persistent Systems Ltd. | 20 Glorious Years | www.persistentsys.com<http://www.persistentsys.com/>


    DISCLAIMER
    ==========
    This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.


    --
    Harsh J
  • Uma Maheswara Rao G 72686 at Jul 24, 2011 at 3:31 am
    Hi Mahesh,

    When starting the NN, it will throw exception with your provided configuration. please check the code snippet below where exactly validation will happen.

    in NameNode:
    public static InetSocketAddress getAddress(URI filesystemURI) {
    String authority = filesystemURI.getAuthority();
    if (authority == null) {
    throw new IllegalArgumentException(String.format(
    "Invalid URI for NameNode address (check %s): %s has no authority.",
    FileSystem.FS_DEFAULT_NAME_KEY, filesystemURI.toString()));
    }
    if (!FSConstants.HDFS_URI_SCHEME.equalsIgnoreCase(
    filesystemURI.getScheme())) {
    throw new IllegalArgumentException(String.format(
    "Invalid URI for NameNode address (check %s): %s is not of scheme '%s'.",
    FileSystem.FS_DEFAULT_NAME_KEY, filesystemURI.toString(),
    FSConstants.HDFS_URI_SCHEME));
    }

    Since NN is specific to HDFS, it will expect scheme as hdfs. Otherwise it will throw the exception.

    If you develop your own file system, then clients will have intelligence to connect to that file system, based on your provided configuration for fs.default.name.

    Coming to clients side, if you pass file:/// as fs uri, it will not try to connect to DFS because your passed fs is related to local. So, it will create LocalFileSystem instead of DistributedFilesystem.


    Regards,
    Uma



    Regards,
    Uma
    ******************************************************************************************
    This email and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained here in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this email in error, please notify the sender by phone or email immediately and delete it!
    *****************************************************************************************

    ----- Original Message -----
    From: Mahesh Shinde <Mahesh_Shinde@persistent.co.in>
    Date: Saturday, July 23, 2011 4:17 pm
    Subject: FW: Question about property "fs.default.name"
    To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>

    Hi,

    I have basic question on property "fs.default.name" is that I am
    not able to open NameNode URL when I set fs.default.name=file:/// .
    If we define not use HDFS as our file system then how hadoop deals
    with File system of local?..
    Please reply.

    Mahesh Shinde | Systems Engineer
    mahesh_shinde@persistent.co.in | Cell: +918308321501 | Tel: +91-20-30235194
    Persistent Systems Ltd. | 20 Glorious Years |
    www.persistentsys.com<http://www.persistentsys.com/>

    DISCLAIMER
    ==========
    This e-mail may contain privileged and confidential information
    which is the property of Persistent Systems Ltd. It is intended
    only for the use of the individual or entity to which it is
    addressed. If you are not the intended recipient, you are not
    authorized to read, retain, copy, print, distribute or use this
    message. If you have received this communication in error, please
    notify the sender and delete all copies of this message. Persistent
    Systems Ltd. does not accept any liability for virus infected mails.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJul 23, '11 at 10:47a
activeJul 24, '11 at 3:31a
posts3
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase