FAQ
Hello,
I am trying to trim down my configuration files to avoid confusion.

If I were to setup something like this: 1 name node (192.168.0.1) , 1
secondary name node (192.168.0.2), and 10 clients. The name node and
secondary namenode will not serve as a data node.

The clients will access thru a shared NFS configuration, core-site.xml
The datanode will only have, dfs.data.dir in hdfs-site.xml
The name node will have, dfs.name.dir in hdfs-site.xml
I am really not sure what I must do for my secondary name node. What
should I place there?

The idea is I want to reduce my configuration files. Any advise?

Search Discussions

  • Allen Wittenauer at Oct 16, 2010 at 5:56 pm

    On Oct 16, 2010, at 5:54 AM, Mag Gam wrote:

    Hello,
    I am trying to trim down my configuration files to avoid confusion.

    If I were to setup something like this: 1 name node (192.168.0.1) , 1
    secondary name node (192.168.0.2), and 10 clients. The name node and
    secondary namenode will not serve as a data node.

    The clients will access thru a shared NFS configuration, core-site.xml
    The datanode will only have, dfs.data.dir in hdfs-site.xml
    The name node will have, dfs.name.dir in hdfs-site.xml
    I am really not sure what I must do for my secondary name node. What
    should I place there?

    The idea is I want to reduce my configuration files. Any advise?


    Short Story:

    You are fighting a losing battle. Throw in the towel now and plan on pushing huge balls of configs. Work on a strategy for clients that are out of sync.

    Longer Story:

    a) Any production Hadoop deployment should expect to have somewhere around 3-4 different sets of configuration files if you configure even some of the intermediate features. [Hello separate audit logs!]

    b) You should expect to have all the config files for all the different services. The devs are just now starting to separate what needs what so hopefully by 0.22 we won't have to have worry whether we have all of our bases covered. Remember: if it isn't defined, it will use a default and therefore still work...just not the way you intend.

    c) Don't use NFS for pushing configuration files unless it is an HA-NFS configuration, full/redundant connectivity to every switch, etc. The first time the NFS server falls out from underneath the grid, you're going to be in a world of hurt.



    We really need https://issues.apache.org/jira/browse/HADOOP-5670 to make this whole mess go away. Until it is nailed down which components need which config vars, it is unlikely to either a) happen or b) work in a way that is actually operable for any reasonable production deployment.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgeneral @
categorieshadoop
postedOct 16, '10 at 12:54p
activeOct 16, '10 at 5:56p
posts2
users2
websitehadoop.apache.org
irc#hadoop

2 users in discussion

Allen Wittenauer: 1 post Mag Gam: 1 post

People

Translate

site design / logo © 2022 Grokbase