FAQ
I know the master node is responsible for namenode and job tracker, but
other than that is there any data stored on that machine? Basically what
I am asking is should there be an generous amount of free space on that
machine?

So for example I have a large drive I want to swap out of my master and
put into another machine which will be a used as a node. Before doing
this, is there anything I should back up from my master? Where is the
namenode data stored?

Thanks

Search Discussions

  • Harsh J at Mar 16, 2011 at 5:46 pm
    NameNode and JobTracker do not require a lot of storage space by
    themselves. The NameNode needs some space to store its edits and
    fsimage, and both require logging space.

    However, you may make use of multiple disks for NameNode, in order to
    have a redundant backup copy of the NN image available in case one of
    the disks crash. Other solutions to this second/third location include
    storing to an HA-NFS mount, or an externally attached disk mount.

    NameNode data is stored in the ${dfs.name.dir} set of directories
    [Defined in hdfs-site.xml]. Ensure its content is preserved perfectly
    (including permissions) if you're planning to switch disks for your
    NN.
    On Wed, Mar 16, 2011 at 10:04 PM, Mark wrote:
    I know the master node is responsible for namenode and job tracker, but
    other than that is there any data stored on that machine? Basically what I
    am asking is should there be an generous amount of free space on that
    machine?

    So for example I have a large drive I want to swap out of my master and put
    into another machine which will be a used as a node. Before doing this, is
    there anything I should back up from my master? Where is the namenode data
    stored?

    Thanks


    --
    Harsh J
    http://harshj.com
  • Mark at Mar 16, 2011 at 6:11 pm
    Ok thanks for the clarification.

    Just to be sure though..

    - The master will have the ${dfs.name.dir} but not ${dfs.data.dir}
    - The nodes will have ${dfs.data.dir} but not ${dfs.name.dir}

    Is that correct?
    On 3/16/11 10:43 AM, Harsh J wrote:
    NameNode and JobTracker do not require a lot of storage space by
    themselves. The NameNode needs some space to store its edits and
    fsimage, and both require logging space.

    However, you may make use of multiple disks for NameNode, in order to
    have a redundant backup copy of the NN image available in case one of
    the disks crash. Other solutions to this second/third location include
    storing to an HA-NFS mount, or an externally attached disk mount.

    NameNode data is stored in the ${dfs.name.dir} set of directories
    [Defined in hdfs-site.xml]. Ensure its content is preserved perfectly
    (including permissions) if you're planning to switch disks for your
    NN.

    On Wed, Mar 16, 2011 at 10:04 PM, Markwrote:
    I know the master node is responsible for namenode and job tracker, but
    other than that is there any data stored on that machine? Basically what I
    am asking is should there be an generous amount of free space on that
    machine?

    So for example I have a large drive I want to swap out of my master and put
    into another machine which will be a used as a node. Before doing this, is
    there anything I should back up from my master? Where is the namenode data
    stored?

    Thanks
  • Harsh J at Mar 16, 2011 at 6:29 pm
    Yes, ${dfs.name.dir} is a NameNode used prop, while the other's a
    DataNode used prop.
    On Wed, Mar 16, 2011 at 11:41 PM, Mark wrote:
    Ok thanks for the clarification.

    Just to be sure though..

    - The master will have the ${dfs.name.dir} but not ${dfs.data.dir}
    - The nodes will have ${dfs.data.dir} but not ${dfs.name.dir}

    Is that correct?
    On 3/16/11 10:43 AM, Harsh J wrote:

    NameNode and JobTracker do not require a lot of storage space by
    themselves. The NameNode needs some space to store its edits and
    fsimage, and both require logging space.

    However, you may make use of multiple disks for NameNode, in order to
    have a redundant backup copy of the NN image available in case one of
    the disks crash. Other solutions to this second/third location include
    storing to an HA-NFS mount, or an externally attached disk mount.

    NameNode data is stored in the ${dfs.name.dir} set of directories
    [Defined in hdfs-site.xml]. Ensure its content is preserved perfectly
    (including permissions) if you're planning to switch disks for your
    NN.

    On Wed, Mar 16, 2011 at 10:04 PM, Markwrote:
    I know the master node is responsible for namenode and job tracker, but
    other than that is there any data stored on that machine? Basically what
    I
    am asking is should there be an generous amount of free space on that
    machine?

    So for example I have a large drive I want to swap out of my master and
    put
    into another machine which will be a used as a node. Before doing this,
    is
    there anything I should back up from my master? Where is the namenode
    data
    stored?

    Thanks


    --
    Harsh J
    http://harshj.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMar 16, '11 at 4:34p
activeMar 16, '11 at 6:29p
posts4
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Mark: 2 posts Harsh J: 2 posts

People

Translate

site design / logo © 2022 Grokbase