FAQ
hello,buddys, i made two mistakes:
1. i run a datanode in the same machine with namenode,i think this is
not awful. (I do not have many manchines)
2. in the namenode,i misconfigured the "dfs.data.dir","dfs.name.dir"
in a disk volume which only have 1 g space. So today i got the
namenode compaining " low on available disk space. Already in safe
mode."
So, i want to change the "dfs.data.dir" for the namenode to other
volume, can i do this by create another dir and copy all the stuff in
dfs.data.dir to the new dir and then change the hdfs-site.xml
correspondingly and then restart those daemons in namenode? or ,do i
need to re format the namenode?
or , i just need to create a new dir for "dfs.data.dir" to make room
for dfs.name.dir and change the hdfs-site.xml correspondingly ,eg.
without coping the stuff and without reformating the namenode,the hdfs
will detect this change and transfer some blocks to the new
"dfs.data.dir" of this node
i do not want to change the "dfs.name.dir " since i think 1 g for
fsimage is enough for my testing cluster,and changing "dfs.name.dir "
must entail reformat namenode.

Search Discussions

  • Harsh J at Mar 21, 2012 at 11:38 am
    Hey toughman,
    On Wed, Mar 21, 2012 at 1:08 PM, XIANGJUN wrote:
    So, i want to change the "dfs.data.dir" for the namenode to other
    volume, can i do this by create another dir and copy all the stuff in
    dfs.data.dir to the new dir and then change the hdfs-site.xml
    correspondingly and then restart those daemons in namenode? or ,do i
    need to re format the name node?
    Your approach is almost right:

    Make new base directory for dfs.data.dir that is on separate disk.
    Stop datanode, reconfigure hdfs-site.xml to point to new directory or
    dfs.data.dir
    Copy (not move yet) old data.dir contents to new location.
    Start datanode.
    If datanode starts and reports blocks properly and fsck reports no
    problems (and you can read files/etc.), then delete old data directory
    copy safely.

    Never format your NameNode unless you want to wipe away all HDFS data.
    or , i just need to create a new dir for "dfs.data.dir" to make room
    for dfs.name.dir and change the hdfs-site.xml correspondingly ,eg.
    without coping the stuff and without reformating the namenode,the hdfs
    will detect this change and transfer some blocks to the new
    "dfs.data.dir" of this node
    i do not want to change the "dfs.name.dir " since i think 1 g for
    fsimage is enough for my testing cluster,and changing "dfs.name.dir "
    must entail reformat namenode.
    Changing dfs.name.dir does not warrant a format either. It can be
    done, but moving dfs.data.dir is safer.

    --
    Harsh J
  • Verditer1 at Oct 14, 2013 at 11:07 am
    Dear Harsh and others,

    I have similar situation as toughman In my case, we want both namenode and
    datanode to be moved to a different folder in another partition having more
    space.

    We do not have any important data in the HDFS, so are ok to lose all data
    in it.
    We changed the dfs.name.dir and dfs.data.dir, and when restarting got the
    error that 'the folder is in a inconsistent state. Storage directory not
    found.'

    Both datanode and namenode are not coming up.

    What do I do? Tried many combinations, tries but nothing worked....

    Please help.

    Suresh.
    On Wednesday, March 21, 2012 5:07:34 PM UTC+5:30, Harsh J wrote:

    Hey toughman,
    On Wed, Mar 21, 2012 at 1:08 PM, XIANGJUN wrote:
    So, i want to change the "dfs.data.dir" for the namenode to other
    volume, can i do this by create another dir and copy all the stuff in
    dfs.data.dir to the new dir and then change the hdfs-site.xml
    correspondingly and then restart those daemons in namenode? or ,do i
    need to re format the name node?
    Your approach is almost right:

    Make new base directory for dfs.data.dir that is on separate disk.
    Stop datanode, reconfigure hdfs-site.xml to point to new directory or
    dfs.data.dir
    Copy (not move yet) old data.dir contents to new location.
    Start datanode.
    If datanode starts and reports blocks properly and fsck reports no
    problems (and you can read files/etc.), then delete old data directory
    copy safely.

    Never format your NameNode unless you want to wipe away all HDFS data.
    or , i just need to create a new dir for "dfs.data.dir" to make room
    for dfs.name.dir and change the hdfs-site.xml correspondingly ,eg.
    without coping the stuff and without reformating the namenode,the hdfs
    will detect this change and transfer some blocks to the new
    "dfs.data.dir" of this node
    i do not want to change the "dfs.name.dir " since i think 1 g for
    fsimage is enough for my testing cluster,and changing "dfs.name.dir "
    must entail reformat namenode.
    Changing dfs.name.dir does not warrant a format either. It can be
    done, but moving dfs.data.dir is safer.

    --
    Harsh J
    --

    ---
    You received this message because you are subscribed to the Google Groups "CDH Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
    For more options, visit https://groups.google.com/a/cloudera.org/groups/opt_out.
  • Varun kumar at Oct 14, 2013 at 11:24 am
    Format your namenode.
    it works

    On Mon, Oct 14, 2013 at 4:37 PM, verditer1 wrote:

    Dear Harsh and others,

    I have similar situation as toughman In my case, we want both namenode and
    datanode to be moved to a different folder in another partition having more
    space.

    We do not have any important data in the HDFS, so are ok to lose all data
    in it.
    We changed the dfs.name.dir and dfs.data.dir, and when restarting got the
    error that 'the folder is in a inconsistent state. Storage directory not
    found.'

    Both datanode and namenode are not coming up.

    What do I do? Tried many combinations, tries but nothing worked....

    Please help.

    Suresh.
    On Wednesday, March 21, 2012 5:07:34 PM UTC+5:30, Harsh J wrote:

    Hey toughman,
    On Wed, Mar 21, 2012 at 1:08 PM, XIANGJUN wrote:
    So, i want to change the "dfs.data.dir" for the namenode to other
    volume, can i do this by create another dir and copy all the stuff in
    dfs.data.dir to the new dir and then change the hdfs-site.xml
    correspondingly and then restart those daemons in namenode? or ,do i
    need to re format the name node?
    Your approach is almost right:

    Make new base directory for dfs.data.dir that is on separate disk.
    Stop datanode, reconfigure hdfs-site.xml to point to new directory or
    dfs.data.dir
    Copy (not move yet) old data.dir contents to new location.
    Start datanode.
    If datanode starts and reports blocks properly and fsck reports no
    problems (and you can read files/etc.), then delete old data directory
    copy safely.

    Never format your NameNode unless you want to wipe away all HDFS data.
    or , i just need to create a new dir for "dfs.data.dir" to make room
    for dfs.name.dir and change the hdfs-site.xml correspondingly ,eg.
    without coping the stuff and without reformating the namenode,the hdfs
    will detect this change and transfer some blocks to the new
    "dfs.data.dir" of this node
    i do not want to change the "dfs.name.dir " since i think 1 g for
    fsimage is enough for my testing cluster,and changing "dfs.name.dir "
    must entail reformat namenode.
    Changing dfs.name.dir does not warrant a format either. It can be
    done, but moving dfs.data.dir is safer.

    --
    Harsh J

    --
    ---
    You received this message because you are subscribed to the Google Groups
    "CDH Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to cdh-user+unsubscribe@cloudera.org.
    For more options, visit
    https://groups.google.com/a/cloudera.org/groups/opt_out.


    --
    Regards,
    Varun Kumar.P

    --

    ---
    You received this message because you are subscribed to the Google Groups "CDH Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
    For more options, visit https://groups.google.com/a/cloudera.org/groups/opt_out.
  • Suresh V S at Oct 14, 2013 at 1:36 pm
    No, formatting namenode didn't work as well. Kept throwing the below error:

    2013-10-14 18:00:06,388 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
    org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /home/gpadmin/data/nn/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible.

    I started reinstalling the CentOS in frustration, and found that there are 2 hard disks in the machine, and are configured as a Logical Volume Manager combining both disks.

    Could it have been the cause for this error?

    Suresh.

    From: varun kumar
    Sent: Monday, October 14, 2013 4:54 PM
    To: cdh-user@cloudera.org; verditer@gmail.com
    Subject: Re: how can i change the "dfs.data.dir" for name node without any damage?

    Format your namenode.
    it works

    On Mon, Oct 14, 2013 at 4:37 PM, verditer1 wrote:
    Dear Harsh and others,

    I have similar situation as toughman In my case, we want both namenode and datanode to be moved to a different folder in another partition having more space.

    We do not have any important data in the HDFS, so are ok to lose all data in it.
    We changed the dfs.name.dir and dfs.data.dir, and when restarting got the error that 'the folder is in a inconsistent state. Storage directory not found.'

    Both datanode and namenode are not coming up.

    What do I do? Tried many combinations, tries but nothing worked....

    Please help.

    Suresh.

    On Wednesday, March 21, 2012 5:07:34 PM UTC+5:30, Harsh J wrote:
    Hey toughman,
    On Wed, Mar 21, 2012 at 1:08 PM, XIANGJUN wrote:
    So, i want to change the "dfs.data.dir" for the namenode to other
    volume, can i do this by create another dir and copy all the stuff in
    dfs.data.dir to the new dir and then change the hdfs-site.xml
    correspondingly and then restart those daemons in namenode? or ,do i
    need to re format the name node?
    Your approach is almost right:

    Make new base directory for dfs.data.dir that is on separate disk.
    Stop datanode, reconfigure hdfs-site.xml to point to new directory or
    dfs.data.dir
    Copy (not move yet) old data.dir contents to new location.
    Start datanode.
    If datanode starts and reports blocks properly and fsck reports no
    problems (and you can read files/etc.), then delete old data directory
    copy safely.

    Never format your NameNode unless you want to wipe away all HDFS data.
    or , i just need to create a new dir for "dfs.data.dir" to make room
    for dfs.name.dir and change the hdfs-site.xml correspondingly ,eg.
    without coping the stuff and without reformating the namenode,the hdfs
    will detect this change and transfer some blocks to the new
    "dfs.data.dir" of this node
    i do not want to change the "dfs.name.dir " since i think 1 g for
    fsimage is enough for my testing cluster,and changing "dfs.name.dir "
    must entail reformat namenode.
    Changing dfs.name.dir does not warrant a format either. It can be
    done, but moving dfs.data.dir is safer.

    --
    Harsh J
    --

    ---
    You received this message because you are subscribed to the Google Groups "CDH Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org .
    For more options, visit https://groups.google.com/a/cloudera.org/groups/opt_out.



    --
    Regards,
    Varun Kumar.P
    --

    ---
    You received this message because you are subscribed to the Google Groups "CDH Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org .
    For more options, visit https://groups.google.com/a/cloudera.org/groups/opt_out.

    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Disclaimer~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Information contained and transmitted by this e-mail is confidential and proprietary to iGATE and its affiliates and is intended for use only by the recipient. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, copying or use of this e-mail is strictly prohibited and you are requested to delete this e-mail immediately and notify the originator or mailadmin@igate.com . iGATE does not enter into any agreement with any party by e-mail. Any views expressed by an individual do not necessarily reflect the view of iGATE. iGATE is not responsible for the consequences of any actions taken on the basis of information provided, through this email. The contents of an attachment to this e-mail may contain software viruses, which could damage your own computer system. While iGATE has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checks before opening an attachment. To know more about iGATE please visit www.igate.com <http://www.igate.com>.
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    --

    ---
    You received this message because you are subscribed to the Google Groups "CDH Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
    For more options, visit https://groups.google.com/a/cloudera.org/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcdh-user @
categorieshadoop
postedMar 21, '12 at 7:38a
activeOct 14, '13 at 1:36p
posts5
users5
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase