Grokbase › Groups › Hadoop › common-user › August 2010
FAQ
The namenode log for a Hadoop-0.20 installation contains this error message:
"/var/lib/hadoop-0.20/cache/hadoop/dfs/name in in an inconsistent state".
This directory does not exist and I would like to understand why this
particular directory name is required (not what the directory is used for,
but why this particular directory name). The *-site.xml files are below (IP
addresses have been masked).
Thanks in advance for your help.
Cliff

core-site.xml:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://xxx.xxx.xxx.xxx:8020</value>
<final>true</final>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/var/lib/hadoop-0.20/cache/hadoop</value>
</property>
</configuration>
-------- end of core-site.xml -----------

hdfs-site.xml:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
<property>
</property>
<name>dfs.data.dir</name>
<value>/DFS1/dfsdata,/DFS2/dfsdata,/DFS3/dfsdata</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.datanode.du.reserved</name>
<value>20000000000</value>
</property>
<property>
<name>dfs.namenode,plugins</name>
<value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
</property>
<property>
<name>dfs.datanode.plugins</name>
<value>org.apache.hadoop.thriftfs.DatanodePlugins</value>
</property>
<property>
<name>dfs.thrift.address</name>
<value>0.0.0.0:9090</value>
</property>
</configuration>
-------- end of core-site.xml -----------

mapred-site.xml:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>xxx.xxx.xxx.xxx:8020</value>
</property>
<property>
<name>jobtracker.thrift.address</name>
<value>0.0.0:9290</value>
</property>
<property>
<name>mapred.jobtracker.plugins</name>

<value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
</property>
</configuration>
-------- end of core-site.xml -----------

Search Discussions

  • Harsh J at Aug 23, 2010 at 1:22 pm ⇧
    Its checking this directory cause your dfs.name.dir (hdfs-site.xml)
    has it in its list of dirs to write a copy to:

    <property>
    <name>dfs.name.dir</name>
    <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
    <property>

    Remove it from this property if you don't need it. If you need
    multiple dirs, you should create the path and allocate proper
    permissions to it so that it may write to it. Its good to use a second
    path for backup purposes (most have this on the NFS). The path is so
    because your distribution of hadoop perhaps placed its files around
    that place and its up in the conf files as some form of default :)
    On Mon, Aug 23, 2010 at 6:45 PM, cliff palmer wrote:
    The namenode log for a Hadoop-0.20 installation contains this error message:
    "/var/lib/hadoop-0.20/cache/hadoop/dfs/name in in an inconsistent state".
    This directory does not exist and I would like to understand why this
    particular directory name is required (not what the directory is used for,
    but why this particular directory name).  The *-site.xml files are below (IP
    addresses have been masked).
    Thanks in advance for your help.
    Cliff

    core-site.xml:

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
    <property>
    <name>fs.default.name</name>
    <value>hdfs://xxx.xxx.xxx.xxx:8020</value>
    <final>true</final>
    </property>
    <property>
    <name>hadoop.tmp.dir</name>
    <value>/var/lib/hadoop-0.20/cache/hadoop</value>
    </property>
    </configuration>
    -------- end of core-site.xml -----------

    hdfs-site.xml:

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
    <property>
    <name>dfs.replication</name>
    <value>3</value>
    </property>
    <property>
    <name>dfs.name.dir</name>
    <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
    <property>
    </property>
    <name>dfs.data.dir</name>
    <value>/DFS1/dfsdata,/DFS2/dfsdata,/DFS3/dfsdata</value>
    </property>
    <property>
    <name>dfs.permissions</name>
    <value>false</value>
    </property>
    <property>
    <name>dfs.datanode.du.reserved</name>
    <value>20000000000</value>
    </property>
    <property>
    <name>dfs.namenode,plugins</name>
    <value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
    </property>
    <property>
    <name>dfs.datanode.plugins</name>
    <value>org.apache.hadoop.thriftfs.DatanodePlugins</value>
    </property>
    <property>
    <name>dfs.thrift.address</name>
    <value>0.0.0.0:9090</value>
    </property>
    </configuration>
    -------- end of core-site.xml -----------

    mapred-site.xml:

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
    <property>
    <name>mapred.job.tracker</name>
    <value>xxx.xxx.xxx.xxx:8020</value>
    </property>
    <property>
    <name>jobtracker.thrift.address</name>
    <value>0.0.0:9290</value>
    </property>
    <property>
    <name>mapred.jobtracker.plugins</name>

    <value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
    </property>
    </configuration>
    -------- end of core-site.xml -----------


    --
    Harsh J
    www.harshj.com
  • Cliff palmer at Aug 23, 2010 at 1:49 pm ⇧
    Thanks Harsh, but I am still not sure I understand what is going on.
    The directory specified in the dfs.name.dir property,
    "/var/lib/hadoop-0.20/dfsname", does exist and rights to that directory have
    been granted to the OS user that is running the Hadoop startup script.
    The directory mentioned in the error message is
    "/var/lib/hadoop-0.20/cache/hadoop/dfs/name".
    I can create this directory and that would (I assume) remove the error, but
    I want to understand how the name is derived. It's not a child of the
    directory name specified in the dfs.name.dir property.
    Thanks again!
    Cliff

    On Mon, Aug 23, 2010 at 9:21 AM, Harsh J wrote:

    Its checking this directory cause your dfs.name.dir (hdfs-site.xml)
    has it in its list of dirs to write a copy to:

    <property>
    <name>dfs.name.dir</name>
    <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
    <property>

    Remove it from this property if you don't need it. If you need
    multiple dirs, you should create the path and allocate proper
    permissions to it so that it may write to it. Its good to use a second
    path for backup purposes (most have this on the NFS). The path is so
    because your distribution of hadoop perhaps placed its files around
    that place and its up in the conf files as some form of default :)
    On Mon, Aug 23, 2010 at 6:45 PM, cliff palmer wrote:
    The namenode log for a Hadoop-0.20 installation contains this error message:
    "/var/lib/hadoop-0.20/cache/hadoop/dfs/name in in an inconsistent state".
    This directory does not exist and I would like to understand why this
    particular directory name is required (not what the directory is used for,
    but why this particular directory name). The *-site.xml files are below (IP
    addresses have been masked).
    Thanks in advance for your help.
    Cliff

    core-site.xml:

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
    <property>
    <name>fs.default.name</name>
    <value>hdfs://xxx.xxx.xxx.xxx:8020</value>
    <final>true</final>
    </property>
    <property>
    <name>hadoop.tmp.dir</name>
    <value>/var/lib/hadoop-0.20/cache/hadoop</value>
    </property>
    </configuration>
    -------- end of core-site.xml -----------

    hdfs-site.xml:

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
    <property>
    <name>dfs.replication</name>
    <value>3</value>
    </property>
    <property>
    <name>dfs.name.dir</name>
    <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
    <property>
    </property>
    <name>dfs.data.dir</name>
    <value>/DFS1/dfsdata,/DFS2/dfsdata,/DFS3/dfsdata</value>
    </property>
    <property>
    <name>dfs.permissions</name>
    <value>false</value>
    </property>
    <property>
    <name>dfs.datanode.du.reserved</name>
    <value>20000000000</value>
    </property>
    <property>
    <name>dfs.namenode,plugins</name>
    <value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
    </property>
    <property>
    <name>dfs.datanode.plugins</name>
    <value>org.apache.hadoop.thriftfs.DatanodePlugins</value>
    </property>
    <property>
    <name>dfs.thrift.address</name>
    <value>0.0.0.0:9090</value>
    </property>
    </configuration>
    -------- end of core-site.xml -----------

    mapred-site.xml:

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
    <property>
    <name>mapred.job.tracker</name>
    <value>xxx.xxx.xxx.xxx:8020</value>
    </property>
    <property>
    <name>jobtracker.thrift.address</name>
    <value>0.0.0:9290</value>
    </property>
    <property>
    <name>mapred.jobtracker.plugins</name>

    <value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
    </property>
    </configuration>
    -------- end of core-site.xml -----------


    --
    Harsh J
    www.harshj.com
  • Harsh J at Aug 23, 2010 at 2:28 pm ⇧
    Can you confirm that this is the right configuration your NN is starting with?
    On Mon, Aug 23, 2010 at 7:19 PM, cliff palmer wrote:
    Thanks Harsh, but I am still not sure I understand what is going on.
    The directory specified in the dfs.name.dir property,
    "/var/lib/hadoop-0.20/dfsname", does exist and rights to that directory have
    been granted to the OS user that is running the Hadoop startup script.
    The directory mentioned in the error message is
    "/var/lib/hadoop-0.20/cache/hadoop/dfs/name".
    I can create this directory and that would (I assume) remove the error, but
    I want to understand how the name is derived.  It's not a child of the
    directory name specified in the dfs.name.dir property.
    Thanks again!
    Cliff

    On Mon, Aug 23, 2010 at 9:21 AM, Harsh J wrote:

    Its checking this directory cause your dfs.name.dir (hdfs-site.xml)
    has it in its list of dirs to write a copy to:

    <property>
    <name>dfs.name.dir</name>
    <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
    <property>

    Remove it from this property if you don't need it. If you need
    multiple dirs, you should create the path and allocate proper
    permissions to it so that it may write to it. Its good to use a second
    path for backup purposes (most have this on the NFS). The path is so
    because your distribution of hadoop perhaps placed its files around
    that place and its up in the conf files as some form of default :)

    On Mon, Aug 23, 2010 at 6:45 PM, cliff palmer <[email protected]>
    wrote:
    The namenode log for a Hadoop-0.20 installation contains this error message:
    "/var/lib/hadoop-0.20/cache/hadoop/dfs/name in in an inconsistent state".
    This directory does not exist and I would like to understand why this
    particular directory name is required (not what the directory is used for,
    but why this particular directory name).  The *-site.xml files are below (IP
    addresses have been masked).
    Thanks in advance for your help.
    Cliff

    core-site.xml:

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
    <property>
    <name>fs.default.name</name>
    <value>hdfs://xxx.xxx.xxx.xxx:8020</value>
    <final>true</final>
    </property>
    <property>
    <name>hadoop.tmp.dir</name>
    <value>/var/lib/hadoop-0.20/cache/hadoop</value>
    </property>
    </configuration>
    -------- end of core-site.xml -----------

    hdfs-site.xml:

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
    <property>
    <name>dfs.replication</name>
    <value>3</value>
    </property>
    <property>
    <name>dfs.name.dir</name>
    <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
    <property>
    </property>
    <name>dfs.data.dir</name>
    <value>/DFS1/dfsdata,/DFS2/dfsdata,/DFS3/dfsdata</value>
    </property>
    <property>
    <name>dfs.permissions</name>
    <value>false</value>
    </property>
    <property>
    <name>dfs.datanode.du.reserved</name>
    <value>20000000000</value>
    </property>
    <property>
    <name>dfs.namenode,plugins</name>
    <value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
    </property>
    <property>
    <name>dfs.datanode.plugins</name>
    <value>org.apache.hadoop.thriftfs.DatanodePlugins</value>
    </property>
    <property>
    <name>dfs.thrift.address</name>
    <value>0.0.0.0:9090</value>
    </property>
    </configuration>
    -------- end of core-site.xml -----------

    mapred-site.xml:

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
    <property>
    <name>mapred.job.tracker</name>
    <value>xxx.xxx.xxx.xxx:8020</value>
    </property>
    <property>
    <name>jobtracker.thrift.address</name>
    <value>0.0.0:9290</value>
    </property>
    <property>
    <name>mapred.jobtracker.plugins</name>

    <value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
    </property>
    </configuration>
    -------- end of core-site.xml -----------


    --
    Harsh J
    www.harshj.com


    --
    Harsh J
    www.harshj.com
  • Cliff palmer at Aug 23, 2010 at 4:09 pm ⇧
    The 3 *-site.xml files are in the /etc/hadoop-0.20/conf directory. I've
    confirmed that these are the files being used.
    Thanks again.
    Cliff
    On Mon, Aug 23, 2010 at 10:26 AM, Harsh J wrote:

    Can you confirm that this is the right configuration your NN is starting
    with?
    On Mon, Aug 23, 2010 at 7:19 PM, cliff palmer wrote:
    Thanks Harsh, but I am still not sure I understand what is going on.
    The directory specified in the dfs.name.dir property,
    "/var/lib/hadoop-0.20/dfsname", does exist and rights to that directory have
    been granted to the OS user that is running the Hadoop startup script.
    The directory mentioned in the error message is
    "/var/lib/hadoop-0.20/cache/hadoop/dfs/name".
    I can create this directory and that would (I assume) remove the error, but
    I want to understand how the name is derived. It's not a child of the
    directory name specified in the dfs.name.dir property.
    Thanks again!
    Cliff

    On Mon, Aug 23, 2010 at 9:21 AM, Harsh J wrote:

    Its checking this directory cause your dfs.name.dir (hdfs-site.xml)
    has it in its list of dirs to write a copy to:

    <property>
    <name>dfs.name.dir</name>
    <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
    <property>

    Remove it from this property if you don't need it. If you need
    multiple dirs, you should create the path and allocate proper
    permissions to it so that it may write to it. Its good to use a second
    path for backup purposes (most have this on the NFS). The path is so
    because your distribution of hadoop perhaps placed its files around
    that place and its up in the conf files as some form of default :)

    On Mon, Aug 23, 2010 at 6:45 PM, cliff palmer <[email protected]>
    wrote:
    The namenode log for a Hadoop-0.20 installation contains this error message:
    "/var/lib/hadoop-0.20/cache/hadoop/dfs/name in in an inconsistent
    state".
    This directory does not exist and I would like to understand why this
    particular directory name is required (not what the directory is used for,
    but why this particular directory name). The *-site.xml files are
    below
    (IP
    addresses have been masked).
    Thanks in advance for your help.
    Cliff

    core-site.xml:

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
    <property>
    <name>fs.default.name</name>
    <value>hdfs://xxx.xxx.xxx.xxx:8020</value>
    <final>true</final>
    </property>
    <property>
    <name>hadoop.tmp.dir</name>
    <value>/var/lib/hadoop-0.20/cache/hadoop</value>
    </property>
    </configuration>
    -------- end of core-site.xml -----------

    hdfs-site.xml:

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
    <property>
    <name>dfs.replication</name>
    <value>3</value>
    </property>
    <property>
    <name>dfs.name.dir</name>
    <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
    <property>
    </property>
    <name>dfs.data.dir</name>
    <value>/DFS1/dfsdata,/DFS2/dfsdata,/DFS3/dfsdata</value>
    </property>
    <property>
    <name>dfs.permissions</name>
    <value>false</value>
    </property>
    <property>
    <name>dfs.datanode.du.reserved</name>
    <value>20000000000</value>
    </property>
    <property>
    <name>dfs.namenode,plugins</name>
    <value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
    </property>
    <property>
    <name>dfs.datanode.plugins</name>
    <value>org.apache.hadoop.thriftfs.DatanodePlugins</value>
    </property>
    <property>
    <name>dfs.thrift.address</name>
    <value>0.0.0.0:9090</value>
    </property>
    </configuration>
    -------- end of core-site.xml -----------

    mapred-site.xml:

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
    <property>
    <name>mapred.job.tracker</name>
    <value>xxx.xxx.xxx.xxx:8020</value>
    </property>
    <property>
    <name>jobtracker.thrift.address</name>
    <value>0.0.0:9290</value>
    </property>
    <property>
    <name>mapred.jobtracker.plugins</name>

    <value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
    </property>
    </configuration>
    -------- end of core-site.xml -----------


    --
    Harsh J
    www.harshj.com


    --
    Harsh J
    www.harshj.com
  • Edward Capriolo at Aug 23, 2010 at 4:13 pm ⇧

    On Mon, Aug 23, 2010 at 12:09 PM, cliff palmer wrote:
    The 3 *-site.xml files are in the /etc/hadoop-0.20/conf directory.  I've
    confirmed that these are the files being used.
    Thanks again.
    Cliff
    On Mon, Aug 23, 2010 at 10:26 AM, Harsh J wrote:

    Can you confirm that this is the right configuration your NN is starting
    with?

    On Mon, Aug 23, 2010 at 7:19 PM, cliff palmer <[email protected]>
    wrote:
    Thanks Harsh, but I am still not sure I understand what is going on.
    The directory specified in the dfs.name.dir property,
    "/var/lib/hadoop-0.20/dfsname", does exist and rights to that directory have
    been granted to the OS user that is running the Hadoop startup script.
    The directory mentioned in the error message is
    "/var/lib/hadoop-0.20/cache/hadoop/dfs/name".
    I can create this directory and that would (I assume) remove the error, but
    I want to understand how the name is derived.  It's not a child of the
    directory name specified in the dfs.name.dir property.
    Thanks again!
    Cliff

    On Mon, Aug 23, 2010 at 9:21 AM, Harsh J wrote:

    Its checking this directory cause your dfs.name.dir (hdfs-site.xml)
    has it in its list of dirs to write a copy to:

    <property>
    <name>dfs.name.dir</name>
    <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
    <property>

    Remove it from this property if you don't need it. If you need
    multiple dirs, you should create the path and allocate proper
    permissions to it so that it may write to it. Its good to use a second
    path for backup purposes (most have this on the NFS). The path is so
    because your distribution of hadoop perhaps placed its files around
    that place and its up in the conf files as some form of default :)

    On Mon, Aug 23, 2010 at 6:45 PM, cliff palmer <[email protected]>
    wrote:
    The namenode log for a Hadoop-0.20 installation contains this error message:
    "/var/lib/hadoop-0.20/cache/hadoop/dfs/name in in an inconsistent
    state".
    This directory does not exist and I would like to understand why this
    particular directory name is required (not what the directory is used for,
    but why this particular directory name).  The *-site.xml files are
    below
    (IP
    addresses have been masked).
    Thanks in advance for your help.
    Cliff

    core-site.xml:

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
    <property>
    <name>fs.default.name</name>
    <value>hdfs://xxx.xxx.xxx.xxx:8020</value>
    <final>true</final>
    </property>
    <property>
    <name>hadoop.tmp.dir</name>
    <value>/var/lib/hadoop-0.20/cache/hadoop</value>
    </property>
    </configuration>
    -------- end of core-site.xml -----------

    hdfs-site.xml:

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
    <property>
    <name>dfs.replication</name>
    <value>3</value>
    </property>
    <property>
    <name>dfs.name.dir</name>
    <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
    <property>
    </property>
    <name>dfs.data.dir</name>
    <value>/DFS1/dfsdata,/DFS2/dfsdata,/DFS3/dfsdata</value>
    </property>
    <property>
    <name>dfs.permissions</name>
    <value>false</value>
    </property>
    <property>
    <name>dfs.datanode.du.reserved</name>
    <value>20000000000</value>
    </property>
    <property>
    <name>dfs.namenode,plugins</name>
    <value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
    </property>
    <property>
    <name>dfs.datanode.plugins</name>
    <value>org.apache.hadoop.thriftfs.DatanodePlugins</value>
    </property>
    <property>
    <name>dfs.thrift.address</name>
    <value>0.0.0.0:9090</value>
    </property>
    </configuration>
    -------- end of core-site.xml -----------

    mapred-site.xml:

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
    <property>
    <name>mapred.job.tracker</name>
    <value>xxx.xxx.xxx.xxx:8020</value>
    </property>
    <property>
    <name>jobtracker.thrift.address</name>
    <value>0.0.0:9290</value>
    </property>
    <property>
    <name>mapred.jobtracker.plugins</name>

    <value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
    </property>
    </configuration>
    -------- end of core-site.xml -----------


    --
    Harsh J
    www.harshj.com


    --
    Harsh J
    www.harshj.com
    Check the permissions of the paths above this one. Make sure parent
    directories have execute(search) permission for the user you run
    hadoop as.
  • Cliff palmer at Aug 23, 2010 at 5:43 pm ⇧
    Yes, the hadoop user has execute permissions on the parent directories.
    Thanks!
    Cliff
    On Mon, Aug 23, 2010 at 12:12 PM, Edward Capriolo wrote:
    On Mon, Aug 23, 2010 at 12:09 PM, cliff palmer wrote:
    The 3 *-site.xml files are in the /etc/hadoop-0.20/conf directory. I've
    confirmed that these are the files being used.
    Thanks again.
    Cliff
    On Mon, Aug 23, 2010 at 10:26 AM, Harsh J wrote:

    Can you confirm that this is the right configuration your NN is starting
    with?

    On Mon, Aug 23, 2010 at 7:19 PM, cliff palmer <[email protected]>
    wrote:
    Thanks Harsh, but I am still not sure I understand what is going on.
    The directory specified in the dfs.name.dir property,
    "/var/lib/hadoop-0.20/dfsname", does exist and rights to that
    directory
    have
    been granted to the OS user that is running the Hadoop startup script.
    The directory mentioned in the error message is
    "/var/lib/hadoop-0.20/cache/hadoop/dfs/name".
    I can create this directory and that would (I assume) remove the
    error,
    but
    I want to understand how the name is derived. It's not a child of the
    directory name specified in the dfs.name.dir property.
    Thanks again!
    Cliff

    On Mon, Aug 23, 2010 at 9:21 AM, Harsh J wrote:

    Its checking this directory cause your dfs.name.dir (hdfs-site.xml)
    has it in its list of dirs to write a copy to:

    <property>
    <name>dfs.name.dir</name>
    <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
    <property>

    Remove it from this property if you don't need it. If you need
    multiple dirs, you should create the path and allocate proper
    permissions to it so that it may write to it. Its good to use a
    second
    path for backup purposes (most have this on the NFS). The path is so
    because your distribution of hadoop perhaps placed its files around
    that place and its up in the conf files as some form of default :)

    On Mon, Aug 23, 2010 at 6:45 PM, cliff palmer <[email protected]
    wrote:
    The namenode log for a Hadoop-0.20 installation contains this error message:
    "/var/lib/hadoop-0.20/cache/hadoop/dfs/name in in an inconsistent
    state".
    This directory does not exist and I would like to understand why
    this
    particular directory name is required (not what the directory is
    used
    for,
    but why this particular directory name). The *-site.xml files are
    below
    (IP
    addresses have been masked).
    Thanks in advance for your help.
    Cliff

    core-site.xml:

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
    <property>
    <name>fs.default.name</name>
    <value>hdfs://xxx.xxx.xxx.xxx:8020</value>
    <final>true</final>
    </property>
    <property>
    <name>hadoop.tmp.dir</name>
    <value>/var/lib/hadoop-0.20/cache/hadoop</value>
    </property>
    </configuration>
    -------- end of core-site.xml -----------

    hdfs-site.xml:

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
    <property>
    <name>dfs.replication</name>
    <value>3</value>
    </property>
    <property>
    <name>dfs.name.dir</name>
    <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
    <property>
    </property>
    <name>dfs.data.dir</name>
    <value>/DFS1/dfsdata,/DFS2/dfsdata,/DFS3/dfsdata</value>
    </property>
    <property>
    <name>dfs.permissions</name>
    <value>false</value>
    </property>
    <property>
    <name>dfs.datanode.du.reserved</name>
    <value>20000000000</value>
    </property>
    <property>
    <name>dfs.namenode,plugins</name>
    <value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
    </property>
    <property>
    <name>dfs.datanode.plugins</name>
    <value>org.apache.hadoop.thriftfs.DatanodePlugins</value>
    </property>
    <property>
    <name>dfs.thrift.address</name>
    <value>0.0.0.0:9090</value>
    </property>
    </configuration>
    -------- end of core-site.xml -----------

    mapred-site.xml:

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
    <property>
    <name>mapred.job.tracker</name>
    <value>xxx.xxx.xxx.xxx:8020</value>
    </property>
    <property>
    <name>jobtracker.thrift.address</name>
    <value>0.0.0:9290</value>
    </property>
    <property>
    <name>mapred.jobtracker.plugins</name>

    <value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
    </property>
    </configuration>
    -------- end of core-site.xml -----------


    --
    Harsh J
    www.harshj.com


    --
    Harsh J
    www.harshj.com
    Check the permissions of the paths above this one. Make sure parent
    directories have execute(search) permission for the user you run
    hadoop as.
  • Allen Wittenauer at Aug 24, 2010 at 7:06 pm ⇧

    On Aug 23, 2010, at 6:49 AM, cliff palmer wrote:

    Thanks Harsh, but I am still not sure I understand what is going on.
    The directory specified in the dfs.name.dir property,
    "/var/lib/hadoop-0.20/dfsname", does exist and rights to that directory have
    been granted to the OS user that is running the Hadoop startup script.
    The directory mentioned in the error message is
    "/var/lib/hadoop-0.20/cache/hadoop/dfs/name".
    I can create this directory and that would (I assume) remove the error, but
    I want to understand how the name is derived.
    From here:
    <property>
    <name>hadoop.tmp.dir</name>
    <value>/var/lib/hadoop-0.20/cache/hadoop</value>
    </property>
    </configuration> because:
    <property>
    <name>dfs.name.dir</name>
    <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
    <property>
    is missing a / in the property line.
  • Cliff palmer at Aug 24, 2010 at 11:15 pm ⇧
    Thanks Allen - that has resolved the problem. Good catch!
    Cliff

    On Tue, Aug 24, 2010 at 3:05 PM, Allen Wittenauer
    wrote:
    On Aug 23, 2010, at 6:49 AM, cliff palmer wrote:

    Thanks Harsh, but I am still not sure I understand what is going on.
    The directory specified in the dfs.name.dir property,
    "/var/lib/hadoop-0.20/dfsname", does exist and rights to that directory have
    been granted to the OS user that is running the Hadoop startup script.
    The directory mentioned in the error message is
    "/var/lib/hadoop-0.20/cache/hadoop/dfs/name".
    I can create this directory and that would (I assume) remove the error, but
    I want to understand how the name is derived.
    From here:
    <property>
    <name>hadoop.tmp.dir</name>
    <value>/var/lib/hadoop-0.20/cache/hadoop</value>
    </property>
    </configuration> because:
    <property>
    <name>dfs.name.dir</name>
    <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
    <property>
    is missing a / in the property line.



  • Hemanth Yamijala at Aug 25, 2010 at 6:05 pm ⇧
    Hmm. Without the / in the property tag, isn't the file malformed XML ?
    I am pretty sure Hadoop complains in such cases ?
    On Wed, Aug 25, 2010 at 4:44 AM, cliff palmer wrote:
    Thanks Allen - that has resolved the problem.  Good catch!
    Cliff

    On Tue, Aug 24, 2010 at 3:05 PM, Allen Wittenauer
    wrote:
    On Aug 23, 2010, at 6:49 AM, cliff palmer wrote:

    Thanks Harsh, but I am still not sure I understand what is going on.
    The directory specified in the dfs.name.dir property,
    "/var/lib/hadoop-0.20/dfsname", does exist and rights to that directory have
    been granted to the OS user that is running the Hadoop startup script.
    The directory mentioned in the error message is
    "/var/lib/hadoop-0.20/cache/hadoop/dfs/name".
    I can create this directory and that would (I assume) remove the error, but
    I want to understand how the name is derived.
    From here:
    <property>
    <name>hadoop.tmp.dir</name>
    <value>/var/lib/hadoop-0.20/cache/hadoop</value>
    </property>
    </configuration> because:
    <property>
    <name>dfs.name.dir</name>
    <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
    <property>
    is missing a / in the property line.



  • Cliff palmer at Aug 25, 2010 at 6:17 pm ⇧
    No complaints from hadoop, other than the error for the mangled directory
    name.
    On Wed, Aug 25, 2010 at 2:04 PM, Hemanth Yamijala wrote:

    Hmm. Without the / in the property tag, isn't the file malformed XML ?
    I am pretty sure Hadoop complains in such cases ?
    On Wed, Aug 25, 2010 at 4:44 AM, cliff palmer wrote:
    Thanks Allen - that has resolved the problem. Good catch!
    Cliff

    On Tue, Aug 24, 2010 at 3:05 PM, Allen Wittenauer
    wrote:
    On Aug 23, 2010, at 6:49 AM, cliff palmer wrote:

    Thanks Harsh, but I am still not sure I understand what is going on.
    The directory specified in the dfs.name.dir property,
    "/var/lib/hadoop-0.20/dfsname", does exist and rights to that
    directory
    have
    been granted to the OS user that is running the Hadoop startup script.
    The directory mentioned in the error message is
    "/var/lib/hadoop-0.20/cache/hadoop/dfs/name".
    I can create this directory and that would (I assume) remove the
    error,
    but
    I want to understand how the name is derived.
    From here:
    <property>
    <name>hadoop.tmp.dir</name>
    <value>/var/lib/hadoop-0.20/cache/hadoop</value>
    </property>
    </configuration> because:
    <property>
    <name>dfs.name.dir</name>
    <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
    <property>
    is missing a / in the property line.



Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedAug 23, '10 at 1:16p
activeAug 25, '10 at 6:17p
posts11
users5
websitehadoop.apache.org...
irc#hadoop

5 users in discussion

Cliff palmer: 6 posts Harsh J: 2 posts Edward Capriolo: 1 post Allen Wittenauer: 1 post Hemanth Yamijala: 1 post

Content

People

Support

Translate

site design / logo © 2023 Grokbase