FAQ
Hello,

I decided to change my pseudo distributed cluster to use YARN. I'm
already running CDH4b1. I followed the steps provided here

https://ccp.cloudera.com/display/CDH4B1/Installing+CDH4+on+a+Single+Linux+Node+in+Pseudo-distributed+Mode#InstallingCDH4onaSingleLinuxNodeinPseudo-distributedMode-InstallingCDH4withYARNonaSingleLinuxNodeinPseudodistributedmode

but unfortunately I am unable to start the data node. When I attempt
to start the data node I receive the message stating, "datanode dead
but pid file exists." I am unable to get HDFS up and running again.

I can start the namenode, but when I start the datanode it logs a
java.io.ioexception: incompatible clusterIDs...

How do I synchronize the namenode and datanode cluster IDs?

Search Discussions

  • Todd Lipcon at Mar 28, 2012 at 8:21 pm
    Hi Kevin,

    Sounds like you reformatted your NameNode somewhere along the way but
    did not reformat your DN. You'll need to blow away your current
    dfs.data.dirs -- see your hdfs-site.xml to see where they're set to.

    -Todd
    On Wed, Mar 28, 2012 at 12:47 PM, Kevin wrote:
    Hello,

    I decided to change my pseudo distributed cluster to use YARN. I'm
    already running CDH4b1. I followed the steps provided here

    https://ccp.cloudera.com/display/CDH4B1/Installing+CDH4+on+a+Single+Linux+Node+in+Pseudo-distributed+Mode#InstallingCDH4onaSingleLinuxNodeinPseudo-distributedMode-InstallingCDH4withYARNonaSingleLinuxNodeinPseudodistributedmode

    but unfortunately I am unable to start the data node. When I attempt
    to start the data node I receive the message stating, "datanode dead
    but pid file exists." I am unable to get HDFS up and running again.

    I can start the namenode, but when I start the datanode it logs a
    java.io.ioexception: incompatible clusterIDs...

    How do I synchronize the namenode and datanode cluster IDs?


    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Kevin at Mar 28, 2012 at 9:32 pm
    Thanks for replying Todd.

    I changed the clusterID in the datanode's VERSION file to the current
    clusterID in the namenode's current VERSION file. This seemed to fix
    that problem.

    [REF: https://issues.apache.org/jira/browse/HDFS-107]

    Unfortunately, and I think it's unrelated to the original problem, I
    am unable to run the hadoop example job. Hadoop returns with an error
    message

    ERROR security.UserGroupInformation: PriviledgedActionException as:
    kevin (auth:SIMPLE) cause: java.io.IOException: the ownership/
    permissions the staing directory /tmp/hadoop-yarn/staging/
    kevin/.staging is not as expected. It is owned by kein and permissions
    are rwxrwxrwt. The directory must be owned by the submitter kevin or
    by kevin and permissions must be rwx------

    It's a detailed message, but I'm can't seem to fix my permissions to
    work with Hadoop.
    On Mar 28, 4:21 pm, Todd Lipcon wrote:
    Hi Kevin,

    Sounds like you reformatted your NameNode somewhere along the way but
    did not reformat your DN. You'll need to blow away your current
    dfs.data.dirs -- see your hdfs-site.xml to see where they're set to.

    -Todd








    On Wed, Mar 28, 2012 at 12:47 PM, Kevin wrote:
    Hello,
    I decided to change my pseudo distributed cluster to use YARN. I'm
    already running CDH4b1. I followed the steps provided here
    https://ccp.cloudera.com/display/CDH4B1/Installing+CDH4+on+a+Single+L...
    but unfortunately I am unable to start the data node. When I attempt
    to start the data node I receive the message stating, "datanode dead
    but pid file exists." I am unable to get HDFS up and running again.
    I can start the namenode, but when I start the datanode it logs a
    java.io.ioexception: incompatible clusterIDs...
    How do I synchronize the namenode and datanode cluster IDs?
    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Todd Lipcon at Mar 28, 2012 at 9:34 pm
    Hi Kevin,

    Sorry, not sure on this one. Hopefully someone else can jump in and
    help out. I'm more an HDFS guy of late :)

    -Todd
    On Wed, Mar 28, 2012 at 2:31 PM, Kevin wrote:
    Thanks for replying Todd.

    I changed the clusterID in the datanode's VERSION file to the current
    clusterID in the namenode's current VERSION file. This seemed to fix
    that problem.

    [REF: https://issues.apache.org/jira/browse/HDFS-107]

    Unfortunately, and I think it's unrelated to the original problem, I
    am unable to run the hadoop example job. Hadoop returns with an error
    message

    ERROR security.UserGroupInformation: PriviledgedActionException as:
    kevin (auth:SIMPLE) cause: java.io.IOException: the ownership/
    permissions the staing directory /tmp/hadoop-yarn/staging/
    kevin/.staging is not as expected. It is owned by kein and permissions
    are rwxrwxrwt. The directory must be owned by the submitter kevin or
    by kevin and permissions must be rwx------

    It's a detailed message, but I'm can't seem to fix my permissions to
    work with Hadoop.
    On Mar 28, 4:21 pm, Todd Lipcon wrote:
    Hi Kevin,

    Sounds like you reformatted your NameNode somewhere along the way but
    did not reformat your DN. You'll need to blow away your current
    dfs.data.dirs -- see your hdfs-site.xml to see where they're set to.

    -Todd








    On Wed, Mar 28, 2012 at 12:47 PM, Kevin wrote:
    Hello,
    I decided to change my pseudo distributed cluster to use YARN. I'm
    already running CDH4b1. I followed the steps provided here
    https://ccp.cloudera.com/display/CDH4B1/Installing+CDH4+on+a+Single+L...
    but unfortunately I am unable to start the data node. When I attempt
    to start the data node I receive the message stating, "datanode dead
    but pid file exists." I am unable to get HDFS up and running again.
    I can start the namenode, but when I start the datanode it logs a
    java.io.ioexception: incompatible clusterIDs...
    How do I synchronize the namenode and datanode cluster IDs?
    --
    Todd Lipcon
    Software Engineer, Cloudera


    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Justin Workman at Mar 28, 2012 at 10:01 pm
    Looks to me the ownership is wrong. Your user id is being reported as "Kevin" and owner in the file system is "kein".

    Sent from my iPhone
    On Mar 28, 2012, at 3:31 PM, Kevin wrote:

    Thanks for replying Todd.

    I changed the clusterID in the datanode's VERSION file to the current
    clusterID in the namenode's current VERSION file. This seemed to fix
    that problem.

    [REF: https://issues.apache.org/jira/browse/HDFS-107]

    Unfortunately, and I think it's unrelated to the original problem, I
    am unable to run the hadoop example job. Hadoop returns with an error
    message

    ERROR security.UserGroupInformation: PriviledgedActionException as:
    kevin (auth:SIMPLE) cause: java.io.IOException: the ownership/
    permissions the staing directory /tmp/hadoop-yarn/staging/
    kevin/.staging is not as expected. It is owned by kein and permissions
    are rwxrwxrwt. The directory must be owned by the submitter kevin or
    by kevin and permissions must be rwx------

    It's a detailed message, but I'm can't seem to fix my permissions to
    work with Hadoop.
    On Mar 28, 4:21 pm, Todd Lipcon wrote:
    Hi Kevin,

    Sounds like you reformatted your NameNode somewhere along the way but
    did not reformat your DN. You'll need to blow away your current
    dfs.data.dirs -- see your hdfs-site.xml to see where they're set to.

    -Todd








    On Wed, Mar 28, 2012 at 12:47 PM, Kevin wrote:
    Hello,
    I decided to change my pseudo distributed cluster to use YARN. I'm
    already running CDH4b1. I followed the steps provided here
    https://ccp.cloudera.com/display/CDH4B1/Installing+CDH4+on+a+Single+L...
    but unfortunately I am unable to start the data node. When I attempt
    to start the data node I receive the message stating, "datanode dead
    but pid file exists." I am unable to get HDFS up and running again.
    I can start the namenode, but when I start the datanode it logs a
    java.io.ioexception: incompatible clusterIDs...
    How do I synchronize the namenode and datanode cluster IDs?
    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Harsh J at Mar 29, 2012 at 4:07 am
    Justin's right. Did you change your permissions of the staging
    directory at some point? Or of the whole FS?

    Run:
    hdfs dfs -chown kevin:kevin /tmp/hadoop-yarn/staging/kevin/
    hdfs dfs -chmod 700 /tmp/hadoop-yarn/staging/kevin/

    You can also disable permissions if its a bother and you do not need
    it. Set "dfs.permissions.enabled" to false and restart NameNode(s).

    On Thu, Mar 29, 2012 at 3:31 AM, Justin Workman
    wrote:
    Looks to me the ownership is wrong. Your user id is being reported as "Kevin" and owner in the file system is "kein".

    Sent from my iPhone
    On Mar 28, 2012, at 3:31 PM, Kevin wrote:

    Thanks for replying Todd.

    I changed the clusterID in the datanode's VERSION file to the current
    clusterID in the namenode's current VERSION file. This seemed to fix
    that problem.

    [REF: https://issues.apache.org/jira/browse/HDFS-107]

    Unfortunately, and I think it's unrelated to the original problem, I
    am unable to run the hadoop example job. Hadoop returns with an error
    message

    ERROR security.UserGroupInformation: PriviledgedActionException as:
    kevin (auth:SIMPLE) cause: java.io.IOException: the ownership/
    permissions the staing directory /tmp/hadoop-yarn/staging/
    kevin/.staging is not as expected. It is owned by kein and permissions
    are rwxrwxrwt. The directory must be owned by the submitter kevin or
    by kevin and permissions must be rwx------

    It's a detailed message, but I'm can't seem to fix my permissions to
    work with Hadoop.
    On Mar 28, 4:21 pm, Todd Lipcon wrote:
    Hi Kevin,

    Sounds like you reformatted your NameNode somewhere along the way but
    did not reformat your DN. You'll need to blow away your current
    dfs.data.dirs -- see your hdfs-site.xml to see where they're set to.

    -Todd








    On Wed, Mar 28, 2012 at 12:47 PM, Kevin wrote:
    Hello,
    I decided to change my pseudo distributed cluster to use YARN. I'm
    already running CDH4b1. I followed the steps provided here
    https://ccp.cloudera.com/display/CDH4B1/Installing+CDH4+on+a+Single+L...
    but unfortunately I am unable to start the data node. When I attempt
    to start the data node I receive the message stating, "datanode dead
    but pid file exists." I am unable to get HDFS up and running again.
    I can start the namenode, but when I start the datanode it logs a
    java.io.ioexception: incompatible clusterIDs...
    How do I synchronize the namenode and datanode cluster IDs?
    --
    Todd Lipcon
    Software Engineer, Cloudera


    --
    Harsh J
  • Kevin at Mar 29, 2012 at 1:38 pm
    Thanks for all the replies.

    I made a typo, Justin. I had to type the error message from the
    terminal and I lost the "v" in "kevin."

    Harsh, I tried everything you said and I keep getting the same error
    message - even disabling permissions doesn't resolve the issue (that
    concerns me). (But I think I know why: read below.)

    After executing the commands that Harsh suggested, I changed the
    permissions to remove the sticky bit and everything passes now ["sudo -
    u hdfs hadoop fs -chmod 0700 /tmp/hadoop-yarn/staging/
    kevin/.staging"].

    Thank you, everyone, for all the help.


    On Mar 29, 12:06 am, Harsh J wrote:
    Justin's right. Did you change your permissions of the staging
    directory at some point? Or of the whole FS?

    Run:
    hdfs dfs -chown kevin:kevin /tmp/hadoop-yarn/staging/kevin/
    hdfs dfs -chmod 700 /tmp/hadoop-yarn/staging/kevin/

    You can also disable permissions if its a bother and you do not need
    it. Set "dfs.permissions.enabled" to false and restart NameNode(s).

    On Thu, Mar 29, 2012 at 3:31 AM, Justin Workman









    wrote:
    Looks to me the ownership is wrong. Your user id is being reported as "Kevin" and owner in the file system is "kein".
    Sent from my iPhone
    On Mar 28, 2012, at 3:31 PM, Kevin wrote:

    Thanks for replying Todd.
    I changed the clusterID in the datanode's VERSION file to the current
    clusterID in the namenode's current VERSION file. This seemed to fix
    that problem.
    [REF:https://issues.apache.org/jira/browse/HDFS-107]
    Unfortunately, and I think it's unrelated to the original problem, I
    am unable to run the hadoop example job. Hadoop returns with an error
    message
    ERROR security.UserGroupInformation: PriviledgedActionException as:
    kevin (auth:SIMPLE) cause: java.io.IOException: the ownership/
    permissions the staing directory /tmp/hadoop-yarn/staging/
    kevin/.staging is not as expected. It is owned by kein and permissions
    are rwxrwxrwt. The directory must be owned by the submitter kevin or
    by kevin and permissions must be rwx------
    It's a detailed message, but I'm can't seem to fix my permissions to
    work with Hadoop.
    On Mar 28, 4:21 pm, Todd Lipcon wrote:
    Hi Kevin,
    Sounds like you reformatted your NameNode somewhere along the way but
    did not reformat your DN. You'll need to blow away your current
    dfs.data.dirs -- see your hdfs-site.xml to see where they're set to.
    -Todd
    On Wed, Mar 28, 2012 at 12:47 PM, Kevin wrote:
    Hello,
    I decided to change my pseudo distributed cluster to use YARN. I'm
    already running CDH4b1. I followed the steps provided here
    but unfortunately I am unable to start the data node. When I attempt
    to start the data node I receive the message stating, "datanode dead
    but pid file exists." I am unable to get HDFS up and running again.
    I can start the namenode, but when I start the datanode it logs a
    java.io.ioexception: incompatible clusterIDs...
    How do I synchronize the namenode and datanode cluster IDs?
    --
    Todd Lipcon
    Software Engineer, Cloudera
    --
    Harsh J
  • Priyanka Jaiswal at Sep 20, 2012 at 9:46 am
    Hi All

    Im getting a similar error while trying to run the sample wordcount program
    This is what get displayed on the screen:

    priyanka@priyanka-PC /usr/local/hadoop
    $ bin/hadoop jar share/hadoop/hadoop-examples-1.0.0.jar wordcount input
    output
    Warning: $HADOOP_HOME is deprecated.

    Patch for HADOOP-7682: Instantiating workaround file system
    12/09/20 13:38:03 ERROR security.UserGroupInformation:
    PriviledgedActionException as:priyanka cause:java.io.IOException: The
    ownership/permissions on the staging directory
    file:/tmp/hadoop-priyanka/mapred/staging/priyanka/.staging is not as
    expected. It is owned by priyanka and permissions are rwxr-xr-x. The
    directory must be owned by the submitter priyanka or by priyanka and
    permissions must be rwx------
    java.io.IOException: The ownership/permissions on the staging directory
    file:/tmp/hadoop-priyanka/mapred/staging/priyanka/.staging is not as
    expected. It is owned by priyanka and permissions are rwxr-xr-x. The
    directory must be owned by the submitter priyanka or by priyanka and
    permissions must be rwx------
    at
    org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:108)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:856)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
    at
    org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:465)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:495)
    at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
    at
    org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
    at
    org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    The permissions seem fine. Can't make out why Im still getting this error.

    priyanka@priyanka-PC /usr/local/hadoop
    $ ls -lart /tmp/hadoop-priyanka/mapred/staging/priyanka/
    total 0
    drwx------+ 1 priyanka None 0 Sep 20 13:23 ..
    drwx------+ 1 priyanka None 0 Sep 20 13:23 .staging
    drwx------+ 1 priyanka None 0 Sep 20 13:23 .

    Im using version 1.0.0

    Please help.

    -Priyanka
    On Thursday, 29 March 2012 19:08:16 UTC+5:30, Kevin wrote:

    Thanks for all the replies.

    I made a typo, Justin. I had to type the error message from the
    terminal and I lost the "v" in "kevin."

    Harsh, I tried everything you said and I keep getting the same error
    message - even disabling permissions doesn't resolve the issue (that
    concerns me). (But I think I know why: read below.)

    After executing the commands that Harsh suggested, I changed the
    permissions to remove the sticky bit and everything passes now ["sudo -
    u hdfs hadoop fs -chmod 0700 /tmp/hadoop-yarn/staging/
    kevin/.staging"].

    Thank you, everyone, for all the help.


    On Mar 29, 12:06 am, Harsh J wrote:
    Justin's right. Did you change your permissions of the staging
    directory at some point? Or of the whole FS?

    Run:
    hdfs dfs -chown kevin:kevin /tmp/hadoop-yarn/staging/kevin/
    hdfs dfs -chmod 700 /tmp/hadoop-yarn/staging/kevin/

    You can also disable permissions if its a bother and you do not need
    it. Set "dfs.permissions.enabled" to false and restart NameNode(s).

    On Thu, Mar 29, 2012 at 3:31 AM, Justin Workman









    wrote:
    Looks to me the ownership is wrong. Your user id is being reported as
    "Kevin" and owner in the file system is "kein".
    Sent from my iPhone
    On Mar 28, 2012, at 3:31 PM, Kevin wrote:

    Thanks for replying Todd.
    I changed the clusterID in the datanode's VERSION file to the current
    clusterID in the namenode's current VERSION file. This seemed to fix
    that problem.
    [REF:https://issues.apache.org/jira/browse/HDFS-107]
    Unfortunately, and I think it's unrelated to the original problem, I
    am unable to run the hadoop example job. Hadoop returns with an error
    message
    ERROR security.UserGroupInformation: PriviledgedActionException as:
    kevin (auth:SIMPLE) cause: java.io.IOException: the ownership/
    permissions the staing directory /tmp/hadoop-yarn/staging/
    kevin/.staging is not as expected. It is owned by kein and
    permissions
    are rwxrwxrwt. The directory must be owned by the submitter kevin or
    by kevin and permissions must be rwx------
    It's a detailed message, but I'm can't seem to fix my permissions to
    work with Hadoop.
    On Mar 28, 4:21 pm, Todd Lipcon wrote:
    Hi Kevin,
    Sounds like you reformatted your NameNode somewhere along the way
    but
    did not reformat your DN. You'll need to blow away your current
    dfs.data.dirs -- see your hdfs-site.xml to see where they're set to.
    -Todd
    On Wed, Mar 28, 2012 at 12:47 PM, Kevin wrote:
    Hello,
    I decided to change my pseudo distributed cluster to use YARN. I'm
    already running CDH4b1. I followed the steps provided here
    https://ccp.cloudera.com/display/CDH4B1/Installing+CDH4+on+a+Single+L...
    but unfortunately I am unable to start the data node. When I
    attempt
    to start the data node I receive the message stating, "datanode
    dead
    but pid file exists." I am unable to get HDFS up and running again.
    I can start the namenode, but when I start the datanode it logs a
    java.io.ioexception: incompatible clusterIDs...
    How do I synchronize the namenode and datanode cluster IDs?
    --
    Todd Lipcon
    Software Engineer, Cloudera
    --
    Harsh J
    --
  • Aaron T. Myers at Sep 27, 2012 at 7:43 pm
    Hi Priyanka,

    On Thu, Sep 20, 2012 at 2:46 AM, Priyanka Jaiswal
    wrote:
    The permissions seem fine. Can't make out why Im still getting this error.

    priyanka@priyanka-PC /usr/local/hadoop
    $ ls -lart /tmp/hadoop-priyanka/mapred/staging/priyanka/
    total 0
    drwx------+ 1 priyanka None 0 Sep 20 13:23 ..
    drwx------+ 1 priyanka None 0 Sep 20 13:23 .staging
    drwx------+ 1 priyanka None 0 Sep 20 13:23 .
    This output looks like it's coming from regular `ls' and is displaying
    information from the local file system, not HDFS. That error message is
    referring to permissions problems in HDFS. What happens if you run the
    following:

    `hadoop fs -ls hdfs:///tmp/hadoop-priyanka/mapred/staging/priyanka/'

    Im using version 1.0.0
    Where did you get this version of Hadoop? Are you attempting to run Apache
    Hadoop 1.0.0, with other CDH components? If so, I recommend you switch to
    using CDH 3u5 or CDH 4.0.1 in their entirety.

    --
    Aaron T. Myers
    Software Engineer, Cloudera

    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcdh-user @
categorieshadoop
postedMar 28, '12 at 7:48p
activeSep 27, '12 at 7:43p
posts9
users6
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase