FAQ
Hi,
after configuring the cluster to be HA, I am required to format the
namenode but the process fails and I get the following msg error:

Re-format filesystem in Storage Directory /data/1/dfs/nn ? (Y or N) Y
Re-format filesystem in Storage Directory /data/2/dfs/nn ? (Y or N) Y
14/03/20 19:30:27 ERROR security.UserGroupInformation:
PriviledgedActionException as:hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Server not found in Kerberos database (7))]
14/03/20 19:30:30 ERROR security.UserGroupInformation:
PriviledgedActionException as:hdfs/hadoopdev11,my.domain.com@MY.DOMAIN.COM
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Server not found in Kerberos database (7))]
14/03/20 19:30:33 ERROR security.UserGroupInformation:
PriviledgedActionException as:hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Server not found in Kerberos database (7))]
14/03/20 19:30:36 ERROR security.UserGroupInformation:
PriviledgedActionException as:hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Server not found in Kerberos database (7))]
14/03/20 19:30:37 ERROR security.UserGroupInformation:
PriviledgedActionException as:hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Server not found in Kerberos database (7))]
14/03/20 19:30:39 ERROR security.UserGroupInformation:
PriviledgedActionException as:hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Server not found in Kerberos database (7))]
14/03/20 19:30:39 WARN ipc.Client: Couldn't setup connection for
hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM to
hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM
14/03/20 19:30:39 ERROR security.UserGroupInformation:
PriviledgedActionException as:hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM
(auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for
hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM to
hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM
14/03/20 19:30:39 FATAL namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Unable to check if
JNs are ready for formatting. 2 successful responses:
10.198.16.38:8485: false
10.198.16.68:8485: false
1 exceptions thrown:
10.198.16.43:8485: Failed on local exception: java.io.IOException: Couldn't
setup connection for hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM to
hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM; Host Details : local host is:
"hadoopdev11.my.domain.com/10.198.16.43"; destination host is:
"hadoopdev11.my.domain.com":8485;
     at
org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
     at
org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:223)
     at
org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.hasSomeData(QuorumJournalManager.java:218)
     at
org.apache.hadoop.hdfs.server.common.Storage.confirmFormat(Storage.java:831)
     at
org.apache.hadoop.hdfs.server.namenode.FSImage.confirmFormat(FSImage.java:175)
     at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:756)
     at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1128)
     at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1233)
14/03/20 19:30:39 INFO util.ExitUtil: Exiting with status 1
14/03/20 19:30:39 INFO namenode.NameNode: SHUTDOWN_MSG:

The journalnodes they all three start up correctly but as soon as I run the
command on one of the namenode to do the format I can see in the logs of
the JN the following msg:

2014-03-20 20:01:23,748 WARN SecurityLogger.org.apache.hadoop.ipc.Server:
Auth failed for 10.198.16.43:52701:null (GSS initiate failed)
2014-03-20 20:01:23,748 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 8485: readAndProcess threw exception
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: Failure unspecified at GSS-API level (Mechanism level:
Specified version of key is not available (44))] from client 10.198.16.43.
Count of bytes read: 0
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: Failure unspecified at GSS-API level (Mechanism level:
Specified version of key is not available (44))]
     at
com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:177)
     at
org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1250)
     at
org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1456)
     at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:759)
     at
org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:557)
     at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:532)
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism
level: Specified version of key is not available (44))
     at
sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:788)
     at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
     at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
     at
com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:155)
     ... 5 more
Caused by: KrbException: Specified version of key is not available (44)
     at sun.security.krb5.EncryptionKey.findKey(EncryptionKey.java:588)
     at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:270)
     at sun.security.krb5.KrbApReq.(InitSecContextToken.java:108)
     at
sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771)
     ... 8 more

Before when I had configured single namenode with secondary namenode it was
running fine.



Any idea?
Thanks
J

To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Search Discussions

  • Vikas Singh at Mar 21, 2014 at 1:15 am
    Seems like you kerberos keys are messed up. Try running "Generate
    Credentials" command from "Adminstrator->Kerberos" menu. That will
    regenerate fresh keys and distribute keytab files and this issue should go
    away then.

    - Vikas

    On Thu, Mar 20, 2014 at 6:07 PM, MrAkhe83 wrote:

    Hi,
    after configuring the cluster to be HA, I am required to format the
    namenode but the process fails and I get the following msg error:

    Re-format filesystem in Storage Directory /data/1/dfs/nn ? (Y or N) Y
    Re-format filesystem in Storage Directory /data/2/dfs/nn ? (Y or N) Y
    14/03/20 19:30:27 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate
    failed [Caused by GSSException: No valid credentials provided (Mechanism
    level: Server not found in Kerberos database (7))]
    14/03/20 19:30:30 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/hadoopdev11,my.domain.com@MY.DOMAIN.COM(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate
    failed [Caused by GSSException: No valid credentials provided (Mechanism
    level: Server not found in Kerberos database (7))]
    14/03/20 19:30:33 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate
    failed [Caused by GSSException: No valid credentials provided (Mechanism
    level: Server not found in Kerberos database (7))]
    14/03/20 19:30:36 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate
    failed [Caused by GSSException: No valid credentials provided (Mechanism
    level: Server not found in Kerberos database (7))]
    14/03/20 19:30:37 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate
    failed [Caused by GSSException: No valid credentials provided (Mechanism
    level: Server not found in Kerberos database (7))]
    14/03/20 19:30:39 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate
    failed [Caused by GSSException: No valid credentials provided (Mechanism
    level: Server not found in Kerberos database (7))]
    14/03/20 19:30:39 WARN ipc.Client: Couldn't setup connection for hdfs/
    hadoopdev11.my.domain.com@my.domain.com to hdfs/
    hadoopdev11.my.domain.com@my.domain.com
    14/03/20 19:30:39 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM(auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for
    hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM to hdfs/
    hadoopdev11.my.domain.com@my.domain.com
    14/03/20 19:30:39 FATAL namenode.NameNode: Exception in namenode join
    org.apache.hadoop.hdfs.qjournal.client.QuorumException: Unable to check if
    JNs are ready for formatting. 2 successful responses:
    10.198.16.38:8485: false
    10.198.16.68:8485: false
    1 exceptions thrown:
    10.198.16.43:8485: Failed on local exception: java.io.IOException:
    Couldn't setup connection for hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COMto hdfs/
    hadoopdev11.my.domain.com@my.domain.com; Host Details : local host is: "
    hadoopdev11.my.domain.com/10.198.16.43"; destination host is: "
    hadoopdev11.my.domain.com":8485;
    at
    org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
    at
    org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:223)
    at
    org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.hasSomeData(QuorumJournalManager.java:218)
    at
    org.apache.hadoop.hdfs.server.common.Storage.confirmFormat(Storage.java:831)
    at
    org.apache.hadoop.hdfs.server.namenode.FSImage.confirmFormat(FSImage.java:175)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:756)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1128)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1233)
    14/03/20 19:30:39 INFO util.ExitUtil: Exiting with status 1
    14/03/20 19:30:39 INFO namenode.NameNode: SHUTDOWN_MSG:

    The journalnodes they all three start up correctly but as soon as I run
    the command on one of the namenode to do the format I can see in the logs
    of the JN the following msg:

    2014-03-20 20:01:23,748 WARN SecurityLogger.org.apache.hadoop.ipc.Server:
    Auth failed for 10.198.16.43:52701:null (GSS initiate failed)
    2014-03-20 20:01:23,748 INFO org.apache.hadoop.ipc.Server: IPC Server
    listener on 8485: readAndProcess threw exception
    javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: Failure unspecified at GSS-API level (Mechanism level:
    Specified version of key is not available (44))] from client 10.198.16.43.
    Count of bytes read: 0
    javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: Failure unspecified at GSS-API level (Mechanism level:
    Specified version of key is not available (44))]
    at
    com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:177)
    at
    org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1250)
    at
    org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1456)
    at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:759)
    at
    org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:557)
    at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:532)
    Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism
    level: Specified version of key is not available (44))
    at
    sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:788)
    at
    sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
    at
    sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
    at
    com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:155)
    ... 5 more
    Caused by: KrbException: Specified version of key is not available (44)
    at sun.security.krb5.EncryptionKey.findKey(EncryptionKey.java:588)
    at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:270)
    at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:144)
    at
    sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:108)
    at
    sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771)
    ... 8 more

    Before when I had configured single namenode with secondary namenode it
    was running fine.



    Any idea?
    Thanks
    J

    To unsubscribe from this group and stop receiving emails from it, send an
    email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • MrAkhe83 at Mar 21, 2014 at 5:01 pm
    Sorry I think i posted this msg in the wrong group.
    I am now actually building a cluster via CDH not cloudera manager.

    J
    On Thursday, March 20, 2014 8:15:31 PM UTC-5, Vikas Singh wrote:

    Seems like you kerberos keys are messed up. Try running "Generate
    Credentials" command from "Adminstrator->Kerberos" menu. That will
    regenerate fresh keys and distribute keytab files and this issue should go
    away then.

    - Vikas


    On Thu, Mar 20, 2014 at 6:07 PM, MrAkhe83 <jla...@gmail.com <javascript:>>wrote:
    Hi,
    after configuring the cluster to be HA, I am required to format the
    namenode but the process fails and I get the following msg error:

    Re-format filesystem in Storage Directory /data/1/dfs/nn ? (Y or N) Y
    Re-format filesystem in Storage Directory /data/2/dfs/nn ? (Y or N) Y
    14/03/20 19:30:27 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/
    hadoopdev11.my.domain.com@my.domain.com <javascript:> (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Server not
    found in Kerberos database (7))]
    14/03/20 19:30:30 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/hadoopdev11,
    my.domain.com@my.domain.com <javascript:> (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Server not
    found in Kerberos database (7))]
    14/03/20 19:30:33 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/
    hadoopdev11.my.domain.com@my.domain.com <javascript:> (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Server not
    found in Kerberos database (7))]
    14/03/20 19:30:36 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/
    hadoopdev11.my.domain.com@my.domain.com <javascript:> (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Server not
    found in Kerberos database (7))]
    14/03/20 19:30:37 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/
    hadoopdev11.my.domain.com@my.domain.com <javascript:> (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Server not
    found in Kerberos database (7))]
    14/03/20 19:30:39 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/
    hadoopdev11.my.domain.com@my.domain.com <javascript:> (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Server not
    found in Kerberos database (7))]
    14/03/20 19:30:39 WARN ipc.Client: Couldn't setup connection for hdfs/
    hadoopdev11.my.domain.com@my.domain.com <javascript:> to hdfs/
    hadoopdev11.my.domain.com@my.domain.com <javascript:>
    14/03/20 19:30:39 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/
    hadoopdev11.my.domain.com@my.domain.com <javascript:> (auth:KERBEROS)
    cause:java.io.IOException: Couldn't setup connection for hdfs/
    hadoopdev11.my.domain.com@my.domain.com <javascript:> to hdfs/
    hadoopdev11.my.domain.com@my.domain.com <javascript:>
    14/03/20 19:30:39 FATAL namenode.NameNode: Exception in namenode join
    org.apache.hadoop.hdfs.qjournal.client.QuorumException: Unable to check
    if JNs are ready for formatting. 2 successful responses:
    10.198.16.38:8485: false
    10.198.16.68:8485: false
    1 exceptions thrown:
    10.198.16.43:8485: Failed on local exception: java.io.IOException:
    Couldn't setup connection for hdfs/
    hadoopdev11.my.domain.com@my.domain.com <javascript:> to hdfs/
    hadoopdev11.my.domain.com@my.domain.com <javascript:>; Host Details :
    local host is: "hadoopdev11.my.domain.com/10.198.16.43"; destination
    host is: "hadoopdev11.my.domain.com":8485;
    at
    org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
    at
    org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:223)
    at
    org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.hasSomeData(QuorumJournalManager.java:218)
    at
    org.apache.hadoop.hdfs.server.common.Storage.confirmFormat(Storage.java:831)
    at
    org.apache.hadoop.hdfs.server.namenode.FSImage.confirmFormat(FSImage.java:175)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:756)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1128)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1233)
    14/03/20 19:30:39 INFO util.ExitUtil: Exiting with status 1
    14/03/20 19:30:39 INFO namenode.NameNode: SHUTDOWN_MSG:

    The journalnodes they all three start up correctly but as soon as I run
    the command on one of the namenode to do the format I can see in the logs
    of the JN the following msg:

    2014-03-20 20:01:23,748 WARN SecurityLogger.org.apache.hadoop.ipc.Server:
    Auth failed for 10.198.16.43:52701:null (GSS initiate failed)
    2014-03-20 20:01:23,748 INFO org.apache.hadoop.ipc.Server: IPC Server
    listener on 8485: readAndProcess threw exception
    javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: Failure unspecified at GSS-API level (Mechanism level:
    Specified version of key is not available (44))] from client 10.198.16.43.
    Count of bytes read: 0
    javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: Failure unspecified at GSS-API level (Mechanism level:
    Specified version of key is not available (44))]
    at
    com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:177)
    at
    org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1250)
    at
    org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1456)
    at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:759)
    at
    org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:557)
    at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:532)
    Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism
    level: Specified version of key is not available (44))
    at
    sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:788)
    at
    sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
    at
    sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
    at
    com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:155)
    ... 5 more
    Caused by: KrbException: Specified version of key is not available (44)
    at sun.security.krb5.EncryptionKey.findKey(EncryptionKey.java:588)
    at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:270)
    at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:144)
    at
    sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:108)
    at
    sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771)
    ... 8 more

    Before when I had configured single namenode with secondary namenode it
    was running fine.



    Any idea?
    Thanks
    J

    To unsubscribe from this group and stop receiving emails from it, send an
    email to scm-users+...@cloudera.org <javascript:>.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Vikas Singh at Mar 21, 2014 at 5:57 pm
    No worries, cdh-users is the correct list for CDH questions.

    In any case, from the error it seems that you have made updates to the
    account in kerberos (probably changed password). That bumps up the key
    version number (kvno) for the key. It seems that the key you have in your
    keytab file has older version number. You need to regenerate your keytab
    file. That should take care of this issue.

    And lastly, CM is a wonderful product, if possible give it a try. It will
    take care of these issues for you.

    - Vikas

    On Fri, Mar 21, 2014 at 10:01 AM, MrAkhe83 wrote:

    Sorry I think i posted this msg in the wrong group.
    I am now actually building a cluster via CDH not cloudera manager.

    J

    On Thursday, March 20, 2014 8:15:31 PM UTC-5, Vikas Singh wrote:

    Seems like you kerberos keys are messed up. Try running "Generate
    Credentials" command from "Adminstrator->Kerberos" menu. That will
    regenerate fresh keys and distribute keytab files and this issue should go
    away then.

    - Vikas

    On Thu, Mar 20, 2014 at 6:07 PM, MrAkhe83 wrote:

    Hi,
    after configuring the cluster to be HA, I am required to format the
    namenode but the process fails and I get the following msg error:

    Re-format filesystem in Storage Directory /data/1/dfs/nn ? (Y or N) Y
    Re-format filesystem in Storage Directory /data/2/dfs/nn ? (Y or N) Y
    14/03/20 19:30:27 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/hadoopdev11.my.domain.
    com@my.domain.com (auth:KERBEROS) cause:javax.security.sasl.SaslException:
    GSS initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Server not found in Kerberos database (7))]
    14/03/20 19:30:30 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/hadoopdev11,my.domain.
    com@my.domain.com (auth:KERBEROS) cause:javax.security.sasl.SaslException:
    GSS initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Server not found in Kerberos database (7))]
    14/03/20 19:30:33 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/hadoopdev11.my.domain.
    com@my.domain.com (auth:KERBEROS) cause:javax.security.sasl.SaslException:
    GSS initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Server not found in Kerberos database (7))]
    14/03/20 19:30:36 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/hadoopdev11.my.domain.
    com@my.domain.com (auth:KERBEROS) cause:javax.security.sasl.SaslException:
    GSS initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Server not found in Kerberos database (7))]
    14/03/20 19:30:37 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/hadoopdev11.my.domain.
    com@my.domain.com (auth:KERBEROS) cause:javax.security.sasl.SaslException:
    GSS initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Server not found in Kerberos database (7))]
    14/03/20 19:30:39 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/hadoopdev11.my.domain.
    com@my.domain.com (auth:KERBEROS) cause:javax.security.sasl.SaslException:
    GSS initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Server not found in Kerberos database (7))]
    14/03/20 19:30:39 WARN ipc.Client: Couldn't setup connection for hdfs/
    hadoopdev11.my.domain.com@my.domain.com to hdfs/hadoopdev11.my.domain.
    com@my.domain.com
    14/03/20 19:30:39 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs/hadoopdev11.my.domain.
    com@my.domain.com (auth:KERBEROS) cause:java.io.IOException: Couldn't
    setup connection for hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM to
    hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM
    14/03/20 19:30:39 FATAL namenode.NameNode: Exception in namenode join
    org.apache.hadoop.hdfs.qjournal.client.QuorumException: Unable to check
    if JNs are ready for formatting. 2 successful responses:
    10.198.16.38:8485: false
    10.198.16.68:8485: false
    1 exceptions thrown:
    10.198.16.43:8485: Failed on local exception: java.io.IOException:
    Couldn't setup connection for hdfs/hadoopdev11.my.domain.
    com@my.domain.com to hdfs/hadoopdev11.my.domain.com@MY.DOMAIN.COM; Host
    Details : local host is: "hadoopdev11.my.domain.com/10.198.16.43";
    destination host is: "hadoopdev11.my.domain.com":8485;
    at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(
    QuorumException.java:81)
    at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.
    rethrowException(QuorumCall.java:223)
    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.
    hasSomeData(QuorumJournalManager.java:218)
    at org.apache.hadoop.hdfs.server.common.Storage.confirmFormat(
    Storage.java:831)
    at org.apache.hadoop.hdfs.server.namenode.FSImage.
    confirmFormat(FSImage.java:175)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.format(
    NameNode.java:756)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.
    createNameNode(NameNode.java:1128)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(
    NameNode.java:1233)
    14/03/20 19:30:39 INFO util.ExitUtil: Exiting with status 1
    14/03/20 19:30:39 INFO namenode.NameNode: SHUTDOWN_MSG:

    The journalnodes they all three start up correctly but as soon as I run
    the command on one of the namenode to do the format I can see in the logs
    of the JN the following msg:

    2014-03-20 20:01:23,748 WARN SecurityLogger.org.apache.hadoop.ipc.Server:
    Auth failed for 10.198.16.43:52701:null (GSS initiate failed)
    2014-03-20 20:01:23,748 INFO org.apache.hadoop.ipc.Server: IPC Server
    listener on 8485: readAndProcess threw exception javax.security.sasl.SaslException:
    GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API
    level (Mechanism level: Specified version of key is not available (44))]
    from client 10.198.16.43. Count of bytes read: 0
    javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: Failure unspecified at GSS-API level (Mechanism level:
    Specified version of key is not available (44))]
    at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(
    GssKrb5Server.java:177)
    at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(
    Server.java:1250)
    at org.apache.hadoop.ipc.Server$Connection.readAndProcess(
    Server.java:1456)
    at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:759)
    at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(
    Server.java:557)
    at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:532)
    Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism
    level: Specified version of key is not available (44))
    at sun.security.jgss.krb5.Krb5Context.acceptSecContext(
    Krb5Context.java:788)
    at sun.security.jgss.GSSContextImpl.acceptSecContext(
    GSSContextImpl.java:342)
    at sun.security.jgss.GSSContextImpl.acceptSecContext(
    GSSContextImpl.java:285)
    at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(
    GssKrb5Server.java:155)
    ... 5 more
    Caused by: KrbException: Specified version of key is not available (44)
    at sun.security.krb5.EncryptionKey.findKey(EncryptionKey.java:588)
    at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:270)
    at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:144)
    at sun.security.jgss.krb5.InitSecContextToken.<init>(
    InitSecContextToken.java:108)
    at sun.security.jgss.krb5.Krb5Context.acceptSecContext(
    Krb5Context.java:771)
    ... 8 more

    Before when I had configured single namenode with secondary namenode it
    was running fine.



    Any idea?
    Thanks
    J

    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedMar 21, '14 at 1:07a
activeMar 21, '14 at 5:57p
posts4
users2
websitecloudera.com
irc#hadoop

2 users in discussion

MrAkhe83: 2 posts Vikas Singh: 2 posts

People

Translate

site design / logo © 2022 Grokbase