Hi,
after configuring the cluster to be HA, I am required to format the
namenode but the process fails and I get the following msg error:
Re-format filesystem in Storage Directory /data/1/dfs/nn ? (Y or N) Y
Re-format filesystem in Storage Directory /data/2/dfs/nn ? (Y or N) Y
14/03/20 19:30:27 ERROR security.UserGroupInformation:
PriviledgedActionException as:hdfs/[email protected]
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Server not found in Kerberos database (7))]
14/03/20 19:30:30 ERROR security.UserGroupInformation:
PriviledgedActionException as:hdfs/hadoopdev11,[email protected]
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Server not found in Kerberos database (7))]
14/03/20 19:30:33 ERROR security.UserGroupInformation:
PriviledgedActionException as:hdfs/[email protected]
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Server not found in Kerberos database (7))]
14/03/20 19:30:36 ERROR security.UserGroupInformation:
PriviledgedActionException as:hdfs/[email protected]
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Server not found in Kerberos database (7))]
14/03/20 19:30:37 ERROR security.UserGroupInformation:
PriviledgedActionException as:hdfs/[email protected]
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Server not found in Kerberos database (7))]
14/03/20 19:30:39 ERROR security.UserGroupInformation:
PriviledgedActionException as:hdfs/[email protected]
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Server not found in Kerberos database (7))]
14/03/20 19:30:39 WARN ipc.Client: Couldn't setup connection for
hdfs/[email protected] to
hdfs/[email protected]
14/03/20 19:30:39 ERROR security.UserGroupInformation:
PriviledgedActionException as:hdfs/[email protected]
(auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for
hdfs/[email protected] to
hdfs/[email protected]
14/03/20 19:30:39 FATAL namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Unable to check if
JNs are ready for formatting. 2 successful responses:
10.198.16.38:8485: false
10.198.16.68:8485: false
1 exceptions thrown:
10.198.16.43:8485: Failed on local exception: java.io.IOException: Couldn't
setup connection for hdfs/[email protected] to
hdfs/hadoop[email protected]; Host Details : local host is:
"hadoopdev11.my.domain.com/10.198.16.43"; destination host is:
"hadoopdev11.my.domain.com":8485;
at
org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
at
org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:223)
at
org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.hasSomeData(QuorumJournalManager.java:218)
at
org.apache.hadoop.hdfs.server.common.Storage.confirmFormat(Storage.java:831)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.confirmFormat(FSImage.java:175)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:756)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1128)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1233)
14/03/20 19:30:39 INFO util.ExitUtil: Exiting with status 1
14/03/20 19:30:39 INFO namenode.NameNode: SHUTDOWN_MSG:
The journalnodes they all three start up correctly but as soon as I run the
command on one of the namenode to do the format I can see in the logs of
the JN the following msg:
2014-03-20 20:01:23,748 WARN SecurityLogger.org.apache.hadoop.ipc.Server:
Auth failed for 10.198.16.43:52701:null (GSS initiate failed)
2014-03-20 20:01:23,748 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 8485: readAndProcess threw exception
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: Failure unspecified at GSS-API level (Mechanism level:
Specified version of key is not available (44))] from client 10.198.16.43.
Count of bytes read: 0
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: Failure unspecified at GSS-API level (Mechanism level:
Specified version of key is not available (44))]
at
com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:177)
at
org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1250)
at
org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1456)
at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:759)
at
org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:557)
at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:532)
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism
level: Specified version of key is not available (44))
at
sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:788)
at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
at
com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:155)
... 5 more
Caused by: KrbException: Specified version of key is not available (44)
at sun.security.krb5.EncryptionKey.findKey(EncryptionKey.java:588)
at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:270)
at sun.security.krb5.KrbApReq.(InitSecContextToken.java:108)
at
sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771)
... 8 more
Before when I had configured single namenode with secondary namenode it was
running fine.
Any idea?
Thanks
J
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
