FAQ
Hello,

After using the cloudera manager to enable kerberos security for hdfs.
and then deploying client configuration.

When i start the service , it works fine for 6min.
i can see the ticket request in the kdc log.
And hdfs starts.

but it says it does not have enough data to test if the namenode is active.

after 6min it shows kritical in cloudera manager, while looking at the
instances of hdfs it is all green.

When i look inside the log files i get the following error :

Can anyone help me ?

2013-10-03 15:54:38,573 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 8022: readAndProcess threw exception
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: Failure unspecified at GSS-API level (Mechanism level:
Encryption type AES256 CTS mode with HMAC SHA1-96 is not
supported/enabled)] from client 10.3.78.51. Count of bytes read: 0
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: Failure unspecified at GSS-API level (Mechanism level:
Encryption type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled)]
     at
com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:159)
     at
org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1249)
     at
org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1455)
     at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:758)
     at
org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:557)
     at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:532)
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism
level: Encryption type AES256 CTS mode with HMAC SHA1-96 is not
supported/enabled)
     at
sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:741)
     at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:323)
     at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:267)
     at
com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:137)
     ... 5 more
Caused by: KrbException: Encryption type AES256 CTS mode with HMAC SHA1-96
is not supported/enabled
     at sun.security.krb5.EncryptionKey.findKey(EncryptionKey.java:481)
     at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:260)
     at sun.security.krb5.KrbApReq.(InitSecContextToken.java:79)
     at
sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:724)
     ... 8 more
2013-10-03 15:54:38,796 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode: RECEIVED SIGNAL 15: SIGTERM
2013-10-03 15:54:38,801 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:

To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Search Discussions

  • Todd Grayson at Oct 3, 2013 at 2:11 pm
    Did you follow this discussion during your setup; the JCE policy files
    are needed for the cluster's JDK install if you are using AES 256

    http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/latest/Configuring-Hadoop-Security-with-Cloudera-Manager/cmchs_JCE_policy_s4.html

    On 10/3/13 8:06 AM, Bert Cauwelier wrote:
    Hello,

    After using the cloudera manager to enable kerberos security for hdfs.
    and then deploying client configuration.

    When i start the service , it works fine for 6min.
    i can see the ticket request in the kdc log.
    And hdfs starts.

    but it says it does not have enough data to test if the namenode is
    active.

    after 6min it shows kritical in cloudera manager, while looking at the
    instances of hdfs it is all green.

    When i look inside the log files i get the following error :

    Can anyone help me ?

    2013-10-03 15:54:38,573 INFO org.apache.hadoop.ipc.Server: IPC Server
    listener on 8022: readAndProcess threw exception
    javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: Failure unspecified at GSS-API level (Mechanism level:
    Encryption type AES256 CTS mode with HMAC SHA1-96 is not
    supported/enabled)] from client 10.3.78.51. Count of bytes read: 0
    javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: Failure unspecified at GSS-API level (Mechanism level:
    Encryption type AES256 CTS mode with HMAC SHA1-96 is not
    supported/enabled)]
    at
    com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:159)
    at
    org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1249)
    at
    org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1455)
    at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:758)
    at
    org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:557)
    at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:532)
    Caused by: GSSException: Failure unspecified at GSS-API level
    (Mechanism level: Encryption type AES256 CTS mode with HMAC SHA1-96 is
    not supported/enabled)
    at
    sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:741)
    at
    sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:323)
    at
    sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:267)
    at
    com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:137)
    ... 5 more
    Caused by: KrbException: Encryption type AES256 CTS mode with HMAC
    SHA1-96 is not supported/enabled
    at sun.security.krb5.EncryptionKey.findKey(EncryptionKey.java:481)
    at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:260)
    at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:134)
    at
    sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:79)
    at
    sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:724)
    ... 8 more
    2013-10-03 15:54:38,796 ERROR
    org.apache.hadoop.hdfs.server.namenode.NameNode: RECEIVED SIGNAL 15:
    SIGTERM
    2013-10-03 15:54:38,801 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:

    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Bert Cauwelier at Oct 3, 2013 at 2:32 pm
    I have set the correct JCE policy's . I runned the kerberos security
    inspector and it also checks that java unlimited encryption is correct.

    The error i gave earlier has been fixt thanks to the JCE policy.


    But after staring the hdfs1 with cloudera manager.
    It starts fine, but you can see in "health tests" = not enough data to
    test if namenode is active.
    everything keeps in the green. and the health test's still says "not
    enough data to test ...
    But after 3 to 6min. that health check fails.
    And it goes red, critial , telling (see attachement)






    2013/10/3 Todd Grayson <tgrayson@cloudera.com>
    Did you follow this discussion during your setup; the JCE policy files are
    needed for the cluster's JDK install if you are using AES 256

    http://www.cloudera.com/**content/cloudera-content/**
    cloudera-docs/CM4Ent/latest/**Configuring-Hadoop-Security-**
    with-Cloudera-Manager/cmchs_**JCE_policy_s4.html<http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/latest/Configuring-Hadoop-Security-with-Cloudera-Manager/cmchs_JCE_policy_s4.html>


    On 10/3/13 8:06 AM, Bert Cauwelier wrote:

    Hello,

    After using the cloudera manager to enable kerberos security for hdfs.
    and then deploying client configuration.

    When i start the service , it works fine for 6min.
    i can see the ticket request in the kdc log.
    And hdfs starts.

    but it says it does not have enough data to test if the namenode is
    active.

    after 6min it shows kritical in cloudera manager, while looking at the
    instances of hdfs it is all green.

    When i look inside the log files i get the following error :

    Can anyone help me ?

    2013-10-03 15:54:38,573 INFO org.apache.hadoop.ipc.Server: IPC Server
    listener on 8022: readAndProcess threw exception javax.security.sasl.**SaslException:
    GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API
    level (Mechanism level: Encryption type AES256 CTS mode with HMAC SHA1-96
    is not supported/enabled)] from client 10.3.78.51. Count of bytes read: 0
    javax.security.sasl.**SaslException: GSS initiate failed [Caused by
    GSSException: Failure unspecified at GSS-API level (Mechanism level:
    Encryption type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled)]
    at com.sun.security.sasl.gsskerb.**GssKrb5Server.**evaluateResponse(*
    *GssKrb5Server.java:159)
    at org.apache.hadoop.ipc.Server$**Connection.saslReadAndProcess(**
    Server.java:1249)
    at org.apache.hadoop.ipc.Server$**Connection.readAndProcess(**
    Server.java:1455)
    at org.apache.hadoop.ipc.Server$**Listener.doRead(Server.java:**758)
    at org.apache.hadoop.ipc.Server$**Listener$Reader.doRunLoop(**
    Server.java:557)
    at org.apache.hadoop.ipc.Server$**Listener$Reader.run(Server.**
    java:532)
    Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism
    level: Encryption type AES256 CTS mode with HMAC SHA1-96 is not
    supported/enabled)
    at sun.security.jgss.krb5.**Krb5Context.acceptSecContext(**
    Krb5Context.java:741)
    at sun.security.jgss.**GSSContextImpl.**acceptSecContext(**
    GSSContextImpl.java:323)
    at sun.security.jgss.**GSSContextImpl.**acceptSecContext(**
    GSSContextImpl.java:267)
    at com.sun.security.sasl.gsskerb.**GssKrb5Server.**evaluateResponse(*
    *GssKrb5Server.java:137)
    ... 5 more
    Caused by: KrbException: Encryption type AES256 CTS mode with HMAC
    SHA1-96 is not supported/enabled
    at sun.security.krb5.**EncryptionKey.findKey(**
    EncryptionKey.java:481)
    at sun.security.krb5.KrbApReq.**authenticate(KrbApReq.java:**260)
    at sun.security.krb5.KrbApReq.<**init>(KrbApReq.java:134)
    at sun.security.jgss.krb5.**InitSecContextToken.<init>(**
    InitSecContextToken.java:79)
    at sun.security.jgss.krb5.**Krb5Context.acceptSecContext(**
    Krb5Context.java:724)
    ... 8 more
    2013-10-03 15:54:38,796 ERROR org.apache.hadoop.hdfs.server.**namenode.NameNode:
    RECEIVED SIGNAL 15: SIGTERM
    2013-10-03 15:54:38,801 INFO org.apache.hadoop.hdfs.server.**namenode.NameNode:
    SHUTDOWN_MSG:

    To unsubscribe from this group and stop receiving emails from it, send an
    email to scm-users+unsubscribe@**cloudera.org<scm-users%2Bunsubscribe@cloudera.org>
    .
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Todd Grayson at Oct 3, 2013 at 5:37 pm
    What do you see when you attempt to connect to the namenode web UI under
    the "services > HDFS" page?.

    Check the /var/log/hadoop-hdfs/hadoop-cmf-hdfs1-NAMENODE-[YOUR
    HOSTNAME].log.out

    Are you able to run hadoop fs command lines?

    If you are kerberized at this point in your configurationand have cross
    realm auth established with AD from your cluster

    kinit -p someuseryoucreated@your.ad.realm

    hadoop fs -ls /

    does this throw an exception?


    Todd

    On 10/3/13 8:32 AM, Bert Cauwelier wrote:
    I have set the correct JCE policy's . I runned the kerberos security
    inspector and it also checks that java unlimited encryption is correct.

    The error i gave earlier has been fixt thanks to the JCE policy.


    But after staring the hdfs1 with cloudera manager.
    It starts fine, but you can see in "health tests" = not enough data
    to test if namenode is active.
    everything keeps in the green. and the health test's still says "not
    enough data to test ...
    But after 3 to 6min. that health check fails.
    And it goes red, critial , telling (see attachement)






    2013/10/3 Todd Grayson <tgrayson@cloudera.com

    Did you follow this discussion during your setup; the JCE policy
    files are needed for the cluster's JDK install if you are using
    AES 256

    http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/latest/Configuring-Hadoop-Security-with-Cloudera-Manager/cmchs_JCE_policy_s4.html




    On 10/3/13 8:06 AM, Bert Cauwelier wrote:

    Hello,

    After using the cloudera manager to enable kerberos security
    for hdfs.
    and then deploying client configuration.

    When i start the service , it works fine for 6min.
    i can see the ticket request in the kdc log.
    And hdfs starts.

    but it says it does not have enough data to test if the
    namenode is active.

    after 6min it shows kritical in cloudera manager, while
    looking at the instances of hdfs it is all green.

    When i look inside the log files i get the following error :

    Can anyone help me ?

    2013-10-03 15:54:38,573 INFO org.apache.hadoop.ipc.Server: IPC
    Server listener on 8022: readAndProcess threw exception
    javax.security.sasl.SaslException: GSS initiate failed [Caused
    by GSSException: Failure unspecified at GSS-API level
    (Mechanism level: Encryption type AES256 CTS mode with HMAC
    SHA1-96 is not supported/enabled)] from client 10.3.78.51.
    Count of bytes read: 0
    javax.security.sasl.SaslException: GSS initiate failed [Caused
    by GSSException: Failure unspecified at GSS-API level
    (Mechanism level: Encryption type AES256 CTS mode with HMAC
    SHA1-96 is not supported/enabled)]
    at
    com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:159)
    at
    org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1249)
    at
    org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1455)
    at
    org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:758)
    at
    org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:557)
    at
    org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:532)
    Caused by: GSSException: Failure unspecified at GSS-API level
    (Mechanism level: Encryption type AES256 CTS mode with HMAC
    SHA1-96 is not supported/enabled)
    at
    sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:741)
    at
    sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:323)
    at
    sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:267)
    at
    com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:137)
    ... 5 more
    Caused by: KrbException: Encryption type AES256 CTS mode with
    HMAC SHA1-96 is not supported/enabled
    at
    sun.security.krb5.EncryptionKey.findKey(EncryptionKey.java:481)
    at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:260)
    at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:134)
    at
    sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:79)
    at
    sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:724)
    ... 8 more
    2013-10-03 15:54:38,796 ERROR
    org.apache.hadoop.hdfs.server.namenode.NameNode: RECEIVED
    SIGNAL 15: SIGTERM
    2013-10-03 15:54:38,801 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:

    To unsubscribe from this group and stop receiving emails from
    it, send an email to scm-users+unsubscribe@cloudera.org

    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Bert Cauwelier at Oct 4, 2013 at 11:33 am
    Todd,

    On the HDFS page it says the system is healty
    when using the command "hdfs hdfs fsck /" it also says the file system is
    healty.
    when i check te log. there are no errors.
    I am able to use the hadoop command lines

    After using kinit , to gain a ticket for the hdfs user.
    i can succesfully look into hdfs.

    But still the health check on cloudera manager fails.
    It says it can not verify the connection with the namenode.
    Also , cloudera manager says it can not find the jobtracker.

    So i think it is purely the health check that fails.
    do you know of any answer of possible problems why cloudera manager keeps
    telling that these checks fail ?



    Op donderdag 3 oktober 2013 19:37:10 UTC+2 schreef Todd Grayson:
    What do you see when you attempt to connect to the namenode web UI under
    the "services > HDFS" page?.

    Check the /var/log/hadoop-hdfs/hadoop-cmf-hdfs1-NAMENODE-[YOUR
    HOSTNAME].log.out

    Are you able to run hadoop fs command lines?

    If you are kerberized at this point in your configurationand have cross
    realm auth established with AD from your cluster

    kinit -p someusery...@YOUR.AD.REALM <javascript:>

    hadoop fs -ls /

    does this throw an exception?


    Todd


    On 10/3/13 8:32 AM, Bert Cauwelier wrote:

    I have set the correct JCE policy's . I runned the kerberos security
    inspector and it also checks that java unlimited encryption is correct.

    The error i gave earlier has been fixt thanks to the JCE policy.


    But after staring the hdfs1 with cloudera manager.
    It starts fine, but you can see in "health tests" = not enough data to
    test if namenode is active.
    everything keeps in the green. and the health test's still says "not
    enough data to test ...
    But after 3 to 6min. that health check fails.
    And it goes red, critial , telling (see attachement)






    2013/10/3 Todd Grayson <tgra...@cloudera.com <javascript:>>
    Did you follow this discussion during your setup; the JCE policy files
    are needed for the cluster's JDK install if you are using AES 256


    http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/latest/Configuring-Hadoop-Security-with-Cloudera-Manager/cmchs_JCE_policy_s4.html


    On 10/3/13 8:06 AM, Bert Cauwelier wrote:

    Hello,

    After using the cloudera manager to enable kerberos security for hdfs.
    and then deploying client configuration.

    When i start the service , it works fine for 6min.
    i can see the ticket request in the kdc log.
    And hdfs starts.

    but it says it does not have enough data to test if the namenode is
    active.

    after 6min it shows kritical in cloudera manager, while looking at the
    instances of hdfs it is all green.

    When i look inside the log files i get the following error :

    Can anyone help me ?

    2013-10-03 15:54:38,573 INFO org.apache.hadoop.ipc.Server: IPC Server
    listener on 8022: readAndProcess threw exception
    javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: Failure unspecified at GSS-API level (Mechanism level:
    Encryption type AES256 CTS mode with HMAC SHA1-96 is not
    supported/enabled)] from client 10.3.78.51. Count of bytes read: 0
    javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: Failure unspecified at GSS-API level (Mechanism level:
    Encryption type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled)]
    at
    com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:159)
    at
    org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1249)
    at
    org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1455)
    at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:758)
    at
    org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:557)
    at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:532)
    Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism
    level: Encryption type AES256 CTS mode with HMAC SHA1-96 is not
    supported/enabled)
    at
    sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:741)
    at
    sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:323)
    at
    sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:267)
    at
    com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:137)
    ... 5 more
    Caused by: KrbException: Encryption type AES256 CTS mode with HMAC
    SHA1-96 is not supported/enabled
    at sun.security.krb5.EncryptionKey.findKey(EncryptionKey.java:481)
    at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:260)
    at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:134)
    at
    sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:79)
    at
    sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:724)
    ... 8 more
    2013-10-03 15:54:38,796 ERROR
    org.apache.hadoop.hdfs.server.namenode.NameNode: RECEIVED SIGNAL 15: SIGTERM
    2013-10-03 15:54:38,801 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:

    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+...@cloudera.org <javascript:>.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedOct 3, '13 at 2:06p
activeOct 4, '13 at 11:33a
posts5
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase