FAQ
Hi,
I have a cluster with kerberos enabled and all keytabs configured
correctly. Each credentials have been generated successfully.
All services (HDFS, mapred, oozie, impala, hive, zookeeper,...) are
starting up without any issues. HA is not enabled yet.
The problem is that when I try to browse hdfs I get an error related to
kerberos:

[hdfs@hadoopha02 ~]$ hadoop fs -ls /
14/04/24 12:04:43 ERROR security.UserGroupInformation:
PriviledgedActionException as:hdfs (auth:KERBEROS)
cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to
find any Kerberos tgt)]
14/04/24 12:04:43 WARN ipc.Client: Exception encountered while connecting
to the server : javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]
14/04/24 12:04:43 ERROR security.UserGroupInformation:
PriviledgedActionException as:hdfs (auth:KERBEROS)
cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Failed to find any Kerberos tgt)]
14/04/24 12:04:43 WARN retry.RetryInvocationHandler: Exception while
invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 1
fail over attempts. Trying to fail over after sleeping for 1395ms.
14/04/24 12:04:45 ERROR security.UserGroupInformation:
PriviledgedActionException as:hdfs (auth:KERBEROS)
cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to
find any Kerberos tgt)]
14/04/24 12:04:45 WARN ipc.Client: Exception encountered while connecting
to the server : javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]
14/04/24 12:04:45 ERROR security.UserGroupInformation:
PriviledgedActionException as:hdfs (auth:KERBEROS)
cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Failed to find any Kerberos tgt)]

The kinit has been executed:
[hdfs@hadoopha02 ~]$ kinit -kt
/var/run/cloudera-scm-agent/process/544-hdfs-NAMENODE/hdfs.keytab
hdfs/hadoopha02.domain.intra@DOMAIN.INTRA

and I can see this by doing klist command:
[hdfs@hadoopha02 ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_868271872
Default principal: hdfs/hadoopha02.domain.intra@DOMAIN.INTRA

Valid starting Expires Service principal
04/24/14 12:04:36 04/24/14 22:04:37 krbtgt/DOMAIN.INTRA@DOMAIN.INTRA
     renew until 04/25/14 12:04:31

[hdfs@hadoopha02 ~]$ id
uid=868271872(hdfs) gid=868271872(hdfs) groups=868271872(hdfs),493(hadoop)

Any idea? let me know if I need to send more info.

Thx


To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Search Discussions

  • Vikram Srivastava at Apr 24, 2014 at 6:00 pm
    You should not use the principals for Hadoop daemons. Instead you should
    create principals for Hadoop clients separately and kinit using that.
    Follow the instructions on:
    http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Configuring-Hadoop-Security-with-Cloudera-Manager/cm5chs_hdfs_su_princ_s15.html

    On Thu, Apr 24, 2014 at 10:52 AM, MrAkhe83 wrote:

    Hi,
    I have a cluster with kerberos enabled and all keytabs configured
    correctly. Each credentials have been generated successfully.
    All services (HDFS, mapred, oozie, impala, hive, zookeeper,...) are
    starting up without any issues. HA is not enabled yet.
    The problem is that when I try to browse hdfs I get an error related to
    kerberos:

    [hdfs@hadoopha02 ~]$ hadoop fs -ls /
    14/04/24 12:04:43 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 12:04:43 WARN ipc.Client: Exception encountered while connecting
    to the server : javax.security.sasl.SaslException: GSS initiate failed
    [Caused by GSSException: No valid credentials provided (Mechanism level:
    Failed to find any Kerberos tgt)]
    14/04/24 12:04:43 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate
    failed [Caused by GSSException: No valid credentials provided (Mechanism
    level: Failed to find any Kerberos tgt)]
    14/04/24 12:04:43 WARN retry.RetryInvocationHandler: Exception while
    invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 1
    fail over attempts. Trying to fail over after sleeping for 1395ms.
    14/04/24 12:04:45 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 12:04:45 WARN ipc.Client: Exception encountered while connecting
    to the server : javax.security.sasl.SaslException: GSS initiate failed
    [Caused by GSSException: No valid credentials provided (Mechanism level:
    Failed to find any Kerberos tgt)]
    14/04/24 12:04:45 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate
    failed [Caused by GSSException: No valid credentials provided (Mechanism
    level: Failed to find any Kerberos tgt)]

    The kinit has been executed:
    [hdfs@hadoopha02 ~]$ kinit -kt
    /var/run/cloudera-scm-agent/process/544-hdfs-NAMENODE/hdfs.keytab
    hdfs/hadoopha02.domain.intra@DOMAIN.INTRA

    and I can see this by doing klist command:
    [hdfs@hadoopha02 ~]$ klist
    Ticket cache: FILE:/tmp/krb5cc_868271872
    Default principal: hdfs/hadoopha02.domain.intra@DOMAIN.INTRA

    Valid starting Expires Service principal
    04/24/14 12:04:36 04/24/14 22:04:37 krbtgt/DOMAIN.INTRA@DOMAIN.INTRA
    renew until 04/25/14 12:04:31

    [hdfs@hadoopha02 ~]$ id
    uid=868271872(hdfs) gid=868271872(hdfs) groups=868271872(hdfs),493(hadoop)

    Any idea? let me know if I need to send more info.

    Thx


    To unsubscribe from this group and stop receiving emails from it, send an
    email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • MrAkhe83 at Apr 24, 2014 at 6:39 pm
    Thanks - I've now configured an hdfs service account and if I do now a
    klist I see:

    From a Datanode:
    [hdfs@hadoopha06 ~]$ klist
    Ticket cache: FILE:/tmp/krb5cc_868271872
    Default principal: hdfs@domain.intra

    Valid starting Expires Service principal
    04/24/14 13:25:18 04/24/14 23:25:22 krbtgt/DOMAIN.INTRA@DOMAIN.INTRA
         renew until 04/25/14 13:22:09
    [hdfs@hadoopha06 ~]$ hadoop fs -ls /
    14/04/24 13:31:37 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 13:31:37 WARN ipc.Client: Exception encountered while connecting
    to the server : javax.security.sasl.SaslException: GSS initiate failed
    [Caused by GSSException: No valid credentials provided (Mechanism level:
    Failed to find any Kerberos tgt)]
    14/04/24 13:31:37 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate
    failed [Caused by GSSException: No valid credentials provided (Mechanism
    level: Failed to find any Kerberos tgt)]
    ls: Failed on local exception: java.io.IOException:
    javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]; Host Details : local host is:
    "hadoopha06.domain.intra/10.198.8.77"; destination host is:
    "hadoopha02.domain.intra":8020;

    hadoopha02 is the primary namenode.
    From the namenode I see instead:

    [hdfs@hadoopha02 ~]$ klist
    Ticket cache: FILE:/tmp/krb5cc_868271872
    Default principal: hdfs@domain.intra

    Valid starting Expires Service principal
    04/24/14 13:21:28 04/24/14 23:21:32 krbtgt/DOMAIN.INTRA@DOMAIN.INTRA
         renew until 04/25/14 13:21:28
    [hdfs@hadoopha02 ~]$ hadoop fs -ls /
    14/04/24 13:30:27 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 13:30:27 WARN ipc.Client: Exception encountered while connecting
    to the server : javax.security.sasl.SaslException: GSS initiate failed
    [Caused by GSSException: No valid credentials provided (Mechanism level:
    Failed to find any Kerberos tgt)]
    14/04/24 13:30:27 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate
    failed [Caused by GSSException: No valid credentials provided (Mechanism
    level: Failed to find any Kerberos tgt)]

    I think I close to get this almost working :-)
    Is there anything else I should check?

    thx
    On Thursday, April 24, 2014 1:00:25 PM UTC-5, Vikram Srivastava wrote:

    You should not use the principals for Hadoop daemons. Instead you should
    create principals for Hadoop clients separately and kinit using that.
    Follow the instructions on:
    http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Configuring-Hadoop-Security-with-Cloudera-Manager/cm5chs_hdfs_su_princ_s15.html


    On Thu, Apr 24, 2014 at 10:52 AM, MrAkhe83 <jla...@gmail.com <javascript:>
    wrote:
    Hi,
    I have a cluster with kerberos enabled and all keytabs configured
    correctly. Each credentials have been generated successfully.
    All services (HDFS, mapred, oozie, impala, hive, zookeeper,...) are
    starting up without any issues. HA is not enabled yet.
    The problem is that when I try to browse hdfs I get an error related to
    kerberos:

    [hdfs@hadoopha02 ~]$ hadoop fs -ls /
    14/04/24 12:04:43 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 12:04:43 WARN ipc.Client: Exception encountered while connecting
    to the server : javax.security.sasl.SaslException: GSS initiate failed
    [Caused by GSSException: No valid credentials provided (Mechanism level:
    Failed to find any Kerberos tgt)]
    14/04/24 12:04:43 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate
    failed [Caused by GSSException: No valid credentials provided (Mechanism
    level: Failed to find any Kerberos tgt)]
    14/04/24 12:04:43 WARN retry.RetryInvocationHandler: Exception while
    invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 1
    fail over attempts. Trying to fail over after sleeping for 1395ms.
    14/04/24 12:04:45 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 12:04:45 WARN ipc.Client: Exception encountered while connecting
    to the server : javax.security.sasl.SaslException: GSS initiate failed
    [Caused by GSSException: No valid credentials provided (Mechanism level:
    Failed to find any Kerberos tgt)]
    14/04/24 12:04:45 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate
    failed [Caused by GSSException: No valid credentials provided (Mechanism
    level: Failed to find any Kerberos tgt)]

    The kinit has been executed:
    [hdfs@hadoopha02 ~]$ kinit -kt
    /var/run/cloudera-scm-agent/process/544-hdfs-NAMENODE/hdfs.keytab
    hdfs/hadoopha02.domain.intra@DOMAIN.INTRA

    and I can see this by doing klist command:
    [hdfs@hadoopha02 ~]$ klist
    Ticket cache: FILE:/tmp/krb5cc_868271872
    Default principal: hdfs/hadoopha02.domain.intra@DOMAIN.INTRA

    Valid starting Expires Service principal
    04/24/14 12:04:36 04/24/14 22:04:37 krbtgt/DOMAIN.INTRA@DOMAIN.INTRA
    renew until 04/25/14 12:04:31

    [hdfs@hadoopha02 ~]$ id
    uid=868271872(hdfs) gid=868271872(hdfs) groups=868271872(hdfs),493(hadoop)

    Any idea? let me know if I need to send more info.

    Thx


    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+...@cloudera.org <javascript:>.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Vikram Srivastava at Apr 24, 2014 at 7:09 pm
    Can you do "kinit -R" and try again?

    On Thu, Apr 24, 2014 at 11:39 AM, MrAkhe83 wrote:

    Thanks - I've now configured an hdfs service account and if I do now a
    klist I see:

    From a Datanode:
    [hdfs@hadoopha06 ~]$ klist
    Ticket cache: FILE:/tmp/krb5cc_868271872
    Default principal: hdfs@domain.intra


    Valid starting Expires Service principal
    04/24/14 13:25:18 04/24/14 23:25:22 krbtgt/DOMAIN.INTRA@DOMAIN.INTRA
    renew until 04/25/14 13:22:09
    [hdfs@hadoopha06 ~]$ hadoop fs -ls /
    14/04/24 13:31:37 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 13:31:37 WARN ipc.Client: Exception encountered while connecting
    to the server : javax.security.sasl.SaslException: GSS initiate failed
    [Caused by GSSException: No valid credentials provided (Mechanism level:
    Failed to find any Kerberos tgt)]
    14/04/24 13:31:37 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate
    failed [Caused by GSSException: No valid credentials provided (Mechanism
    level: Failed to find any Kerberos tgt)]
    ls: Failed on local exception: java.io.IOException:
    javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]; Host Details : local host is:
    "hadoopha06.domain.intra/10.198.8.77"; destination host is:
    "hadoopha02.domain.intra":8020;

    hadoopha02 is the primary namenode.
    From the namenode I see instead:


    [hdfs@hadoopha02 ~]$ klist
    Ticket cache: FILE:/tmp/krb5cc_868271872
    Default principal: hdfs@domain.intra


    Valid starting Expires Service principal
    04/24/14 13:21:28 04/24/14 23:21:32 krbtgt/DOMAIN.INTRA@DOMAIN.INTRA
    renew until 04/25/14 13:21:28

    [hdfs@hadoopha02 ~]$ hadoop fs -ls /
    14/04/24 13:30:27 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 13:30:27 WARN ipc.Client: Exception encountered while connecting
    to the server : javax.security.sasl.SaslException: GSS initiate failed
    [Caused by GSSException: No valid credentials provided (Mechanism level:
    Failed to find any Kerberos tgt)]
    14/04/24 13:30:27 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate
    failed [Caused by GSSException: No valid credentials provided (Mechanism
    level: Failed to find any Kerberos tgt)]

    I think I close to get this almost working :-)
    Is there anything else I should check?

    thx

    On Thursday, April 24, 2014 1:00:25 PM UTC-5, Vikram Srivastava wrote:

    You should not use the principals for Hadoop daemons. Instead you should
    create principals for Hadoop clients separately and kinit using that.
    Follow the instructions on: http://www.cloudera.com/
    content/cloudera-content/cloudera-docs/CM5/latest/
    Configuring-Hadoop-Security-with-Cloudera-Manager/cm5chs_
    hdfs_su_princ_s15.html

    On Thu, Apr 24, 2014 at 10:52 AM, MrAkhe83 wrote:

    Hi,
    I have a cluster with kerberos enabled and all keytabs configured
    correctly. Each credentials have been generated successfully.
    All services (HDFS, mapred, oozie, impala, hive, zookeeper,...) are
    starting up without any issues. HA is not enabled yet.
    The problem is that when I try to browse hdfs I get an error related to
    kerberos:

    [hdfs@hadoopha02 ~]$ hadoop fs -ls /
    14/04/24 12:04:43 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 12:04:43 WARN ipc.Client: Exception encountered while
    connecting to the server : javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]
    14/04/24 12:04:43 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]
    14/04/24 12:04:43 WARN retry.RetryInvocationHandler: Exception while
    invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after
    1 fail over attempts. Trying to fail over after sleeping for 1395ms.
    14/04/24 12:04:45 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 12:04:45 WARN ipc.Client: Exception encountered while
    connecting to the server : javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]
    14/04/24 12:04:45 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]

    The kinit has been executed:
    [hdfs@hadoopha02 ~]$ kinit -kt /var/run/cloudera-scm-agent/
    process/544-hdfs-NAMENODE/hdfs.keytab hdfs/hadoopha02.domain.intra@
    DOMAIN.INTRA

    and I can see this by doing klist command:
    [hdfs@hadoopha02 ~]$ klist
    Ticket cache: FILE:/tmp/krb5cc_868271872
    Default principal: hdfs/hadoopha02.domain.intra@DOMAIN.INTRA

    Valid starting Expires Service principal
    04/24/14 12:04:36 04/24/14 22:04:37 krbtgt/DOMAIN.INTRA@DOMAIN.INTRA
    renew until 04/25/14 12:04:31

    [hdfs@hadoopha02 ~]$ id
    uid=868271872(hdfs) gid=868271872(hdfs) groups=868271872(hdfs),493(
    hadoop)

    Any idea? let me know if I need to send more info.

    Thx


    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • MrAkhe83 at Apr 24, 2014 at 9:25 pm
    Still same issue. Could this be related to jce?
    What is very interesting is that I can browse the file system via the web
    GUI http://hadoopha02:50070/dfshealth.jsp and
    http://hadoopha05.domain.intra:1006/browseDirectory.jsp?namenodeInfoPort=50070&dir=/&...

    I wonder if hdfs@domain.intra should have a specific userPrincipaleName
    and/or servicePrincipaleName.
    Currently the userPrincipaleName is set to hdfs@domain.intra

    Thx


    On Thursday, April 24, 2014 2:09:51 PM UTC-5, Vikram Srivastava wrote:

    Can you do "kinit -R" and try again?


    On Thu, Apr 24, 2014 at 11:39 AM, MrAkhe83 <jla...@gmail.com <javascript:>
    wrote:
    Thanks - I've now configured an hdfs service account and if I do now a
    klist I see:

    From a Datanode:
    [hdfs@hadoopha06 ~]$ klist
    Ticket cache: FILE:/tmp/krb5cc_868271872
    Default principal: hdfs@domain.intra


    Valid starting Expires Service principal
    04/24/14 13:25:18 04/24/14 23:25:22 krbtgt/DOMAIN.INTRA@DOMAIN.INTRA
    renew until 04/25/14 13:22:09
    [hdfs@hadoopha06 ~]$ hadoop fs -ls /
    14/04/24 13:31:37 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 13:31:37 WARN ipc.Client: Exception encountered while connecting
    to the server : javax.security.sasl.SaslException: GSS initiate failed
    [Caused by GSSException: No valid credentials provided (Mechanism level:
    Failed to find any Kerberos tgt)]
    14/04/24 13:31:37 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate
    failed [Caused by GSSException: No valid credentials provided (Mechanism
    level: Failed to find any Kerberos tgt)]
    ls: Failed on local exception: java.io.IOException:
    javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]; Host Details : local host is:
    "hadoopha06.domain.intra/10.198.8.77"; destination host is:
    "hadoopha02.domain.intra":8020;

    hadoopha02 is the primary namenode.
    From the namenode I see instead:


    [hdfs@hadoopha02 ~]$ klist
    Ticket cache: FILE:/tmp/krb5cc_868271872
    Default principal: hdfs@domain.intra


    Valid starting Expires Service principal
    04/24/14 13:21:28 04/24/14 23:21:32 krbtgt/DOMAIN.INTRA@DOMAIN.INTRA
    renew until 04/25/14 13:21:28

    [hdfs@hadoopha02 ~]$ hadoop fs -ls /
    14/04/24 13:30:27 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 13:30:27 WARN ipc.Client: Exception encountered while connecting
    to the server : javax.security.sasl.SaslException: GSS initiate failed
    [Caused by GSSException: No valid credentials provided (Mechanism level:
    Failed to find any Kerberos tgt)]
    14/04/24 13:30:27 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate
    failed [Caused by GSSException: No valid credentials provided (Mechanism
    level: Failed to find any Kerberos tgt)]

    I think I close to get this almost working :-)
    Is there anything else I should check?

    thx

    On Thursday, April 24, 2014 1:00:25 PM UTC-5, Vikram Srivastava wrote:

    You should not use the principals for Hadoop daemons. Instead you should
    create principals for Hadoop clients separately and kinit using that.
    Follow the instructions on: http://www.cloudera.com/
    content/cloudera-content/cloudera-docs/CM5/latest/
    Configuring-Hadoop-Security-with-Cloudera-Manager/cm5chs_
    hdfs_su_princ_s15.html

    On Thu, Apr 24, 2014 at 10:52 AM, MrAkhe83 wrote:

    Hi,
    I have a cluster with kerberos enabled and all keytabs configured
    correctly. Each credentials have been generated successfully.
    All services (HDFS, mapred, oozie, impala, hive, zookeeper,...) are
    starting up without any issues. HA is not enabled yet.
    The problem is that when I try to browse hdfs I get an error related to
    kerberos:

    [hdfs@hadoopha02 ~]$ hadoop fs -ls /
    14/04/24 12:04:43 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused
    by GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 12:04:43 WARN ipc.Client: Exception encountered while
    connecting to the server : javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]
    14/04/24 12:04:43 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]
    14/04/24 12:04:43 WARN retry.RetryInvocationHandler: Exception while
    invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after
    1 fail over attempts. Trying to fail over after sleeping for 1395ms.
    14/04/24 12:04:45 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused
    by GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 12:04:45 WARN ipc.Client: Exception encountered while
    connecting to the server : javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]
    14/04/24 12:04:45 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]

    The kinit has been executed:
    [hdfs@hadoopha02 ~]$ kinit -kt /var/run/cloudera-scm-agent/
    process/544-hdfs-NAMENODE/hdfs.keytab hdfs/hadoopha02.domain.intra@
    DOMAIN.INTRA

    and I can see this by doing klist command:
    [hdfs@hadoopha02 ~]$ klist
    Ticket cache: FILE:/tmp/krb5cc_868271872
    Default principal: hdfs/hadoopha02.domain.intra@DOMAIN.INTRA

    Valid starting Expires Service principal
    04/24/14 12:04:36 04/24/14 22:04:37 krbtgt/DOMAIN.INTRA@DOMAIN.INTRA
    renew until 04/25/14 12:04:31

    [hdfs@hadoopha02 ~]$ id
    uid=868271872(hdfs) gid=868271872(hdfs) groups=868271872(hdfs),493(
    hadoop)

    Any idea? let me know if I need to send more info.

    Thx


    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+...@cloudera.org <javascript:>.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • MrAkhe83 at Apr 28, 2014 at 6:03 pm
    The issue is that I also had to restart the Cloudera Management Services.
    This is something that I couldn't find in the documentations....


    On Thursday, April 24, 2014 2:09:51 PM UTC-5, Vikram Srivastava wrote:

    Can you do "kinit -R" and try again?


    On Thu, Apr 24, 2014 at 11:39 AM, MrAkhe83 <jla...@gmail.com <javascript:>
    wrote:
    Thanks - I've now configured an hdfs service account and if I do now a
    klist I see:

    From a Datanode:
    [hdfs@hadoopha06 ~]$ klist
    Ticket cache: FILE:/tmp/krb5cc_868271872
    Default principal: hdfs@domain.intra


    Valid starting Expires Service principal
    04/24/14 13:25:18 04/24/14 23:25:22 krbtgt/DOMAIN.INTRA@DOMAIN.INTRA
    renew until 04/25/14 13:22:09
    [hdfs@hadoopha06 ~]$ hadoop fs -ls /
    14/04/24 13:31:37 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 13:31:37 WARN ipc.Client: Exception encountered while connecting
    to the server : javax.security.sasl.SaslException: GSS initiate failed
    [Caused by GSSException: No valid credentials provided (Mechanism level:
    Failed to find any Kerberos tgt)]
    14/04/24 13:31:37 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate
    failed [Caused by GSSException: No valid credentials provided (Mechanism
    level: Failed to find any Kerberos tgt)]
    ls: Failed on local exception: java.io.IOException:
    javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]; Host Details : local host is:
    "hadoopha06.domain.intra/10.198.8.77"; destination host is:
    "hadoopha02.domain.intra":8020;

    hadoopha02 is the primary namenode.
    From the namenode I see instead:


    [hdfs@hadoopha02 ~]$ klist
    Ticket cache: FILE:/tmp/krb5cc_868271872
    Default principal: hdfs@domain.intra


    Valid starting Expires Service principal
    04/24/14 13:21:28 04/24/14 23:21:32 krbtgt/DOMAIN.INTRA@DOMAIN.INTRA
    renew until 04/25/14 13:21:28

    [hdfs@hadoopha02 ~]$ hadoop fs -ls /
    14/04/24 13:30:27 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 13:30:27 WARN ipc.Client: Exception encountered while connecting
    to the server : javax.security.sasl.SaslException: GSS initiate failed
    [Caused by GSSException: No valid credentials provided (Mechanism level:
    Failed to find any Kerberos tgt)]
    14/04/24 13:30:27 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate
    failed [Caused by GSSException: No valid credentials provided (Mechanism
    level: Failed to find any Kerberos tgt)]

    I think I close to get this almost working :-)
    Is there anything else I should check?

    thx

    On Thursday, April 24, 2014 1:00:25 PM UTC-5, Vikram Srivastava wrote:

    You should not use the principals for Hadoop daemons. Instead you should
    create principals for Hadoop clients separately and kinit using that.
    Follow the instructions on: http://www.cloudera.com/
    content/cloudera-content/cloudera-docs/CM5/latest/
    Configuring-Hadoop-Security-with-Cloudera-Manager/cm5chs_
    hdfs_su_princ_s15.html

    On Thu, Apr 24, 2014 at 10:52 AM, MrAkhe83 wrote:

    Hi,
    I have a cluster with kerberos enabled and all keytabs configured
    correctly. Each credentials have been generated successfully.
    All services (HDFS, mapred, oozie, impala, hive, zookeeper,...) are
    starting up without any issues. HA is not enabled yet.
    The problem is that when I try to browse hdfs I get an error related to
    kerberos:

    [hdfs@hadoopha02 ~]$ hadoop fs -ls /
    14/04/24 12:04:43 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused
    by GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 12:04:43 WARN ipc.Client: Exception encountered while
    connecting to the server : javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]
    14/04/24 12:04:43 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]
    14/04/24 12:04:43 WARN retry.RetryInvocationHandler: Exception while
    invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after
    1 fail over attempts. Trying to fail over after sleeping for 1395ms.
    14/04/24 12:04:45 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused
    by GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 12:04:45 WARN ipc.Client: Exception encountered while
    connecting to the server : javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]
    14/04/24 12:04:45 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]

    The kinit has been executed:
    [hdfs@hadoopha02 ~]$ kinit -kt /var/run/cloudera-scm-agent/
    process/544-hdfs-NAMENODE/hdfs.keytab hdfs/hadoopha02.domain.intra@
    DOMAIN.INTRA

    and I can see this by doing klist command:
    [hdfs@hadoopha02 ~]$ klist
    Ticket cache: FILE:/tmp/krb5cc_868271872
    Default principal: hdfs/hadoopha02.domain.intra@DOMAIN.INTRA

    Valid starting Expires Service principal
    04/24/14 12:04:36 04/24/14 22:04:37 krbtgt/DOMAIN.INTRA@DOMAIN.INTRA
    renew until 04/25/14 12:04:31

    [hdfs@hadoopha02 ~]$ id
    uid=868271872(hdfs) gid=868271872(hdfs) groups=868271872(hdfs),493(
    hadoop)

    Any idea? let me know if I need to send more info.

    Thx


    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+...@cloudera.org <javascript:>.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Vikram Srivastava at Apr 28, 2014 at 6:09 pm
    Thanks for pointing this out. I've created an internal issue to add this to
    our docs.

    On Mon, Apr 28, 2014 at 11:03 AM, MrAkhe83 wrote:

    The issue is that I also had to restart the Cloudera Management Services.
    This is something that I couldn't find in the documentations....



    On Thursday, April 24, 2014 2:09:51 PM UTC-5, Vikram Srivastava wrote:

    Can you do "kinit -R" and try again?

    On Thu, Apr 24, 2014 at 11:39 AM, MrAkhe83 wrote:

    Thanks - I've now configured an hdfs service account and if I do now a
    klist I see:

    From a Datanode:
    [hdfs@hadoopha06 ~]$ klist
    Ticket cache: FILE:/tmp/krb5cc_868271872
    Default principal: hdfs@domain.intra


    Valid starting Expires Service principal
    04/24/14 13:25:18 04/24/14 23:25:22 krbtgt/DOMAIN.INTRA@DOMAIN.INTRA
    renew until 04/25/14 13:22:09
    [hdfs@hadoopha06 ~]$ hadoop fs -ls /
    14/04/24 13:31:37 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 13:31:37 WARN ipc.Client: Exception encountered while
    connecting to the server : javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]
    14/04/24 13:31:37 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]
    ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException:
    GSS initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local
    host is: "hadoopha06.domain.intra/10.198.8.77"; destination host is:
    "hadoopha02.domain.intra":8020;

    hadoopha02 is the primary namenode.
    From the namenode I see instead:


    [hdfs@hadoopha02 ~]$ klist
    Ticket cache: FILE:/tmp/krb5cc_868271872
    Default principal: hdfs@domain.intra


    Valid starting Expires Service principal
    04/24/14 13:21:28 04/24/14 23:21:32 krbtgt/DOMAIN.INTRA@DOMAIN.INTRA
    renew until 04/25/14 13:21:28

    [hdfs@hadoopha02 ~]$ hadoop fs -ls /
    14/04/24 13:30:27 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
    GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 13:30:27 WARN ipc.Client: Exception encountered while
    connecting to the server : javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]
    14/04/24 13:30:27 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]

    I think I close to get this almost working :-)
    Is there anything else I should check?

    thx

    On Thursday, April 24, 2014 1:00:25 PM UTC-5, Vikram Srivastava wrote:

    You should not use the principals for Hadoop daemons. Instead you
    should create principals for Hadoop clients separately and kinit using
    that. Follow the instructions on: http://www.cloudera.com/co
    ntent/cloudera-content/cloudera-docs/CM5/latest/Configuring-
    Hadoop-Security-with-Cloudera-Manager/cm5chs_hdfs_su_princ_s15.html

    On Thu, Apr 24, 2014 at 10:52 AM, MrAkhe83 wrote:

    Hi,
    I have a cluster with kerberos enabled and all keytabs configured
    correctly. Each credentials have been generated successfully.
    All services (HDFS, mapred, oozie, impala, hive, zookeeper,...) are
    starting up without any issues. HA is not enabled yet.
    The problem is that when I try to browse hdfs I get an error related
    to kerberos:

    [hdfs@hadoopha02 ~]$ hadoop fs -ls /
    14/04/24 12:04:43 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused
    by GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 12:04:43 WARN ipc.Client: Exception encountered while
    connecting to the server : javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]
    14/04/24 12:04:43 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]
    14/04/24 12:04:43 WARN retry.RetryInvocationHandler: Exception while
    invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB
    after 1 fail over attempts. Trying to fail over after sleeping for 1395ms.
    14/04/24 12:04:45 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:javax.security.sasl.SaslException: GSS initiate failed [Caused
    by GSSException: No valid credentials provided (Mechanism level: Failed to
    find any Kerberos tgt)]
    14/04/24 12:04:45 WARN ipc.Client: Exception encountered while
    connecting to the server : javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]
    14/04/24 12:04:45 ERROR security.UserGroupInformation:
    PriviledgedActionException as:hdfs (auth:KERBEROS)
    cause:java.io.IOException: javax.security.sasl.SaslException: GSS
    initiate failed [Caused by GSSException: No valid credentials provided
    (Mechanism level: Failed to find any Kerberos tgt)]

    The kinit has been executed:
    [hdfs@hadoopha02 ~]$ kinit -kt /var/run/cloudera-scm-agent/pr
    ocess/544-hdfs-NAMENODE/hdfs.keytab hdfs/hadoopha02.domain.intra@D
    OMAIN.INTRA

    and I can see this by doing klist command:
    [hdfs@hadoopha02 ~]$ klist
    Ticket cache: FILE:/tmp/krb5cc_868271872
    Default principal: hdfs/hadoopha02.domain.intra@DOMAIN.INTRA

    Valid starting Expires Service principal
    04/24/14 12:04:36 04/24/14 22:04:37 krbtgt/DOMAIN.INTRA@DOMAIN.INTRA
    renew until 04/25/14 12:04:31

    [hdfs@hadoopha02 ~]$ id
    uid=868271872(hdfs) gid=868271872(hdfs) groups=868271872(hdfs),493(had
    oop)

    Any idea? let me know if I need to send more info.

    Thx


    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to scm-users+...@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedApr 24, '14 at 5:52p
activeApr 28, '14 at 6:09p
posts7
users2
websitecloudera.com
irc#hadoop

2 users in discussion

MrAkhe83: 4 posts Vikram Srivastava: 3 posts

People

Translate

site design / logo © 2022 Grokbase