FreeIPA: command-line tools do not work, 'No Kerberos credentials available'

Question

We have a working FreeIPA installation, it's in production since February. Almost everything works as expected but when we try to run command-line FreeIPA-related tools none of them work:

[admin@ipa ~]$ kinit admin
Password for [email protected]: 
[admin@ipa ~]$ klist
Ticket cache: KEYRING:persistent:8800000
Default principal: [email protected]

Valid starting       Expires              Service principal
06/30/2014 21:19:30  07/01/2014 21:19:12  krbtgt/[email protected]
[admin@ipa ~]$ ipa pwpolicy-show global_policy
ipa: ERROR: Kerberos error: ('Unspecified GSS failure.  Minor code may provide more information', 851968)/('No Kerberos credentials available', -1765328243)
[admin@ipa ~]$

I'm not a Kerberos expert and don't really know what to check. How can we debug and resolve this?

Update: when I add -vv I get the following:

[admin@ipa ~]$ ipa -vv pwpolicy-show global_policy
ipa: INFO: trying https://ipa.example.com/ipa/xml
ipa: INFO: Forwarding 'pwpolicy_show' to server 'https://ipa.example.com/ipa/xml'
ipa: ERROR: Kerberos error: ('Unspecified GSS failure.  Minor code may provide more information', 851968)/('No Kerberos credentials available', -1765328243)
[admin@ipa ~]$

Update 2: the content of /etc/krb5.conf follows:

includedir /var/lib/sss/pubconf/krb5.include.d/

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = EXAMPLE.COM
 dns_lookup_realm = false
 dns_lookup_kdc = true
 rdns = false
 ticket_lifetime = 24h
 forwardable = yes
 default_ccache_name = KEYRING:persistent:%{uid}

[realms]
 EXAMPLE.COM = {
  kdc = ipa.example.com:88
  master_kdc = ipa.example.com:88
  admin_server = ipa.example.com:749
  default_domain = example.com
  pkinit_anchors = FILE:/etc/ipa/ca.crt
}

[domain_realm]
 .example.com = EXAMPLE.COM
 example.com = EXAMPLE.COM

[dbmodules]
  EXAMPLE.COM = {
    db_library = ipadb.so
  }

Update 3: This is a single-server installation, the distro is Fedora 19 and FreeIPA version is 3.3.5

Solution

The main difference I see in your configuration compared to my live FreeIPA (on Fedora 20) is that I do not use the kernel keyring as a ticket cache.

 default_ccache_name = KEYRING:persistent:%{uid}

My /etc/krb5.conf does not specify this at all, so the default FILE is used. Removing this should get you going again.

As Matthew Ife indicates in a comment, the kernel keyring is more secure and will (eventually) be the way to go, but at the moment it doesn't appear to be stable enough for production use. You may wish to report this to Fedora as a bug.

Comments

[Michael Hampton]

What Linux distributions and versions of FreeIPA are you running, both on this server and on the KDC?

[Alex]

There is only a single server actually, it's Fedora 19 w/all the latest updates, so this is FreeIPA 3.3.5

[Michael Hampton]

I'd start looking at krb5kdc.log then.

[Alex]

Unfortunately nothing gets added to this log when I run ipa ... but I noticed that ipa command tried to open a /var/kerberos/krb5/user/8800000/client.keytab file but the system did not have it. So, it looks like a keytab problem.

[Michael Hampton]

Yeah, your keytab should be elsewhere. I think. Have you just tried opening the web UI yourself?

[Alex]

Sure, and the web UI works well.

[Michael Hampton]

(Remember to edit the rest of the information into your question as well.) I don't like that default_ccache_name being set at all. It's not set in my FreeIPA installation and so my ticket cache is in the default location. I'd remove it.

[Alex]

Yes! That was it, thank you very much! Command-line tools started to work as soon as I commented out that setting.

[Matthew Ife]

The keyring is technically more secure. Its based off of kernel keyrings (see man keyctl) but I suspect its not fully supported yet as its still very new (only started to be supported in Fedora 19 as I understand it). Keyrings have scope and are not present outside of their scope. In the long run it is a better way of storing keytabs.

[Michael Hampton]

I've made a full answer of it, then.

[Bratchley]

How is the kernel keyring not stable? It's been around forever (around ten years I think). Or do you just mean the krb5->keyring code?

[Michael Hampton]

@Bratchley Right, the integration between Kerberos and the kernel keyring apparently still needs work.

[abbra]

Fedora 19 has experimental support for KEYRING: type of ccache. In particular, there was issue that kernel parts weren't yet fully bug-free. You need to remove default_ccache_name from /etc/krb5.conf to get back to the default FILE: ccache type. At the point of release of FreeIPA 3.3 we were targeting Fedora 20 already so Fedora 19 got a backport but nobody reported this kind of a bug. Now Fedora 19 is EOL and Fedora 20+ all have properly working KEYRING: ccache type.

[user2402902]

Well, I did not need to change anything at krb5.conf. I have just typed kinit admin, and entered its password. After that all ipa commands started to work as expected such as ipa user-show ausernamehere --all.