Kerberos DNS issue with /etc/hosts

Question

Thank you all for your time and attention. Very simply I'm getting this error when invoking kinit to test kerberos authentication against my AD domain (which is hosted on a Server 2012 box).

The error: kinit: Cannot contact any KDC for realm 'MYDOMAIN.LOCAL' while getting initial credentials

Now, everything is fixed when I either explicitly set the KDC to an IP address instead of the host name in /etc/krb5.conf (e.g. kdc = 172.16.0.10 vs kdc = ad0.mydomain.local and same with admin_server) and it also works when the KDC is set as the hostname in /etc/krb5.conf and the KDC also has an explicit entry in /etc/hosts. My real question is why is this necessary? The whole point of DNS is to not have to use these local hosts files and I don't want my config setup that way. Does anyone know why this works this way? All the usual network tools and the host -t SRV ... queries all seem to work correctly (details below) without these extra steps (/etc/krb5.conf lists host name and no entry in /etc/hosts).

So can anyone tell me why kinit seems to be ignoring my resolv.conf file?

Tokens

• local host IP is 172.16.0.20
• Local hostname is ps
• Local FQDN is ps.mydomain.local
• AD host IP is 172.16.0.10
• AD hostname is ad0
• AD FQDN is ad0.mydomain.local
• AD is primary DNS
• domain is mydomain.local (I know .local is bad, working on it)
• Network is 172.16.0.0/24

/etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
  address 172.16.0.20
  netmask 255.255.255.0
  gateway 172.16.0.1
  network 172.16.0.0
  broadcast 172.16.0.255
  dns-nameservers 172.16.0.10 8.8.8.8
  dns-search MYDOMAIN.LOCAL

/etc/hosts

127.0.0.1 localhost
172.16.0.20 ps.mydomain.local ps

# IPv6 stuff, don't think it's relevant to the problem
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

/etc/resolv.conf - automatically generated

nameserver 172.16.0.10
nameserver 8.8.8.8
search MYDOMAIN.LOCAL

/etc/krb5.conf

[libdefaults]
  default_realm = MYDOMAIN.LOCAL
  dns_lookup_kdc = true
  dns_lookup_realm = true
  # tried dns_lookup_realm as both true and false, no change

[realms]
  MYDOMAIN.LOCAL = {
    kdc = ad0.mydomain.local
    admin_server = ad0.mydomain.local
    default_domain = mydomain.local
  }

[domain_realm]
  .opticonwa.local = OPTICONWA.LOCAL
  opticonwa.local = OPTICONWA.LOCAL

Not including smb.conf as this all works once kinit functions.

Tests

user@ps:~$ KRB5_TRACE=/dev/stdout kinit administrator
[1634] 1447281918.814976: Getting initial credentials for [email protected]
[1634] 1447281918.815297: Sending request (181 bytes) to MYDOMAIN.LOCAL
[1634] 1447281918.815346: Resolving hostname ad0.mydomain.local
[1634] 1447281923.817731: Resolving hostname ad0.mydomain.local
kinit: Cannot contact any KDC for realm 'MYDOMAIN.LOCAL' while getting initial credentials

user@ps:~$ host -t SRV _kerberos._udp.mydomain.local
_kerberos._udp.mydomain.local has SRV record 0 0 88 ad0.mydomain.local.
_kerberos._udp.mydomain.local has SRV record 0 100 88 ad0.mydomain.local.

user@ps:~$ host -t SRV _kerberos._tcp.mydomain.local
_kerberos._tcp.mydomain.local has SRV record 0 0 88 ad0.mydomain.local.
_kerberos._tcp.mydomain.local has SRV record 0 100 88 ad0.mydomain.local.



user@ps:~$ dig -x 172.16.0.10
; <<>> DiG 9.9.5-3ubuntu0.5-Ubuntu <<>> -x 172.16.0.10
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19749
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;10.0.16.172.in-addr.arpa.  IN  PTR

;; ANSWER SECTION:
10.0.16.172.in-addr.arpa. 1200 IN   PTR ad0.mydomain.local.

;; Query time: 2 msec
;; SERVER: 172.16.0.10#53(172.16.0.10)
;; WHEN: Thu Nov 12 11:19:27 PST 2015
;; MSG SIZE  rcvd: 92

Not sure why I'm showing 2 SRV records as the server only has 1 (at least that I'm aware of).

output of gentent hosts ...

user@ps:~$ getent hosts ad0.mydonaim.local


user@ps:~$ getent hosts 172.16.0.10
172.16.0.10 ad0.mydomain.local

So reverse lookups work when the hosts file does not contain an entry for the domain controller. Forward lookups do not. Both work when the entry is in hosts...

Output of nsswitch.conf unmodified from base install...

passwd:     files winbind
group:      files winbind
#passwd:         compat
#group:          compat
shadow:         compat

hosts:          files mdns4_minimal [NOTFOUND=return] dns
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       nis

So can anyone tell me why kinit seems to ignore /etc/resolv.conf? Does it have to do with the incomplete DNS implementation for the admin_server stanza in /etc/krb5.conf? Or am I stuck in some kind of DNS catch-22 that I'm failing to notice? But why only Kerberos? As an add-on question, why does the local host name need to be in /etc/hosts anyway? Can someone point me to a thorough writeup on linux networking that would explain how this works on the back end?

Thanks again for your help. This is really bugging me.

Solution

For kinit/kerberos to work properly, host name lookups should work for both forward and reverse zones. On Linux hosts nsswitch.conf determines the sources to obtain name-service information. Each line in that file controls the database sources, for example hosts line determines what sources would be looked up for host name resolution.

With default setting of hosts

hosts: files mdns4_minimal [NOTFOUND=return] dns

DNS will not be looked up at all and it would return if host record was not found in files (/etc/hosts) and mdns4_minimal (This source seem to be related to avahi-daemon multicast DNS, however it seems unable to resolve your kerberos host)

Removing [NOTFOUND=return] allowed host resolution process to contact dns servers in /etc/resolv.conf (which seem to be working as per dig requests seen above)

Comments

[VenkatC]

Is reverse lookup for KDC working via DNS ? dig -x 172.16.0.10

[Arkaxandai]

updated post with dig output. Looks like yes though I'm not sure why the DNS server isn't showing as authoratative.

[VenkatC]

that seems ok, still issue seems related to name resolution. Does lookup via 'getent hosts' work fine as well? for both ad0.mydomain.local and 172.16.0.10 (without /etc/hosts entry in place)

[Arkaxandai]

Updated with getent hosts output. Been a bit busy, sorry for my delay. And thanks again for your attention.

[VenkatC]

So, basically something is wrong with host lookup order, even though dig works, kinit uses gethostbyname/addr internally and 'getent hosts' should work for both forrward and reverse lookups. Check nsswitch.conf entries for 'hosts' and make sure it has dns for host lookup after files

[VenkatC]

Can you try modifying hosts line as below in nsswitch.conf and check name resolution with getent hosts

[VenkatC]

hosts: files dns

[VenkatC]

(or) Just getrid of NOTFOUND part, so host lookup continues in DNS instead of returning if not found hosts: files mdns4_minimal dns. read man nsswitch.conf for more info

[Arkaxandai]

Removing the [NOTFOUND=return] parameter does indeed make it so that the getent hosts hostname command does indeed resolve to a IP. Subsequently the kinit command now also works. Interestingly, before the change to nsswitch.conf I did get kinit to work when I left the @mydomain.local part off of the username parameter but it was failing as in my original post when the domain was appended to the username (which I thought was required). Now both work. If you can shed light on that it would be great but at this point I consider my original problem solved. I'll look into man nsswitch. Thanks!

[VenkatC]

Cool, I have posted the same as answer with some more info. As your issue is resolved,accept that as answer