Wrong principal in request (SSH/ GSSAPI/Kerberos/Debian)

Question

I've set up two VMs on an "internal" (in VirtualBox meaning) network, one being a DNS server (dns1.example.com) and the other - a KDC and Kerberos admin server (kdc.example.com). The default and the only realm is EXAMPLE.COM. Both machines use freshly installed Debian Squeeze.

The problem: I can login via ssh on kdc.example.com from kdc.example.com, but I can't login via ssh from dns1.example.com.

On kdc.example.com, sshd in debug mode says:

debug1: Unspecified GSS failure.  Minor code may provide more information
Wrong principal in request

debug1: Got no client credentials
debug3: mm_request_send entering: type 41
debug3: mm_request_receive entering
debug1: userauth-request for user tom service ssh-connection method gssapi-with-mic
debug1: attempt 2 failures 1
debug2: input_userauth_request: try method gssapi-with-mic
debug1: userauth-request for user tom service ssh-connection method gssapi-with-mic
debug1: attempt 3 failures 1
debug2: input_userauth_request: try method gssapi-with-mic

at which point the client is asked for a password. A tcpdump file processed by Wireshark shows there has been some exchange of crypted packets, but I can't deduct more as they are, well, crypted :). After 2 days of googling I'm stuck and would appreciate any help.

Even more would I appreciate any advice /links/hints on a general sane configuration debugging strategy, when it comes to Kerberos and friends. For example, I'm out of ideas where to look for what's wrong with 'Wrong principal', and what is that principal the server receives instead of the right one. Something tells me the real adventures are yet to come :).

Below are configs and diagnostic outputs. Hope I haven't forgot anything.

kdc:~# cat /etc/krb5kdc/kdc.conf 
[kdcdefaults]
    kdc_ports = 750,88

[realms]
    EXAMPLE.COM = {
        database_name = /var/lib/krb5kdc/principal
        admin_keytab = FILE:/etc/krb5kdc/kadm5.keytab
        acl_file = /etc/krb5kdc/kadm5.acl
        key_stash_file = /etc/krb5kdc/stash
        kdc_ports = 750,88
        max_life = 10h 0m 0s
        max_renewable_life = 7d 0h 0m 0s
        master_key_type = des3-hmac-sha1
        supported_enctypes = aes256-cts:normal arcfour-hmac:normal des3-hmac-sha1:normal des-cbc-crc:normal des:normal des:v4 des:norealm des:onlyrealm des:afs3
        default_principal_flags = +preauth
    }

kdc:~# kadmin.local -q 'listprincs'
Authenticating as principal root/[email protected] with password.
K/[email protected]
host/[email protected]
host/[email protected]
host/[email protected]
host/[email protected]
kadmin/[email protected]
kadmin/[email protected]
kadmin/[email protected]
kadmin/[email protected]
krbtgt/[email protected]
root/[email protected]
[email protected]

kdc:~# cat /etc/ssh/sshd_config |grep '^[^#]'
Port 22
ListenAddress 172.16.3.3
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
UsePrivilegeSeparation yes
KeyRegenerationInterval 3600
ServerKeyBits 768
SyslogFacility AUTH
LogLevel INFO
LoginGraceTime 120
PermitRootLogin yes
StrictModes yes
RSAAuthentication yes
PubkeyAuthentication yes
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPIKeyExchange yes
GSSAPICleanupCredentials yes
X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server
UsePAM yes

/etc/krb5.conf is identical on both kdc and dns1.

dns1:~$ cat /etc/krb5.conf 
[libdefaults]
    default_realm = EXAMPLE.COM
    dns_lookup_realm = true
    dns_lookup_kdc = true 
    forwardable = true
[realms]
    EXAMPLE.COM={
        admin_server = kdc.example.com
    }
[domain_realm]
    example.com = EXAMPLE.COM
    .example.com = EXAMPLE.COM
[logging]
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmin.log
    default = FILE:/var/log/krb5lib.log

The TGT is forwardable. On ssh client:

dns1:~$ klist -f
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: [email protected]

Valid starting     Expires            Service principal
01/03/12 20:00:03  01/04/12 06:00:03  krbtgt/[email protected]
    renew until 01/04/12 20:00:00, Flags: FRIA
01/03/12 20:00:21  01/04/12 06:00:03  host/[email protected]
    renew until 01/04/12 20:00:00, Flags: FRAT

Keytab does also seem to be OK:

dns1:~# klist -k
Keytab name: WRFILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   5 host/[email protected]
   5 host/[email protected]
   5 host/[email protected]
   5 host/[email protected]

DNS (incl. PTR, TXT, SRV) works as it should.

dns1:~# cat /var/cache/bind/db.example.com 
$ORIGIN example.com.
$TTL    86400
@   IN  SOA dns1.example.com. root.example.com. (
            2012010301  ; Serial
             604800     ; Refresh
              86400     ; Retry
            2419200     ; Expire
              86400 )   ; Negative Cache TTL
;
@   IN  NS  dns1.example.com.
dns1    IN  A   172.16.3.2
www IN  A   172.16.3.8
mail    IN  A   172.16.3.9
fed IN  A   172.16.3.100

kdc IN  A   172.16.3.3
;kds    IN  A   172.16.3.4

_kerberos   TXT "EXAMPLE.COM"

krb IN  CNAME   kdc

_kerberos._udp      SRV 0 0 88  kdc
_kerberos-master._udp   SRV 0 0 88  kdc
_kerberos-adm._tcp  SRV 0 0 749 kdc
_kpasswd._udp       SRV 0 0 464 kdc

dns1:~# cat /var/cache/bind/db.3.16.172.in-addr.arpa 
$ORIGIN 3.16.172.in-addr.arpa.
$TTL    86400
@   IN  SOA dns1.example.com. root.example.com. (
            2012010102  ; Serial
             604800     ; Refresh
              86400     ; Retry
            2419200     ; Expire
              86400 )   ; Negative Cache TTL
;
@   IN  NS  dns1.example.com.
2   IN  PTR dns1.example.com.
3   IN  PTR kdc.example.com.

8   IN  PTR example.com.
9   IN  PTR mail.example.com.

Solution 1

Found this while Googling for exactly the same error on a new server build - pointed me in the right direction :)

In my case, I had incorrect reverse DNS - when I updated this and cleared my caching nameservers' caches, it worked.

Solution 2

I have to be more attentive. There was a line left in /etc/hosts resolving 127.0.0.1 to FQDN (now commented out):

kdc:~$ cat /etc/hosts
127.0.0.1   localhost
#127.0.0.1  kdc.example.com kdc
172.16.3.3  kdc.example.com kdc

After purging the related principals from DB and keytab and restarting both VMs, everything works as desired. Ufff...

Comments

[Wesley]

You are ServerFault's 100,000th question. Sadly, we have no cake.

[Tim Brigham]

The cake is a lie... the cake is a lie...

[yrk]

try configuring the realm and kdc in /etc/krb5.conf at dns1 (bypassing its kdc lookup over dns)

[FooBee]

Nothing in the question would suggest that this is the problem, and the OP self-answered and showed the issue is something else entirely.