Kerberized nfs4 mounts ERROR: No credentials found for connection to server

Question

My client/servers are both running ubuntu 14.04 and kerberos user authentication works as intended. regular nfs4 mounts also work fine. All machines are running heimdal libraries.

I haven't been able to get kerberized nfs4 working though.

When mounting a share, I get the following logs:

CLIENT:

# mount -t nfs4 -o sec=krb5 server:/ /mnt/tmp -vvvvvv                                                                        
mount: fstab path: "/etc/fstab"
mount: mtab path:  "/etc/mtab"
mount: lock path:  "/etc/mtab~"
mount: temp path:  "/etc/mtab.tmp"
mount: UID:        0
mount: eUID:       0
mount: spec:  "SERVER:/"
mount: node:  "/mnt/tmp"
mount: types: "nfs4"
mount: opts:  "sec=krb5"
mount: external mount: argv[0] = "/sbin/mount.nfs4"
mount: external mount: argv[1] = "SERVER:/"
mount: external mount: argv[2] = "/mnt/tmp"
mount: external mount: argv[3] = "-v"
mount: external mount: argv[4] = "-o"
mount: external mount: argv[5] = "rw,sec=krb5"
mount.nfs4: timeout set for Sun Jun 15 01:10:30 2014
mount.nfs4: trying text-based options 'sec=krb5,addr=XXX.XXX.XXX.52,clientaddr=XXX.XXX.XXX.17'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting SERVER:/

rpc.gssd:

Jun 15 01:31:15 client rpc.gssd[24146]: destroying client /run/rpc_pipefs/nfsd4_cb/clnt4
Jun 15 01:31:15 client rpc.gssd[24146]: destroying client /run/rpc_pipefs/nfsd4_cb/clnt3
Jun 15 01:31:15 client rpc.gssd[24146]: destroying client /run/rpc_pipefs/nfsd4_cb/clnt2
Jun 15 01:31:15 client rpc.gssd[24146]: destroying client /run/rpc_pipefs/nfsd4_cb/clnt0
Jun 15 01:31:15 client rpc.gssd[24146]: handling gssd upcall (/run/rpc_pipefs/nfs/clntf)
Jun 15 01:31:15 client rpc.gssd[24146]: handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 '
Jun 15 01:31:15 client rpc.gssd[24146]: handling krb5 upcall (/run/rpc_pipefs/nfs/clntf)
Jun 15 01:31:15 client rpc.gssd[24146]: process_krb5_upcall: service is '<null>'
Jun 15 01:31:15 client rpc.gssd[24146]: Full hostname for 'server.example.com' is 'server.example.com'
Jun 15 01:31:15 client rpc.gssd[24146]: Full hostname for 'client.example.com' is 'CLIENT.example.com'
Jun 15 01:31:15 client rpc.gssd[24146]: No key table entry found for [email protected] while getting keytab entry for 'DEVEL01$@'
Jun 15 01:31:15 client rpc.gssd[24146]: No key table entry found for root/[email protected] while getting keytab entry for 'root/CLIENT.example.com@'
Jun 15 01:31:15 client rpc.gssd[24146]: Success getting keytab entry for 'nfs/client.example.com@'
Jun 15 01:31:15 client rpc.gssd[24146]: WARNING: Cryptosystem internal error while getting initial ticket for principal 'nfs/[email protected]' using keytab 'FILE:/etc/krb5.keytab'
Jun 15 01:31:15 client rpc.gssd[24146]: ERROR: No credentials found for connection to server server.example.com
Jun 15 01:31:15 client rpc.gssd[24146]: doing error downcall
Jun 15 01:31:15 client rpc.gssd[24146]: destroying client /run/rpc_pipefs/nfs/clnt55
Jun 15 01:31:15 client rpc.gssd[24146]: destroying client /run/rpc_pipefs/nfsd4_cb/clnt4
Jun 15 01:31:15 client rpc.gssd[24146]: destroying client /run/rpc_pipefs/nfsd4_cb/clnt3
Jun 15 01:31:15 client rpc.gssd[24146]: destroying client /run/rpc_pipefs/nfsd4_cb/clnt2
Jun 15 01:31:15 client rpc.gssd[24146]: destroying client /run/rpc_pipefs/nfsd4_cb/clnt0

Client keytab:

Vno  Type                     Principal                                    Aliases
  1  aes256-cts-hmac-sha1-96  nfs/[email protected]  
  1  des3-cbc-sha1            nfs/[email protected]
  1  arcfour-hmac-md5         nfs/[email protected]

Server:

KDC:

Jun 15 01:44:34 server kdc[13705]: AS-REQ nfs/[email protected] from IPv4:XXX.XXX.XXX.17 for krbtgt/[email protected]
Jun 15 01:44:34 server kdc[13705]: Client sent patypes: REQ-ENC-PA-REP
Jun 15 01:44:34 server kdc[13705]: Looking for PK-INIT(ietf) pa-data -- nfs/[email protected]
Jun 15 01:44:34 server kdc[13705]: Looking for PK-INIT(win2k) pa-data -- nfs/[email protected]
Jun 15 01:44:34 server kdc[13705]: Looking for ENC-TS pa-data -- nfs/[email protected]
Jun 15 01:44:34 server kdc[13705]: Need to use PA-ENC-TIMESTAMP/PA-PK-AS-REQ
Jun 15 01:44:34 server kdc[13705]: sending 292 bytes to IPv4:XXX.XXX.XXX.17

Any pointers on what is wrong here?

Solution

Jun 15 01:31:15 client rpc.gssd[24146]: WARNING: Cryptosystem internal error while getting initial ticket for principal 'nfs/[email protected]' using keytab 'FILE:/etc/krb5.keytab'

Can you use kinit to get a tgt using this command ( assumes mit kinit )

kinit -k -t /etc/krb5.keytab nfs/[email protected]

I'm guessing you've sanitized this, but case matters in kerberos principals. The principal in the error message is not the same as in the keytab. Do you do funny things with DNS ( like returning upper case DNS host names )?

Looking at the KDC messages, my guess is that you do not have the correct key for nfs/client.example.com in the keytab.

Comments

[kofemann]

May be encryption is too strong/weak. What is in your krb5.conf?

[cebalrai]

NTP daemons are running on all involved machines, the logs aren't showing the same event, but the logs would be the same either way.

[cebalrai]

As you guessed, that doesn't work on this specific host. It works on other hosts with keys that were created using the same settings (randomn key, unlimited lifetime, etc.) I've recreated the keys and exported them but a kinit on the nfs principal gives ma a "Password incorrect". Kinit on other hosts works flawless. I'm at a loss what is happening here.

[Jorge Silva]

Is there a reason why running kadmin from the nfs client (instead of the KDC) doesn't work. Should that never work? or should it?