Server Not Found in Kerberos database - where is the database located?

Question

Testing setup:

• Weblogic 12.2.1.4 running on a Windows 10 machine joined to an active directory
  • JVM 1.8.0_281
  • The java web application is using Java GSSAPI to access the fileshare over Samba essentially using the code from https://github.com/hierynomus/smbj/issues/304#issuecomment-375603115
  • This is a developer machine - the java system property "user.name" shows the developer's windows username
• Linux fileserver running Red Hat Enterprise Linux 7, configured with sssd to connect to Active Directory
• Linux fileserver running Red Hat Enterprise Linux 6, not using sssd, not using winbind (unclear how precisely it is configured for Active Directory)

From what I've been able to gather third-hand on conversations with sysadmins, MIT Kerberos is somehow involved in connecting the linux servers to Active Directory - but I have no more information on that.

(note: serverfault is having markdown table rendering issue - table displays fine in preview, but not in actual posted question, so surrounded in code block for now so that it doesn't all run together)

Test Results

| Source (all on same Windows 10 machine)  | Target FileServer     | Result  |
|------------------------------------------|-----------------------|---------|
| Weblogic application                     | RHEL 6                | Success |
| Weblogic application                     | RHEL 7                | **Fail:** Server not found in Kerberos database |
| Windows Explorer                         | RHEL 6                | Success |
| Windows Explorer                         | RHEL 7                | Success |

All tests done using the same user credentials.

The trace results from Weblogic (after setting the java system property sun.security.krb5.debug to true) are:

KrbAsReqBuild: PREAUTH FAILED/REQ, re-send AS-REQ
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 18 17 16 23
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 18 17 16 23
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KrbAsReq creating message
getKDCFromDNS using UDP
>>> KrbKdcReq send: kdc=***************. UDP:88, timeout=30000, number of retries =3, #bytes=233
>>> KDCCommunication: kdc=***************. UDP: 88, timeout=30000,Attempt =1, #bytes=233
>>> KrbKdcReq send: #bytes read=100
>>> KrbKdcReq send: kdc=*****************. TCP:88, timeout=30000, number of retires =3, #bytes=233
>>> KDCCommunication: kdc=****************. TCP:88, timeout=30000,Attempt =1, #bytes=233
>>>DEBUG: TCPClient reading 2695 bytes
>>>KrbKdcReq send: #bytes read=2695
>>>KdcAccessibility: remove **********************.:88
>>>Etype: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>>KrbAsRep cons in KrbAsReq.getReply ******
Found ticket for ******@******** to go to krbtgt/******@****** expiring on ******
Entered Krb5Context.initSecContext with state=STATE_NEW
Found ticket for ******@******** to go to krbtgt/******@****** expiring on ******
Service ticket not found in the subject
>>> Credentials serviceCredsSingle: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 18 17 16 23
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> CksumType: sun.security.krb5.internal.crypto.HmacSha1Aes256CksumType
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
getKDCFromDNS using UDP
>>> KrbKdcReq send: kdc=************. TCP:88, timeout=30000, number of retries =3, #bytes=2633
>>> KDCCommunication: kdc=************. TCP:88, timeout=30000,Attempt =1, #bytes=2633
>>>DEBUG: TCPClient reading 104 bytes
>>> KrbKdcReq send: #bytes read=104
>>> KdcAccessibility: remove *************.:88
>>> KDCRep: init() encoding tag is 126 req type is 13
>>>KRBError:
        STime is **********
        suSec is **********
        error code is 7
        error Message is Server not found in Kerberos database
        sname is cifs/***********@***********
        msgType is 30

Searching for "Server not found in Kerberos database" yields a number of possibilities (DNS seems to be most common suggestion, other answers have suggested SPN registrations, TLS certs, not using FQDN, invalid host to realm mapping, host not part of domain, IPV4 vs IPV6)

The network admins say DNS is correct, which would appear to be substantiated by the fact that Windows explorer can connect to the RHEL 7 server just fine. But neither am I prepared to just blame the java code, since it does successfully connect to the RHEL 6 server.

I'm having difficulty finding a clear explanation of what Kerberos entries need to be configured where.

Is this "Kerberos database" on the Windows 10 machine, the fileserver, or the Active Directory KDC? Or are there multiple copies of this Kerberos database that each need entries?

Edit - Additional Details I've learned a few new things and can provide some additional details.

The following are all from a Window's command prompt on the Developer machine.

Realm is determined by:

C:\>echo %userdnsdomain%
DC1.DC2.DC3

For the successful connection (RHEL 6 server)

C:\>powershell Get-ADComputer server1

DistinguishedName : CN=SERVER1,OU=ou1,OU=ou2,OU=ou3,DC=dc1,DC=dc2,DC=dc3
DNSHostName       : server1.dc2.dc3
Enabled           : True
Name              : SERVER1
ObjectClass       : Computer
ObjectGUID        : aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
SamAccountName    : server1$
SID               : ************************************
UserPrincipalName :

C:\>setspn -L server1
Registered ServicePrincipalNames for CN=SERVER1,OU=ou1,OU=ou2,OU=ou3,DC=dc1,DC=dc2, DC=dc3:
        HOST/server1.dc2.dc3
        HOST/SERVER1

C:\>nslookup server1
Server:  aa1.dc2.dc3
Address:  123.456.789.01

Name:    server1.dc2.dc3
Address:  123.456.7.890

C:\>nslookup 123.456.7.890
Server:  aa1.dc2.dc3
Address:  123.456.789.01

Name:    server1.dc2.dc3
Address:  123.456.7.890 

For the failing connection (RHEL 7 server)

C:\>powershell Get-ADComputer server2

DistinguishedName : CN=SERVER2,OU=ou4,DC=dc1,DC=dc2,DC=dc3
DNSHostName       : server2.dc1.dc2.dc3
Enabled           : True
Name              : SERVER2
ObjectClass       : Computer
ObjectGUID        : bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb
SamAccountName    : SERVER2$
SID               : ************************************
UserPrincipalName :

C:\>setspn -L server2
Registered ServicePrincipalNames for CN=SERVER2,OU=ou4,DC=dc1,DC=dc2,DC=dc3:
        RestrictedKrbHost/SERVER2
        HOST/SERVER2
        RestrictedKrbHost/SERVER2.dc1.dc2.dc3
        HOST/SERVER2.dc1.dc2.dc3

C:\>nslookup server2
Server:  aa1.dc2.dc3
Address:  123.456.789.01

Name:    server2.dc1.dc2.dc3
Address:  12.345.6.78

C:\>nslookup 12.345.6.78
Server:  aa1.dc2.dc3
Address:  123.456.789.01

Name:    server2.dc2.dc3
Address:  12.345.6.78

Comparing the Weblogic trace results for the success vs failure:

success:

>>> DEBUG: ----Credentials----
        client: [email protected]
        server: cifs/[email protected]
        ticket: sname: cifs/[email protected]

failure:

>>>KRBError:
  ...
         error code is 7
         error Message is is Server not found in Kerberos database
         sname is cifs/[email protected]
         msgType is 30

So I note a few things:

• The successful (RHEL 6) server omits "dc1" in a number of places (but not all)
• The successful and failing sname differ only in the server portion - they otherwise line up on the "dc" values.
• The two servers are in different Organizational Units (though I don't think this is the issue)
• There are a few difference in capitalization in the command results
• There appears to be a difference in the nslookup results for server2 depending on whether it is looked up by server name (includes dc1) or ip address (excludes dc1)

Solution

Finally got this working.

The critical pieces

• Reverse DNS must match Forward DNS
• The SPN (Service Principal Name) must be explicitly added in some cases - merely joining to the Active Directory Domain will not always register all the necessary HOST SPNs.
• In some cases, it may additionally be necessary to explicitly associate a server with a realm in the krb5.conf file. This appears to be applicable when you have multiple Kerberos realms involved. Note: you must restart WebLogic or reboot the server for changes to krb5.conf to take effect.

Summary

• I believe it is the case that the "Kerberos Database" lives with the "Key Distribution Center", which for Windows is Active Directory.

• Forward and Reverse dns entries (as demonstrated e.g. by nslookup in Windows) must match.

  • I believe the reverse DNS lookup will be used to get a servername which will then be used to query the KDC for the purposes of validating the server.

• There must be a two-way trust relationship between the linux server (maybe MIT Kerberos?) and the Active Directory. On the linux side, this is established with a keytab.

• The server portion of the Service Principal Name (SPN) may in some cases not match the DNS name reported by nslookup

  • This can still work if the SPN is added with the proper credentials. (edit - after further review, this was not indicated in our environment - I got mixed up on which setting was the original)
  • There should be an SPN (for samba, that would be a HOST spn) that coincides with the name reported by nslookup. In most cases, the correct HOST SPNs should be automatically added when the machine is joined to the domain, but in some cases it may be necessary to be explicitly add the SPN with e.g. setspn -A
  • If the server portion of the SPN cannot be resolved by DNS, that SPN will not be usable.

• DNSHostName as reported by powershell Get-ADComputer does not appear to be important for getting things to work. (Note this can be different from the results reported by nslookup)

• Service Principals can show up on Windows side or linux side - and they are not necessarily mirror images.

  • For example, on linux side we would see a cifs/[email protected] in the samba logs, but from Windows neither setspn -L server1 nor setspn -L server2 ever showed that SPN.

The process that fixed the issue:

1. We removed the linux server from Active Directory

2. We modified the dns entries so that using nslookup by name or ip returned the same results

3. We recreated the keytab in linux

   • This allowed linux to trust Active Directory, but not vice versa

4. We rejoined the linux server to Active Directory

   • at this stage, we still could not connect in java, but we could connect in Windows Explorer
   • at this stage setspn -L server2 contained the SPN HOST/server2.dc1.dc2.dc3 but did not have the SPN HOST/server2.dc2.dc3

5. We added the SPN using setspn from a Windows command prompt.

   • This requires domain admin or delegated permissions (I think for the machine principal)
   • setspn -A HOST/server2.dc2.dc3 server2$
   • The above command creates a new service principal using the machine account credentials

Uncertainties

Still not entirely clear why Windows Explorer always worked, but I do note that Java has its own portion of Kerberos implementation - so that may play role in the requirement for the additional SPN to be explicitly registered.

It is a bit interesting that we actually found that there were two different Kerberos Realms involved.

1. DC1.DC2.DC3 as determined from the Windows command echo %userdnsdomain%
2. ccc.dc2.dc3 as determined form the linux krb5.conf

However, once we added the final SPN, we could connect using either Kerberos Realm (case-sensitive).

Final Configuration

The final configuration that worked was as follows. Some of the spns are probably unnecessary (e.g I don't think the RestrictedKrbHost spns are necessary, because server1 doesn't have them), and I'm pretty sure that DNSHostName as determent by Get-ADComputer is irrelevant to this issue (as it did not change form the non-working and working configurations.)

C:\> echo %userdnsdomain%
DC1.DC2.DC3

C:\> powershell Get-ADComputer server2
DistinguishedName : CN=SERVER2,OU=ou4,DC=dc1,DC=dc2,DC=dc3
DNSHostName       : server2.dc1.dc2.dc3
Enabled           : True
Name              : SERVER2
ObjectClass       : computer
ObjectGUID        : bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb
SamAccountName    : SERVER2$
SID               : ************************************
UserPrincipalName :

C:\>setspn -L server2
Registered ServicePrincipalNames for CN=SERVER2,OU=ou4,DC=dc1,DC=dc2,DC=dc3:
        HOST/server2.dc2.dc3
        RestrictedKrbHost/SERVER2
        HOST/SERVER2
        RestrictedKrbHost/SERVER2.dc1.dc2.dc3
        HOST/SERVER2.dc1.dc2.dc3

C:\>nslookup server2
Server:  aa1.dc2.dc3
Address:  123.456.789.01

Name:    server2.dc2.dc3
Address:  12.345.6.78

C:\>nslookup 12.345.6.78
Server:  aa1.dc2.dc3
Address:  123.456.789.01

Name:    server2.dc2.dc3
Address:  12.345.6.78

In our case, when we later deployed the application to WebLogic running on RHEL7:

• we again got "Server not found in Kerberos database" when we specified the user in the "linux" realm ccc.dc2.dc3 running MIT Kerberos

• we got "No Service Creds" when we specified the user in the "Active Directory" realm DC1.DC2.DC3.

  • The solution was to modify /etc/krb5.conf on the linux weblogic server to explicitly associate the fileserver with the Active Directory realm, e.g.:

  [domain_realm]
  server2.dc2.dc3 = DC1.DC2.DC3
  

Comments

[Nathan]

I don't think have permissions to see that level of information - however, since Windows explorer does connect to the server, wouldn't that indicate that the server is registered in the AD?

[Nathan]

Looked into available powershell modules, and using Get-ADComputer I can see that both the RHEL 6 and RHEL 7 servers return information. But I notice that the DNSHostName for the RHEL 7 server has an extra component. So I'm going to try to followup on your hint about the DNS name differing from the AD domain name.