Security error when adding a secondary ADFS Server

Question

I have created an ADFS server according to the guide on technet. However, when attempting to add a secondary ADFS server using the latter part of this guide on technet, the process fails.

PS > Import-Module ADFS
PS > $serviceAccountCredential = Get-Credential -Message "Enter the credential for the Federation Service Account."
PS > Add-AdfsFarmNode `
>> -CertificateThumbprint:"REDACTED" `
>> -OverwriteConfiguration:$true `
>> -PrimaryComputerName:"awsfed01.ad.redacted.com" `
>> -ServiceAccountCredential:$serviceAccountCredential
>>
Add-AdfsFarmNode : MSIS7711: PolicyOperationFault
At line:1 char:1
+ Add-AdfsFarmNode `
+ ~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Add-AdfsFarmNode], FaultException
    + FullyQualifiedErrorId : DeploymentTask,Microsoft.IdentityServer.Deployment.Commands.JoinFarmCommand


Message                                 Context                                                                  Status
-------                                 -------                                                                  ------
Unable to synchronize local database... DeploymentTask                                                            Error

The following errors now appear in the event log on the server I am attempting to configure every five minutes:

Source: AD FS, Event ID 344:

There was an error doing synchronization. Synchronization of data from the primary federation server to a secondary federation server did not occur. 

Source: AD FS, Event ID 345:

There was a communication error during AD FS configuration database synchronization. Synchronization of data from the primary federation server to a secondary federation server did not occur. 

Additional Data 

Master Name : awsfed01.ad.redacted.com 
Endpoint Uri : http://awsfed01.ad.redacted.com/adfs/services/policystoretransfer 
Exception details: 
System.ServiceModel.Security.SecurityNegotiationException: The caller was not authenticated by the service. ---> System.ServiceModel.FaultException: The request for security token could not be satisfied because authentication failed.
   at System.ServiceModel.Security.SecurityUtils.ThrowIfNegotiationFault(Message message, EndpointAddress target)
   at System.ServiceModel.Security.SspiNegotiationTokenProvider.GetNextOutgoingMessageBody(Message incomingMessage, SspiNegotiationTokenProviderState sspiState)

The primary ADFS server's Security Audit Log contains an audit failures every time the secondary attempts to connect with the following details:

An account failed to log on.

Subject:
    Security ID:        NULL SID
    Account Name:       -
    Account Domain:     -
    Logon ID:       0x0

Logon Type:         3

Account For Which Logon Failed:
    Security ID:        NULL SID
    Account Name:       msa-adfs$
    Account Domain:     RDC

Failure Information:
    Failure Reason:     An Error occured during Logon.
    Status:         0x80090302
    Sub Status:     0xC0000418

Process Information:
    Caller Process ID:  0x0
    Caller Process Name:    -

Network Information:
    Workstation Name:   AWSFED20
    Source Network Address: -
    Source Port:        -

Detailed Authentication Information:
    Logon Process:      NtLmSsp 
    Authentication Package: NTLM
    Transited Services: -
    Package Name (NTLM only):   -
    Key Length:     0

Environment details

• I am using Windows Server 2012 R2 with ADFS 3.0
• My domain is ad.redacted.com (or RDC\). Additionally, I have redacted.com and redacted.co.uk as UPNs.
• I am using the WID database, not an external SQL server.
• I used a manually created a gMSA service account for my primary ADFS installation named RDC\msa-adfs$ - and am using this same account when trying to configure the secondary ADFS server. I configured its SPN - in accordance with various sources around the internet - to the following:
  • host/adfs.ad.redacted.com
  • http/adfs
  • host/adfs.ad.redacted.com
  • http/adfs
• The primary ADFS server's name is awsfed01.ad.redacted.com
• The secondary ADFS server - that I'm trying to configure - name is awsfed02.ad.redacted.com
• There is a DNS CNAME record for adfs.ad.redacted.com pointing at awsfed01.ad.redacted.com
• I used a SSL certificate signed by an external CA, with adfs.ad.redacted.com as the primary domain, and the following Subject Alternative Names:
  • adfs.ad.redacted.com
  • enterpriseregistration.ad.redacted.com
  • enterpriseregistration.redacted.com
  • enterpriseregistration.redacted.co.uk
• Both ADFS servers are configured with the Windows firewall enabled, but the network is configured to permit all traffic between them

I have tried configuring this multiple times from a blank server. Each time, the secondary ADFS server fails in the same way with the same error message.

Update: Reproduce with PowerShell

To try and reproduce this as reliably as possible, I have recreated what I'm doing with PowerShell.

Prerequisites: * Domain controller awsdc01 for domain ad.redacted.com aka RDC\ * Two federation servers: awsfed10, awsfed20 in a group named ADFS Servers

On awsdc01:

   New-ADServiceAccount -Name msa-adfs `
       -DNSHostName adfs.ad.redacted.com `
       -PrincipalsAllowedToRetrieveManagedPassword "ADFS Servers" 
       -ServicePrincipalNames "http/adfs.ad.redacted.com"

Executed successfully.

On awsfed10:

   Install-WindowsFeature adfs-federation –IncludeManagementTools
   Add-WindowsFeature RSAT-AD-PowerShell

   $password = ConvertTo-SecureString -String "Redacted" -Force -AsPlainText
   Import-PfxCertificate -FilePath C:\files\cert.pfx  cert:\localMachine\my -Password $password

   Import-Module ActiveDirectory
   Import-Module ADFS

   Install-ADServiceAccount msa-adfs
   Install-AdfsFarm -CertificateThumbprint:"XXX" -FederationServiceName:"adfs.ad.redacted.com" -GroupServiceAccountIdentifier RDC\msa-adfs$
   Initialize-ADDeviceRegistration -ServiceAccountName RDC\msa-adfs$
   Enable-AdfsDeviceRegistration

All executed successfully.

On awsfed20:

  Install-WindowsFeature adfs-federation –IncludeManagementTools
   Add-WindowsFeature RSAT-AD-PowerShell

   $password = ConvertTo-SecureString -String "Redacted" -Force -AsPlainText
   Import-PfxCertificate -FilePath C:\files\cert.pfx  cert:\localMachine\my -Password $password

   Import-Module ActiveDirectory
   Import-Module ADFS

   Install-ADServiceAccount msa-adfs
   Install-AdfsFarm -CertificateThumbprint:"XXX" -PrimaryComputerName:"awsfed10.ad.redacted.com" -GroupServiceAccountIdentifier RDC\msa-adfs$

Failed with the same errors as above.

Solution

The GMSA must be created with a host SPN (not http) based on the FederationServiceName parameter. So if you use adfs.ad.redacted.com then the SPN will be host/adfs.ad.redacted.com.

You should not be using CNAME entries to point adfs.ad.redacted.com to individual servers. That's going to cause Kerberos authentication issues as explained in https://blogs.technet.microsoft.com/askds/2009/06/22/internet-explorer-behaviors-with-kerberos-authentication/ scenario 2.

I assume you have some load balancing in use. Therefore the adfs.ad.redacted.com should resolve to the virtual IP of the load balancer that sits in front of the AD FS farm nodes.

Is NTLM blocking in use? Because it appears so. https://blogs.technet.microsoft.com/askds/2009/10/08/ntlm-blocking-and-you-application-analysis-and-auditing-methodologies-in-windows-7/ has more details of NTLM blocking.

Just use NTLM auditing for now (don't block) and retry adding the 2nd node after correcting the configuration as suggested earlier.

If you are still having issues, I suggest raising a support case with Microsoft.

Comments

[Cameron]

The problem was NTLM blocking. Although there was no GPO with it set, at some point an NTLM blocking GPO must have been configured, and it appears that the settings to not revert to default on any existing servers after you remove the GPO.