All DCs fail DNS:RReg test, Reports SRV Records Missing on PDC - They are present
This was resolved by removing IPv6 on the two DCs that had it running, and also by re-arranging the DNS configuration on the Network Cards.
DC2008S-0
Ethernet adapter Local Area Connection 2:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection #2
Physical Address. . . . . . . . . : 00-0C-29-9A-77-BA
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
IPv4 Address. . . . . . . . . . . : 10.1.1.3(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.1.1.1
DNS Servers . . . . . . . . . . . : 10.1.1.27
10.1.1.3
NetBIOS over Tcpip. . . . . . . . : Enabled
DC2008E-0
Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : vmxnet3 Ethernet Adapter
Physical Address. . . . . . . . . : 00-50-56-12-34-56
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
IPv4 Address. . . . . . . . . . . : 10.1.1.27(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.1.1.1
DNS Servers . . . . . . . . . . . : 10.1.1.28
10.1.1.27
NetBIOS over Tcpip. . . . . . . . : Enabled
DC2008E-1
Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
Physical Address. . . . . . . . . : 00-0C-29-75-FF-46
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
IPv4 Address. . . . . . . . . . . : 10.1.1.28(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.1.1.1
DNS Servers . . . . . . . . . . . : 10.1.1.27
10.1.1.28
NetBIOS over Tcpip. . . . . . . . : Enabled
Related videos on Youtube
03 : 38
05 : 11
02 : 59
02 : 48
03 : 05
Sam K
Former WISP/FISP NOC Technician Currently Network Admin (but really a glorified SysAdmin).
Updated on September 18, 2022Comments
-
Sam K over 1 year
Background
We are currently in the process of doing several Domain Controller upgrades. Before I started here a previous Admin had started the process of moving our DCs from 2008 R2 Standard to 2008 R2 Enterprise. There was a PDC, DC2008S-0, and one additional DC, DC2008E-1 running. There was a 3rd 2008 Enterprise DC that was sitting on a VM that was shutdown. ALL of this was a leftover project from upgrading the DCs from 2003. The previous admin felt that Standard was not enough for the DCs and that those licenses were purchased in error, so after floating two standard DCs the enterprise DC was added and a standard DC was demoted.
The Enterprise DC was not replicating SYSVOL at all. The MSDCS zone was missing on the Enterprise DC as well. There was also some meta-data cleanup that had to occur for the fully tombstoned DC (the spare 2008E that was sitting on a shutdown VM). After quite a bit of troubleshooting we did an authoritative restore from the PDC. Afterwards SYSVOL appeared to be replicating properly, we added MSDCS manually and all the records pulled in. This was probably 8 or 9 months ago. Everything has been working smoothly since; logins, gpo replication, new gpos, new AD accounts - as well as a Hybrid migration to O365, and all the AD sync and Dir sync stuff worked great as well.
After that time period we've returned to this DC project. My task list was as follows:
Update the functional level of the Domain and Forest from 2003 to 2008 (this included migrating from FRS to DFRS) Nuke the shutdown 2nd Enterprise DC, reinstall it, give it a DC role and add it to the domain. Move FSMO roles, etc to the first Enterprise DC and make it the PDC. Decommission the Standard DC.
I am on the precipice of decommissioning the standard DC when this DNS RReg issue came to light. I don't believe it existed after the replication of SYSVOL and AD and DNS items, but I could be wrong.
Current Issue
All of our DCs are failing the RReg test from DCDIAG.
This is our only failure when checking DC health with DCDIAG against each DC. When running the gui AD Replication Status Tool v1.0 as well as two PS scripts from TechNET, the AD and SYSVOL Replication/Latency Convergence Checks.
Here is the failure output from DCDIAG DNS tests
Summary of DNS test results: Auth Basc Forw Del Dyn RReg Ext _________________________________________________________________ Domain: domain.com DC2008S-0 PASS PASS PASS PASS PASS FAIL n/a DC2008E-0 PASS PASS PASS PASS PASS FAIL n/a DC2008E-1 PASS PASS PASS PASS PASS FAIL n/a Total Time taken to test all the DCs:2 min. 55 sec. ......................... domain.com failed test DNSThe failures are all in regards to a single CNAME, and single A record, and multiple SRV records on the new PDC DC2008E-0
Starting test: DNS Test results for domain controllers: DC: DC2008E-0.domain.com Domain: domain.com TEST: Records registration (RReg) Network Adapter [00000007] vmxnet3 Ethernet Adapter: Warning: Missing CNAME record at DNS server 10.1.1.27: 7ae71958-74b2-4dc3-bf0e-224ec881bafa._msdcs.domain.com Warning: Missing A record at DNS server 10.1.1.27: DC2008E-0.domain.com Error: Missing SRV record at DNS server 10.1.1.27: _ldap._tcp.domain.com Error: Missing SRV record at DNS server 10.1.1.27: _ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com Error: Missing SRV record at DNS server 10.1.1.27: _kerberos._tcp.dc._msdcs.domain.com Error: Missing SRV record at DNS server 10.1.1.27: _ldap._tcp.dc._msdcs.domain.com Error: Missing SRV record at DNS server 10.1.1.27: _kerberos._tcp.domain.com Error: Missing SRV record at DNS server 10.1.1.27: _kerberos._udp.domain.com Error: Missing SRV record at DNS server 10.1.1.27: _kpasswd._tcp.domain.com Error: Missing SRV record at DNS server 10.1.1.27: _ldap._tcp.siteName._sites.domain.com Error: Missing SRV record at DNS server 10.1.1.27: _kerberos._tcp.siteName._sites.dc._msdcs.domain.com Error: Missing SRV record at DNS server 10.1.1.27: _ldap._tcp.siteName._sites.dc._msdcs.domain.com Error: Missing SRV record at DNS server 10.1.1.27: _kerberos._tcp.siteName._sites.domain.com Error: Missing SRV record at DNS server 10.1.1.27: _ldap._tcp.gc._msdcs.domain.com Warning: Missing A record at DNS server 10.1.1.27: gc._msdcs.domain.com Error: Missing SRV record at DNS server 10.1.1.27: _gc._tcp.siteName._sites.domain.com Error: Missing SRV record at DNS server 10.1.1.27: _ldap._tcp.siteName._sites.gc._msdcs.domain.com Error: Missing SRV record at DNS server 10.1.1.27: _ldap._tcp.pdc._msdcs.domain.com Error: Record registrations cannot be found for all the network adapters Summary of DNS test results: Auth Basc Forw Del Dyn RReg Ext _________________________________________________________________ Domain: domain.com DC2008E-0 PASS PASS PASS PASS PASS FAIL n/a ......................... domain.com failed test DNS
Investigation So Far
I have manually inspected all these records and I can confirm that all the records exist on all my DCs.
I have also compared the MCDCS zone on all the DCs and all other records match.
The Zone Serial number on the SOA match on all DCs, this is true for all zones on all DCs as well, not just the MCDCS zone.
I'm not sure if this is best way to express that I can find the records manually, but I ran NSLOOKUP against all three DCs for one of the records listed above and it appears that it is found on all three.
c:\Users\userName\Desktop\replication>nslookup -type=SRV _ldap._tcp.pdc._msdcs.domain.com Server: DC2008E-0.domain.com Address: 10.1.1.27 _ldap._tcp.pdc._msdcs.domain.com SRV service location: priority = 0 weight = 100 port = 389 svr hostname = DC2008E-0.domain.com DC2008E-0.domain.com internet address = 10.1.1.27 c:\Users\userName\Desktop\replication>nslookup -type=SRV _ldap._tcp.pdc._msdcs.domain.com DC2008S-0 Server: DC2008S-0.domain.com Address: 10.1.1.3 _ldap._tcp.pdc._msdcs.domain.com SRV service location: priority = 0 weight = 100 port = 389 svr hostname = DC2008E-0.domain.com DC2008E-0.domain.com internet address = 10.1.1.27 c:\Users\userName\Desktop\replication>nslookup -type=SRV _ldap._tcp.pdc._msdcs.domain.com DC2008E-1 Server: DC2008E-1.domain.com Address: 10.1.1.28 _ldap._tcp.pdc._msdcs.domain.com SRV service location: priority = 0 weight = 100 port = 389 svr hostname = DC2008E-0.domain.com DC2008E-0.domain.com internet address = 10.1.1.27I also inspected the CNAME records from the root of the _MSDCS zone, this is the only place I found things to be odd. The records themselves are all 100% correct, and the permissions look correct - at least, I should say, they all match between the 3 CNAME records and how each DC views the CNAME records. However, the Owners are set differently. DC2008S-0's record is owned by SYSTEM, DC2008E-0's record is owned by DC2008E-0$, and DC2008E-1's record is owned by DC2008E-1$ (DOMAIN\DC2008E-1$). This is the same no matter which DC I'm looking at the record on.
I don't know if that is pertinent at all, but it seems to be the ONLY thing I can find that doesn't match and/or follow the same pattern. It may very well be a misnomer.
From DC2008E-0 I have also run ipconfig /registerdns and no errors were reported to the Event Viewer. I have also run nltest /dsregdns
C:\Windows\system32>nltest /dsregdns Flags: 0 Connection Status = 0 0x0 NERR_Success The command completed successfullyThis does not appear to fix the issue.
Further Investigate
It would appear that I had overlooked some output from the full DCDIAG set of tests I was running. There are some more specific errors being reported. And there's also much more granularity when it comes to how the DNS SRV records are being reported.
I'll post the relevant output from dcdiag.exe /V /C /D /E /s:dc0 (Actually, I have to post snippets as I'm hitting the character limit)
DC: DC2008S-0.domain.com Domain: domain.com Adapter [00000012] Intel(R) PRO/1000 MT Network Connection:
MAC address is 00:0C:29:9A:77:BA IP Address is static IP address: 10.1.1.3 DNS servers: 10.1.1.3 (DC2008S-0) [Valid] 10.1.1.27 (DC2008E-0) [Valid] 127.0.0.1 (DC2008S-0) [Valid] The A host record(s) for this DC was found The SOA record for the Active Directory zone was found The Active Directory zone on this DC/DNS server was found primary Root zone on this DC/DNS server was not found TEST: Records registration (RReg) Network Adapter [00000012] Intel(R) PRO/1000 MT Network Connection: Matching CNAME record found at DNS server 10.1.1.3: f11ae1a7-ab57-47d9-bf47-11eca1e33936._msdcs.domain.com Matching A record found at DNS server 10.1.1.3: DC2008S-0.domain.com Matching SRV record found at DNS server 10.1.1.3: _ldap._tcp.domain.com Matching SRV record found at DNS server 10.1.1.3: _ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com[...]
Matching CNAME record found at DNS server 10.1.1.27: f11ae1a7-ab57-47d9-bf47-11eca1e33936._msdcs.domain.com Matching A record found at DNS server 10.1.1.27: DC2008S-0.domain.com Matching SRV record found at DNS server 10.1.1.27: _ldap._tcp.domain.com Matching SRV record found at DNS server 10.1.1.27: _ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com[...]
Warning: Missing CNAME record at DNS server 10.1.1.3: f11ae1a7-ab57-47d9-bf47-11eca1e33936._msdcs.domain.com [Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)] Warning: Missing A record at DNS server 10.1.1.3: DC2008S-0.domain.com [Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)] Error: Missing SRV record at DNS server 10.1.1.3: _ldap._tcp.domain.com [Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)] Error: Missing SRV record at DNS server 10.1.1.3: _ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com [Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)] Error: Record registrations cannot be found for all the network adapters Total query time:0 min. 0 sec.. Total RPC connection time:0 min. 0 sec. Total WMI connection time:1 min. 3 sec. Total Netuse connection time:0 min. 0 sec.[...]
DC: DC2008E-0.domain.com Domain: domain.com Network adapters information: Adapter [00000007] vmxnet3 Ethernet Adapter: MAC address is 00:50:56:12:34:56 IP Address is static IP address: 10.1.1.27, fe80::3464:a8c8:13fa:7116 DNS servers: 10.1.1.3 (DC2008S-0) [Valid] 10.1.1.27 (DC2008E-0) [Valid] 127.0.0.1 (DC2008E-0) [Valid] The A host record(s) for this DC was found The SOA record for the Active Directory zone was found The Active Directory zone on this DC/DNS server was found primary Root zone on this DC/DNS server was not found TEST: Records registration (RReg) Network Adapter [00000007] vmxnet3 Ethernet Adapter: Matching CNAME record found at DNS server 10.1.1.3: 7ae71958-74b2-4dc3-bf0e-224ec881bafa._msdcs.domain.com Matching A record found at DNS server 10.1.1.3: DC2008E-0.domain.com Matching SRV record found at DNS server 10.1.1.3: _ldap._tcp.domain.com Matching SRV record found at DNS server 10.1.1.3: _ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com[...]
Matching CNAME record found at DNS server 10.1.1.27: 7ae71958-74b2-4dc3-bf0e-224ec881bafa._msdcs.domain.com Matching A record found at DNS server 10.1.1.27: DC2008E-0.domain.com Matching SRV record found at DNS server 10.1.1.27: _ldap._tcp.domain.com Matching SRV record found at DNS server 10.1.1.27: _ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com[...]
Warning: Missing CNAME record at DNS server 10.1.1.27: 7ae71958-74b2-4dc3-bf0e-224ec881bafa._msdcs.domain.com [Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)] Warning: Missing A record at DNS server 10.1.1.27: DC2008E-0.domain.com [Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)] Error: Missing SRV record at DNS server 10.1.1.27: _ldap._tcp.domain.com [Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)] Error: Missing SRV record at DNS server 10.1.1.27: _ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com [Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)][...]
Error: Record registrations cannot be found for all the network adapters Total query time:0 min. 4 sec.. Total RPC connection time:0 min. 0 sec. Total WMI connection time:1 min. 3 sec. Total Netuse connection time:0 min. 0 sec.[...]
DC: DC2008E-1.domain.com Domain: domain.com Network adapters information: Adapter [00000007] Intel(R) PRO/1000 MT Network Connection: MAC address is 00:0C:29:75:FF:46 IP Address is static IP address: 10.1.1.28, fe80::b81a:c109:24a0:9d3d DNS servers: 10.1.1.3 (DC2008S-0) [Valid] 10.1.1.27 (DC2008E-0) [Valid] 127.0.0.1 (DC2008E-1) [Valid] The A host record(s) for this DC was found The SOA record for the Active Directory zone was found The Active Directory zone on this DC/DNS server was found primary Root zone on this DC/DNS server was not found TEST: Records registration (RReg) Network Adapter [00000007] Intel(R) PRO/1000 MT Network Connection: Matching CNAME record found at DNS server 10.1.1.3: eafe6486-f76c-4900-8a20-46404fdbae57._msdcs.domain.com Matching A record found at DNS server 10.1.1.3: DC2008E-1.domain.com Matching SRV record found at DNS server 10.1.1.3: _ldap._tcp.domain.com Matching SRV record found at DNS server 10.1.1.3: _ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com[...]
Matching CNAME record found at DNS server 10.1.1.27: eafe6486-f76c-4900-8a20-46404fdbae57._msdcs.domain.com Matching A record found at DNS server 10.1.1.27: DC2008E-1.domain.com Matching SRV record found at DNS server 10.1.1.27: _ldap._tcp.domain.com Matching SRV record found at DNS server 10.1.1.27: _ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com[...]
Warning: Missing CNAME record at DNS server 10.1.1.28: eafe6486-f76c-4900-8a20-46404fdbae57._msdcs.domain.com [Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)] Warning: Missing A record at DNS server 10.1.1.28: DC2008E-1.domain.com [Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)] Error: Missing SRV record at DNS server 10.1.1.28: _ldap._tcp.domain.com [Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)] Error: Missing SRV record at DNS server 10.1.1.28: _ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com [Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)] Error: Record registrations cannot be found for all the network adapters Total query time:0 min. 0 sec.. Total RPC connection time:0 min. 0 sec. Total WMI connection time:0 min. 44 sec. Total Netuse connection time:0 min. 0 sec.So it appears that there may be something going on with the NIC setup? That's where I'm starting to lean now.
NIC Configs
DC2008S-0
Ethernet adapter Local Area Connection 2: Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection #2 Physical Address. . . . . . . . . : 00-0C-29-9A-77-BA DHCP Enabled. . . . . . . . . . . : No Autoconfiguration Enabled . . . . : Yes IPv4 Address. . . . . . . . . . . : 10.1.1.3(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.255.0 Default Gateway . . . . . . . . . : 10.1.1.1 DNS Servers . . . . . . . . . . . : 10.1.1.3 10.1.1.27 127.0.0.1 NetBIOS over Tcpip. . . . . . . . : EnabledDC2008E-0
Ethernet adapter Local Area Connection: Connection-specific DNS Suffix . : Description . . . . . . . . . . . : vmxnet3 Ethernet Adapter Physical Address. . . . . . . . . : 00-50-56-12-34-56 DHCP Enabled. . . . . . . . . . . : No Autoconfiguration Enabled . . . . : Yes Link-local IPv6 Address . . . . . : fe80::3464:a8c8:13fa:7116%15(Preferred) IPv4 Address. . . . . . . . . . . : 10.1.1.27(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.255.0 Default Gateway . . . . . . . . . : 10.1.1.1 DHCPv6 IAID . . . . . . . . . . . : 335564886 DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-22-4A-CD-9F-00-50-56-12-34-56 DNS Servers . . . . . . . . . . . : ::1 10.1.1.3 10.1.1.27 127.0.0.1 NetBIOS over Tcpip. . . . . . . . : EnabledDC2008E-1
Ethernet adapter Local Area Connection: Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection Physical Address. . . . . . . . . : 00-0C-29-75-FF-46 DHCP Enabled. . . . . . . . . . . : No Autoconfiguration Enabled . . . . : Yes Link-local IPv6 Address . . . . . : fe80::b81a:c109:24a0:9d3d%10(Preferred) IPv4 Address. . . . . . . . . . . : 10.1.1.28(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.255.0 Default Gateway . . . . . . . . . : 10.1.1.1 DHCPv6 IAID . . . . . . . . . . . : 251661353 DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-23-34-D6-43-00-0C-29-75-FF-46 DNS Servers . . . . . . . . . . . : ::1 10.1.1.3 10.1.1.27 127.0.0.1 NetBIOS over Tcpip. . . . . . . . : Enabled-
duct_tape_coder over 5 yearsWhat DNS servers are the NICs for the DCs configured to use? I generally force a DC to use other DCs as their primary DNS before themselves. For example, DC1 will point to DC2 and then DC1 and DC2 will point to DC1 and then DC2. This helps prevent a boot up issue where AD fails because it loaded before the DNS service loads.
-
Sam K over 5 yearsI haven't updated the DNS servers since I started this whole thing as of yet. So they still have the same settings they had before which is as follows: The primary on all 3 DC is 10.1.1.3 (DC2008S-0) and the secondary is 10.1.1.27 (DC2008E-0), and then they all have an entry for 127.0.0.1 as the tertiary. I am questioning that setup, but I'm not sure if that expressly leads into the RReg failure.
-
Sam K over 5 yearsI added the original output from the NICs above. After reviewing some additional articles on best practices for DNS settings on the NIC for a DC I think that do the loopback using the loopback address is wrong. I've adjusted the DNS settings so that the first DNS server is another DC, the second being the server's own static ip address as a 'loopback' instead of the 127.0.0.1. Will give everything some time in case records need to replicate or converge and then run dcdiag again.
-
duct_tape_coder over 5 yearsMake sure to reboot each DC one by one (wait until fully booted before rebooting the next) after making the DNS change to ensure they've cleaned themselves up.
-
duct_tape_coder over 5 yearsBTW, since we're going down my usual book of AD tricks, have you made sure everyone is time synchronized? Use
w32tm /monitorto check time synchronization between DCs. -
Sam K over 5 yearsAhh, I posted the 'answer' below, but that is good advice, I will make sure to do a reboot, but DCDIAG is now showing everything is good after the changes I outlined below - thanks again for the help!
-
