Background
We are currently in the process of doing several Domain Controller upgrades. Before I started here a previous Admin had started the process of moving our DCs from 2008 R2 Standard to 2008 R2 Enterprise. There was a PDC, DC2008S-0, and one additional DC, DC2008E-1 running. There was a 3rd 2008 Enterprise DC that was sitting on a VM that was shutdown. ALL of this was a leftover project from upgrading the DCs from 2003. The previous admin felt that Standard was not enough for the DCs and that those licenses were purchased in error, so after floating two standard DCs the enterprise DC was added and a standard DC was demoted.
The Enterprise DC was not replicating SYSVOL at all. The MSDCS zone was missing on the Enterprise DC as well. There was also some meta-data cleanup that had to occur for the fully tombstoned DC (the spare 2008E that was sitting on a shutdown VM). After quite a bit of troubleshooting we did an authoritative restore from the PDC. Afterwards SYSVOL appeared to be replicating properly, we added MSDCS manually and all the records pulled in. This was probably 8 or 9 months ago. Everything has been working smoothly since; logins, gpo replication, new gpos, new AD accounts – as well as a Hybrid migration to O365, and all the AD sync and Dir sync stuff worked great as well.
After that time period we've returned to this DC project. My task list was as follows:
Update the functional level of the Domain and Forest from 2003 to 2008 (this included migrating from FRS to DFRS)
Nuke the shutdown 2nd Enterprise DC, reinstall it, give it a DC role and add it to the domain.
Move FSMO roles, etc to the first Enterprise DC and make it the PDC.
Decommission the Standard DC.
I am on the precipice of decommissioning the standard DC when this DNS RReg issue came to light. I don't believe it existed after the replication of SYSVOL and AD and DNS items, but I could be wrong.
Current Issue
All of our DCs are failing the RReg test from DCDIAG.
This is our only failure when checking DC health with DCDIAG against each DC. When running the gui AD Replication Status Tool v1.0 as well as two PS scripts from TechNET, the AD and SYSVOL Replication/Latency Convergence Checks.
Here is the failure output from DCDIAG DNS tests
Summary of DNS test results:
Auth Basc Forw Del Dyn RReg Ext
_________________________________________________________________
Domain: domain.com
DC2008S-0 PASS PASS PASS PASS PASS FAIL n/a
DC2008E-0 PASS PASS PASS PASS PASS FAIL n/a
DC2008E-1 PASS PASS PASS PASS PASS FAIL n/a
Total Time taken to test all the DCs:2 min. 55 sec.
......................... domain.com failed test DNS
The failures are all in regards to a single CNAME, and single A record, and multiple SRV records on the new PDC DC2008E-0
Starting test: DNS
Test results for domain controllers:
DC: DC2008E-0.domain.com
Domain: domain.com
TEST: Records registration (RReg)
Network Adapter [00000007] vmxnet3 Ethernet Adapter:
Warning:
Missing CNAME record at DNS server 10.1.1.27:
7ae71958-74b2-4dc3-bf0e-224ec881bafa._msdcs.domain.com
Warning:
Missing A record at DNS server 10.1.1.27:
DC2008E-0.domain.com
Error:
Missing SRV record at DNS server 10.1.1.27:
_ldap._tcp.domain.com
Error:
Missing SRV record at DNS server 10.1.1.27:
_ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com
Error:
Missing SRV record at DNS server 10.1.1.27:
_kerberos._tcp.dc._msdcs.domain.com
Error:
Missing SRV record at DNS server 10.1.1.27:
_ldap._tcp.dc._msdcs.domain.com
Error:
Missing SRV record at DNS server 10.1.1.27:
_kerberos._tcp.domain.com
Error:
Missing SRV record at DNS server 10.1.1.27:
_kerberos._udp.domain.com
Error:
Missing SRV record at DNS server 10.1.1.27:
_kpasswd._tcp.domain.com
Error:
Missing SRV record at DNS server 10.1.1.27:
_ldap._tcp.siteName._sites.domain.com
Error:
Missing SRV record at DNS server 10.1.1.27:
_kerberos._tcp.siteName._sites.dc._msdcs.domain.com
Error:
Missing SRV record at DNS server 10.1.1.27:
_ldap._tcp.siteName._sites.dc._msdcs.domain.com
Error:
Missing SRV record at DNS server 10.1.1.27:
_kerberos._tcp.siteName._sites.domain.com
Error:
Missing SRV record at DNS server 10.1.1.27:
_ldap._tcp.gc._msdcs.domain.com
Warning:
Missing A record at DNS server 10.1.1.27:
gc._msdcs.domain.com
Error:
Missing SRV record at DNS server 10.1.1.27:
_gc._tcp.siteName._sites.domain.com
Error:
Missing SRV record at DNS server 10.1.1.27:
_ldap._tcp.siteName._sites.gc._msdcs.domain.com
Error:
Missing SRV record at DNS server 10.1.1.27:
_ldap._tcp.pdc._msdcs.domain.com
Error: Record registrations cannot be found for all the network adapters
Summary of DNS test results:
Auth Basc Forw Del Dyn RReg Ext
_________________________________________________________________
Domain: domain.com
DC2008E-0 PASS PASS PASS PASS PASS FAIL n/a
......................... domain.com failed test DNS
Investigation So Far
I have manually inspected all these records and I can confirm that all the records exist on all my DCs.
I have also compared the MCDCS zone on all the DCs and all other records match.
The Zone Serial number on the SOA match on all DCs, this is true for all zones on all DCs as well, not just the MCDCS zone.
I'm not sure if this is best way to express that I can find the records manually, but I ran NSLOOKUP against all three DCs for one of the records listed above and it appears that it is found on all three.
c:\Users\userName\Desktop\replication>nslookup -type=SRV _ldap._tcp.pdc._msdcs.domain.com
Server: DC2008E-0.domain.com
Address: 10.1.1.27
_ldap._tcp.pdc._msdcs.domain.com SRV service location:
priority = 0
weight = 100
port = 389
svr hostname = DC2008E-0.domain.com
DC2008E-0.domain.com internet address = 10.1.1.27
c:\Users\userName\Desktop\replication>nslookup -type=SRV _ldap._tcp.pdc._msdcs.domain.com DC2008S-0
Server: DC2008S-0.domain.com
Address: 10.1.1.3
_ldap._tcp.pdc._msdcs.domain.com SRV service location:
priority = 0
weight = 100
port = 389
svr hostname = DC2008E-0.domain.com
DC2008E-0.domain.com internet address = 10.1.1.27
c:\Users\userName\Desktop\replication>nslookup -type=SRV _ldap._tcp.pdc._msdcs.domain.com DC2008E-1
Server: DC2008E-1.domain.com
Address: 10.1.1.28
_ldap._tcp.pdc._msdcs.domain.com SRV service location:
priority = 0
weight = 100
port = 389
svr hostname = DC2008E-0.domain.com
DC2008E-0.domain.com internet address = 10.1.1.27
I also inspected the CNAME records from the root of the _MSDCS zone, this is the only place I found things to be odd. The records themselves are all 100% correct, and the permissions look correct – at least, I should say, they all match between the 3 CNAME records and how each DC views the CNAME records. However, the Owners are set differently. DC2008S-0's record is owned by SYSTEM, DC2008E-0's record is owned by DC2008E-0$, and DC2008E-1's record is owned by DC2008E-1$ (DOMAIN\DC2008E-1$). This is the same no matter which DC I'm looking at the record on.
I don't know if that is pertinent at all, but it seems to be the ONLY thing I can find that doesn't match and/or follow the same pattern. It may very well be a misnomer.
From DC2008E-0 I have also run ipconfig /registerdns and no errors were reported to the Event Viewer. I have also run nltest /dsregdns
C:\Windows\system32>nltest /dsregdns
Flags: 0
Connection Status = 0 0x0 NERR_Success
The command completed successfully
This does not appear to fix the issue.
Further Investigate
It would appear that I had overlooked some output from the full DCDIAG set of tests I was running. There are some more specific errors being reported. And there's also much more granularity when it comes to how the DNS SRV records are being reported.
I'll post the relevant output from dcdiag.exe /V /C /D /E /s:dc0
(Actually, I have to post snippets as I'm hitting the character limit)
DC: DC2008S-0.domain.com
Domain: domain.com
Adapter [00000012] Intel(R) PRO/1000 MT Network Connection:
MAC address is 00:0C:29:9A:77:BA
IP Address is static
IP address: 10.1.1.3
DNS servers:
10.1.1.3 (DC2008S-0) [Valid]
10.1.1.27 (DC2008E-0) [Valid]
127.0.0.1 (DC2008S-0) [Valid]
The A host record(s) for this DC was found
The SOA record for the Active Directory zone was found
The Active Directory zone on this DC/DNS server was found primary
Root zone on this DC/DNS server was not found
TEST: Records registration (RReg)
Network Adapter
[00000012] Intel(R) PRO/1000 MT Network Connection:
Matching CNAME record found at DNS server 10.1.1.3:
f11ae1a7-ab57-47d9-bf47-11eca1e33936._msdcs.domain.com
Matching A record found at DNS server 10.1.1.3:
DC2008S-0.domain.com
Matching SRV record found at DNS server 10.1.1.3:
_ldap._tcp.domain.com
Matching SRV record found at DNS server 10.1.1.3:
_ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com
[…]
Matching CNAME record found at DNS server 10.1.1.27:
f11ae1a7-ab57-47d9-bf47-11eca1e33936._msdcs.domain.com
Matching A record found at DNS server 10.1.1.27:
DC2008S-0.domain.com
Matching SRV record found at DNS server 10.1.1.27:
_ldap._tcp.domain.com
Matching SRV record found at DNS server 10.1.1.27:
_ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com
[…]
Warning:
Missing CNAME record at DNS server 10.1.1.3:
f11ae1a7-ab57-47d9-bf47-11eca1e33936._msdcs.domain.com
[Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)]
Warning:
Missing A record at DNS server 10.1.1.3:
DC2008S-0.domain.com
[Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)]
Error:
Missing SRV record at DNS server 10.1.1.3:
_ldap._tcp.domain.com
[Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)]
Error:
Missing SRV record at DNS server 10.1.1.3:
_ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com
[Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)]
Error: Record registrations cannot be found for all the network
adapters
Total query time:0 min. 0 sec.. Total RPC connection
time:0 min. 0 sec.
Total WMI connection time:1 min. 3 sec. Total Netuse connection
time:0 min. 0 sec.
[…]
DC: DC2008E-0.domain.com
Domain: domain.com
Network adapters information:
Adapter [00000007] vmxnet3 Ethernet Adapter:
MAC address is 00:50:56:12:34:56
IP Address is static
IP address: 10.1.1.27, fe80::3464:a8c8:13fa:7116
DNS servers:
10.1.1.3 (DC2008S-0) [Valid]
10.1.1.27 (DC2008E-0) [Valid]
127.0.0.1 (DC2008E-0) [Valid]
The A host record(s) for this DC was found
The SOA record for the Active Directory zone was found
The Active Directory zone on this DC/DNS server was found primary
Root zone on this DC/DNS server was not found
TEST: Records registration (RReg)
Network Adapter [00000007] vmxnet3 Ethernet Adapter:
Matching CNAME record found at DNS server 10.1.1.3:
7ae71958-74b2-4dc3-bf0e-224ec881bafa._msdcs.domain.com
Matching A record found at DNS server 10.1.1.3:
DC2008E-0.domain.com
Matching SRV record found at DNS server 10.1.1.3:
_ldap._tcp.domain.com
Matching SRV record found at DNS server 10.1.1.3:
_ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com
[…]
Matching CNAME record found at DNS server 10.1.1.27:
7ae71958-74b2-4dc3-bf0e-224ec881bafa._msdcs.domain.com
Matching A record found at DNS server 10.1.1.27:
DC2008E-0.domain.com
Matching SRV record found at DNS server 10.1.1.27:
_ldap._tcp.domain.com
Matching SRV record found at DNS server 10.1.1.27:
_ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com
[…]
Warning:
Missing CNAME record at DNS server 10.1.1.27:
7ae71958-74b2-4dc3-bf0e-224ec881bafa._msdcs.domain.com
[Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)]
Warning:
Missing A record at DNS server 10.1.1.27:
DC2008E-0.domain.com
[Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)]
Error:
Missing SRV record at DNS server 10.1.1.27:
_ldap._tcp.domain.com
[Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)]
Error:
Missing SRV record at DNS server 10.1.1.27:
_ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com
[Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)]
[…]
Error: Record registrations cannot be found for all the network
adapters
Total query time:0 min. 4 sec.. Total RPC connection
time:0 min. 0 sec.
Total WMI connection time:1 min. 3 sec. Total Netuse connection
time:0 min. 0 sec.
[…]
DC: DC2008E-1.domain.com
Domain: domain.com
Network adapters information:
Adapter [00000007] Intel(R) PRO/1000 MT Network Connection:
MAC address is 00:0C:29:75:FF:46
IP Address is static
IP address: 10.1.1.28, fe80::b81a:c109:24a0:9d3d
DNS servers:
10.1.1.3 (DC2008S-0) [Valid]
10.1.1.27 (DC2008E-0) [Valid]
127.0.0.1 (DC2008E-1) [Valid]
The A host record(s) for this DC was found
The SOA record for the Active Directory zone was found
The Active Directory zone on this DC/DNS server was found primary
Root zone on this DC/DNS server was not found
TEST: Records registration (RReg)
Network Adapter
[00000007] Intel(R) PRO/1000 MT Network Connection:
Matching CNAME record found at DNS server 10.1.1.3:
eafe6486-f76c-4900-8a20-46404fdbae57._msdcs.domain.com
Matching A record found at DNS server 10.1.1.3:
DC2008E-1.domain.com
Matching SRV record found at DNS server 10.1.1.3:
_ldap._tcp.domain.com
Matching SRV record found at DNS server 10.1.1.3:
_ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com
[…]
Matching CNAME record found at DNS server 10.1.1.27:
eafe6486-f76c-4900-8a20-46404fdbae57._msdcs.domain.com
Matching A record found at DNS server 10.1.1.27:
DC2008E-1.domain.com
Matching SRV record found at DNS server 10.1.1.27:
_ldap._tcp.domain.com
Matching SRV record found at DNS server 10.1.1.27:
_ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com
[…]
Warning:
Missing CNAME record at DNS server 10.1.1.28:
eafe6486-f76c-4900-8a20-46404fdbae57._msdcs.domain.com
[Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)]
Warning:
Missing A record at DNS server 10.1.1.28:
DC2008E-1.domain.com
[Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)]
Error:
Missing SRV record at DNS server 10.1.1.28:
_ldap._tcp.domain.com
[Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)]
Error:
Missing SRV record at DNS server 10.1.1.28:
_ldap._tcp.5f315a51-10e4-4785-a4db-50312543bf35.domains._msdcs.domain.com
[Error details: 10054 (Type: Win32 - Description: An existing connection was forcibly closed by the remote host.)]
Error: Record registrations cannot be found for all the network
adapters
Total query time:0 min. 0 sec.. Total RPC connection
time:0 min. 0 sec.
Total WMI connection time:0 min. 44 sec. Total Netuse connection
time:0 min. 0 sec.
So it appears that there may be something going on with the NIC setup? That's where I'm starting to lean now.
NIC Configs
DC2008S-0
Ethernet adapter Local Area Connection 2:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection #2
Physical Address. . . . . . . . . : 00-0C-29-9A-77-BA
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
IPv4 Address. . . . . . . . . . . : 10.1.1.3(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.1.1.1
DNS Servers . . . . . . . . . . . : 10.1.1.3
10.1.1.27
127.0.0.1
NetBIOS over Tcpip. . . . . . . . : Enabled
DC2008E-0
Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : vmxnet3 Ethernet Adapter
Physical Address. . . . . . . . . : 00-50-56-12-34-56
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
Link-local IPv6 Address . . . . . : fe80::3464:a8c8:13fa:7116%15(Preferred)
IPv4 Address. . . . . . . . . . . : 10.1.1.27(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.1.1.1
DHCPv6 IAID . . . . . . . . . . . : 335564886
DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-22-4A-CD-9F-00-50-56-12-34-56
DNS Servers . . . . . . . . . . . : ::1
10.1.1.3
10.1.1.27
127.0.0.1
NetBIOS over Tcpip. . . . . . . . : Enabled
DC2008E-1
Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
Physical Address. . . . . . . . . : 00-0C-29-75-FF-46
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
Link-local IPv6 Address . . . . . : fe80::b81a:c109:24a0:9d3d%10(Preferred)
IPv4 Address. . . . . . . . . . . : 10.1.1.28(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.1.1.1
DHCPv6 IAID . . . . . . . . . . . : 251661353
DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-23-34-D6-43-00-0C-29-75-FF-46
DNS Servers . . . . . . . . . . . : ::1
10.1.1.3
10.1.1.27
127.0.0.1
NetBIOS over Tcpip. . . . . . . . : Enabled
Best Answer
This was resolved by removing IPv6 on the two DCs that had it running, and also by re-arranging the DNS configuration on the Network Cards.
DC2008S-0
DC2008E-0
DC2008E-1