Active Directory – Resolving Domain Replication Summary Issues

active-directory

Looks like I've found myself in a bit of trouble here…

Background information

I am trying to clean up a domain that had 5 domain controllers, bringing it down to a two-DC environment. It is a very small environment with less than 20 machines. We ended up with 5 domain controllers simply as a result of unfinished work that we're finally getting around to in an effort to raise the domain functional level from 2012 to 2019.

The setup is as follows:

  • bmdc9 is Windows Server 2016 the PDC with all of the FSMO roles.
  • bmdc8 is Windows Server 2016.
  • bmdc10 is Windows Server 2016 Core.

The goal is to create a new domain controller bmdc13 and bmdc14 which will both be Windows Server 2019. Then, to raise the domain functional level to 2019.

Current issues:

#1: Whenever we run the DC demote wizard on bmdc8 we're getting an error saying:

No other domain controller could be contacted, but other domain controller objects are in the directory. If you are certain that this is the last domain controller for the domain and want to proceed, confirm that this is the last domain controller in the domain.

The problem is, this is NOT the last DC. VMs bmdc9 and bmdc10 are still up and active.

#2: Whenever we shut down bmdc10 we completely lose all DNS for external addresses. Meaning, if we try to ping www.google.com we get an error message saying: (Answered this myself, see Nov 5th comment below)

Ping request could not find host www.google.com. Please check the name and try again.

#3: When I run repadmin /replsum I am getting the following output:

enter image description here

What really strange here is that bmdc6 no longer exists. It was demoted earlier today (without errors) and removed from the domain gracefully.

#4: Whenever I go into Active Directory Sites and Services I am seeing more DCs than I should do:

enter image description here

For some reference:

  • bmdc4 was demoted earlier today.
  • bmdc6 was demoted earlier today.
  • bmdc8 up and running, but I would like to demote.
  • bmdc9 up and running. The current PDC.
  • bmdc10 up and running, but I would like to demote.

Other information:

In case it is helpful, below is some additional information:

enter image description here

Question(s)

Basically, I'm at a loss as to how unhealthy my environment is at the moment.

  • Why am I getting the replication errors? And, how do I clean that up?
  • Why are the old demoted DCs still showing up under Active Directory Sites and Domains? (i.e. bmdc4 and bmdc6). How do I remove them?
  • Why is DNS not working unless bmdc10 is up even though all NICs in our network have bmdc9 as the primary DNS server and bmdc10 as the secondary?
  • Is it safe to proceed with demoting bmdc8 even with the aforementioned 'last domain controller' warning?
  • What's the quickest non-intrusive way to fix all of this?

My ultimate goal is to migrate over to Server 2019 and raise the domain functional level to 2019. I don't care if that involves creating more DCs and retiring everything I have now, just as long as I don't lose the objects/users/computers/passwords in my current domain.

If anyone can help me out, that would be amazing. Thanks in advance.

Update(s)

2021-11-05 @ 14:06: This is what I get when I run repadmin /showreps

Default-First-Site-Name\BMDC9
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: **REMOVED**
DSA invocationID: **REMOVED**

==== INBOUND NEIGHBORS ======================================

DC=xxxx,DC=local
    Default-First-Site-Name\BMDC8 via RPC
        DSA object GUID: **REMOVED**
        Last attempt @ 2021-11-05 13:49:37 failed, result 8456 (0x2108):
            The source server is currently rejecting replication requests.
        5700 consecutive failure(s).
        Last success @ 2021-06-15 11:19:34.
    Default-First-Site-Name\BMDC10 via RPC
        DSA object GUID: **REMOVED**
        Last attempt @ 2021-11-05 13:56:14 was successful.

CN=Configuration,DC=xxxx,DC=local
    Default-First-Site-Name\BMDC8 via RPC
        DSA object GUID: **REMOVED**
        Last attempt @ 2021-11-05 13:49:37 failed, result 8456 (0x2108):
            The source server is currently rejecting replication requests.
        3444 consecutive failure(s).
        Last success @ 2021-06-15 10:51:35.
    Default-First-Site-Name\BMDC10 via RPC
        DSA object GUID: **REMOVED**
        Last attempt @ 2021-11-05 13:49:37 was successful.

CN=Schema,CN=Configuration,DC=xxxx,DC=local
    Default-First-Site-Name\BMDC8 via RPC
        DSA object GUID: **REMOVED**
        Last attempt @ 2021-11-05 13:49:38 failed, result 8456 (0x2108):
            The source server is currently rejecting replication requests.
        3427 consecutive failure(s).
        Last success @ 2021-06-15 10:51:35.
    Default-First-Site-Name\BMDC10 via RPC
        DSA object GUID: **REMOVED**
        Last attempt @ 2021-11-05 13:49:38 was successful.

DC=DomainDnsZones,DC=xxxx,DC=local
    Default-First-Site-Name\BMDC8 via RPC
        DSA object GUID: **REMOVED**
        Last attempt @ 2021-11-05 13:49:38 failed, result 8456 (0x2108):
            The source server is currently rejecting replication requests.
        3465 consecutive failure(s).
        Last success @ 2021-06-16 17:20:40.
    Default-First-Site-Name\BMDC10 via RPC
        DSA object GUID: **REMOVED**
        Last attempt @ 2021-11-05 13:49:38 was successful.

DC=ForestDnsZones,DC=xxxx,DC=local
    Default-First-Site-Name\BMDC8 via RPC
        DSA object GUID: **REMOVED**
        Last attempt @ 2021-11-05 13:49:38 failed, result 8456 (0x2108):
            The source server is currently rejecting replication requests.
        3431 consecutive failure(s).
        Last success @ 2021-06-15 10:51:35.
    Default-First-Site-Name\BMDC10 via RPC
        DSA object GUID: **REMOVED**
        Last attempt @ 2021-11-05 13:49:38 was successful.

Source: Default-First-Site-Name\BMDC8
******* 5700 CONSECUTIVE FAILURES since 2021-06-16 17:20:40
Last error: 8456 (0x2108):
            The source server is currently rejecting replication requests.

Best Answer

OK, firstly I want to attribute some credit to @GregAskew as some of his responses helped me troubleshoot the issue.

Should anyone else run into this problem, below is how I resolve it:

  1. Per Greg's suggestion, running repadmin /showreps highlighted that the problem was bmdc8. I was able to fix this by simply removing that machine from the network. Unfortunately, demoting the domain controller did not work . (i.e. the wizard errored out). As such, I had to follow the Microsoft documentation to manually remove a domain controller from the domain. Once that was done, everything worked just fine from a replication standpoint.

  2. Manually deleted the old decommissioned DCs from Active Directory Sites and Services.

  3. DNS was a mess. Each DC had another DC as Forwarders. What I did was cleared out all Forwarders, then added a public DNS provider as our forwarder. Once I did that, the internet would work fine even when bmdc10 was down.

  4. Demoting the DC would not work. And so, I had to manually remove bmdc8 from the domain.

  5. None of this involved downtime. Nor did I lose any domain objects. Everything has been fine and could have been done during business hours.

Related Topic