I already have DNS scavenging turned on, but it never seems to clean up any of the references to DC2, the domain controller and DNS server that failed some time ago.
You need to enable scavenging both at the server level and at the Domain level. Check properties of both to enable scavenging or delete the specific records yourself.
Can I just delete the entire Forward Lookup Zone? Does it recreate itself?
This is what you definitely do NOT want to do. Once you have your new DC up with DNS installed, make sure your Domain is AD Integrated and set it for replication to all domain or forest DNS servers. I prefer all forest but that's just me. When you decommission the old DC and remove everything make sure to re-point all of your clients and servers to the new DNS server IP address. I would suggest running them in parallel for a period of time to get all the clients and servers updated to using the new IP while the old one is still also available.
You are absolutely spot on in rebuilding the current DC as a second DC. You ALWAYS want to have 2 DCs and 2 DNS servers for an AD infrastructure. I personally insist on having at least one be a physical DC versus having both virtualized.
I am worried that in step 3 of my plan, I will end up replicating bad DNS records to my new domain controller. What is the best way to clean up my existing DNS before replicating it to my new server? It seems like it would be best just to have a clean Forward Lookup Zone, but I don't really understand how that zone works.
When in doubt, keep it. If things are working as expected and you don't want to break them, let sleeping dogs lie. If there are records you know are bad, get rid of them, but only records that you know what they do and only if you know for a fact that they are superfluous.
This could be due to a couple of different things:
1) Are your clients obtaining IPs via a Windows DHCP Server? If so, your DHCP server may not be configured to auto-register their IP with the DNS server.
To check, right-click your DHCP scope and go to properties. On the DNS tab enable DNS dynamic updates and set to "Always dynamically update..." Also enable Dynamic Update for clients that do not request updates.
1a) Even if these are enabled, you might need to make sure the DHCP server has permissions to update DNS records. If your DHCP server is also a domain controller, then you are probably fine, if not, then you may want to see if the DHCP server is a member of the "DnsUpdateProxy" group in AD. Then check the Security tab on the Reverse Zone and make sure that group is authorized to create all child objects (DNS records)
2) If your statically-configured hosts are not updating the reverse zone, make sure their NICs are configured to register their IP in DNS (Windows hosts are enabled for this by default). If they are in the forward zone but not the reverse, then something else is going on...
3) If the issue is that your reverse zones are mismatched between domain controllers (meaning a host was able to register with one of the DCs, but the registration did not get replicated to the others) it could mean the zones themselves aren't replicating between domain controllers. Make sure the reverse zone is AD-Integrated and also check the Zone Transfers tab and make sure they are allowed (generally Only to servers listed on the Name Servers tab). Also make sure your DNS server are listed on the Name Servers tab.
That's all I can think of for now...
EDIT
Ok so given that your DHCP server is not registering the records on behalf of the clients, and aside from your windows clients not being set to register with DNS on their NICs (which you should verify in TCP/IP properties on the client NIC), I would check the security settings on your reverse zone to make sure clients are allowed to register in the reverse zone. This article covers the default security settings for DNS zones: http://technet.microsoft.com/en-us/library/cc780538%28WS.10%29.aspx If the settings are jacked up (or even if they are fine), you might try just blowing away the reverse zone and recreating it since you basically have a non-functioning reverse zone as it is...
Try running an "ipconfig /registerdns" on one of the clients and see if it adds a record to the reverse zone. If it didn't, you might try looking in the DNS event log on the server to see if there are any issues there. I'm not sure if it would say whether or not a client failed to register or not, but may show you other issues with DNS if they exist.
Best Answer
New domains under an existing Forward Lookup Domain add domains to the left of the domain. So in your example, adding a domain to 'abc.com' would mean adding 'def.abc.com'. Since you want to add 'xyz.com', you need to add a new Forward Lookup domain in order to do that.