An "open DNS resolver" is a DNS server that's willing to resolve recursive DNS lookups for anyone on the internet. It's much like an open SMTP relay, in that the simple lack of authentication allows malicious 3rd parties to propagate their payloads using your unsecured equipment. With open SMTP relays, the problem is that they forward spam. With open DNS resolvers, the problem is that they allow a denial of service attack known as a DNS Amplification Attack.

The way this attack works is pretty simple - because your server will resolve recursive DNS queries from anyone, an attacker can cause it to participate in a DDoS by sending your server a recursive DNS query that will return a large amount of data, much larger than the original DNS request packet. By spoofing (faking) their IP address, they'll direct this extra traffic to their victim's computers instead of their own, and of course, they'll make as many requests as fast as they can to your server, and any other open DNS resolvers they can find. In this manner, someone with a relatively small pipe can "amplify" a denial of service attack by using all the bandwidth on their pipe to direct a much larger volume of traffic at their victims.

ArsTechnica did a decent article on the recent DNS Amplification DDoS attack against Spamhaus, and is worth a quick read to get the basics (and a good visual of the amplification).

The simplest way to protect your system from being abused like this is to limit the addresses your server will perform recursive lookups for to your local subnets. (The specifics of which depend on which DNS server you're using, of course).

For example, if I were using BIND 9, and wanted to simply prevent DNS recursion from outside addresses, I would use the following code in my config:

options {
    directory "/var/named/master";
    allow-recursion { 127.0.0.1; 10.0.0.0/8; 192.168.0.0/16; 172.16.0.0/12; };

That line of code tells my BIND server to only process recursive DNS requests for the local loopback address (which I guess I could/should set to the local loopback block, the whole /8) and the 3 Private IPv4 address spaces.

For Windows Server 2012, which you say you're using, you have the options below.

1. Separate your DNS server from your IIS server.

• At least in a perfect world, there's no reason you need to be running DNS on the same box as IIS.
  • Put DNS on an internal box that isn't NATed, so the outside world can't get at it, and let IIS reside on the external-facing box that the rest of the world can get at. You can use dual-homing or firewall rules to selectively allow access to your DNS server from your IIS server.

2. Block external DNS requests with a firewall, such as the built in Windows firewall.

• To my surprise, Windows DNS does not allow you to restrict the addresses to which recursive DNS requests are honored, so this actually the recommended method by Microsoft.
• 
  • Select the DNS rules (TCP and UDP), go to the Remote IP address section and add the subnets in use on your LAN, as well as any public-facing IP addresses of servers that need access to Active Directory. As with the BIND example, IPv4 private address spaces are 127.0.0.0/8 10.0.0.0/8 192.168.0.0/16 and 172.16.0.0/12.

3. Disable recursion.

• I'm honestly not sure what effect this will have on your environment, since you haven't really stated how DNS and AD are configured in your environment, and accordingly, it's the last option.
• 
  1. Open DNS Manager.
  2. In the console tree, right-click the applicable DNS server, then click Properties.
  3. Where?
  4. DNS/applicable DNS server
  5. Click the Advanced tab.
  6. In Server options, select the Disable recursion check box, and then click OK.
     • Since we have a multi-forest environment, and use conditional forwarders for that to work, I'm not going to check that box. Might be something for you to consider as well.

If your DNS server is authoritative for a publicly accessible domain, which I'm guessing is the case for you, then there's not a lot more you can do. The warning is due to the fact that your DNS server will respond to any request, which a public facing authoritative server will need to do.

In a spoofed request amplification attack, the source address of the DNS request is spoofed such that replies from the DNS server will be sent to the target (victim) host instead of the source. In addition to disabling recursion, filters can be put in place to reject packets with spoofed source addresses in order to mitigate these attacks. This would need to be performed by your routers or firewalls, though. Doing so will not prevent this warning from coming up in the security scan but you would then have evidence that you have taken action to mitigate the threat.