IIS 7 – Resolving 401.2 Error by Hostname but Not by IP Address

I am experiencing an issue with an IIS 7 application on our network. The problem is as follows:

If a user browses to the site by IP address (http://172.16.10.32/site) the application loads fine, with no errors, and worth perfectly.

If the same user browses to the site by hostname (http://dms-live/site) an "Internet Explorer cannot display the webpage" message is displayed. Digging deeper results in the actual error code 401.2 – Unauthorized.

Some further information:

• This is an internal site with no external access whatsoever

• Internet Explorer is set by group policy to place the site in the
  Intranet zone

• The application allows "Anonymous Authentication" (using IUSR),
  "ASP.NET Impersonation" (set to Authenticated User), "Basic Authentication" (no default domain or realm specified) and
  "Windows Authentication" (with Kernel-mode authentication enabled)

• The user can ping dms-live, which returns the correct IP address 172.16.10.32

• The user is able to browse to both http://172.16.10.32 and
  http://dms-live without issues and sees the default IIS 7 splash
  screen

• The hostname "dms-live" is an additional A record specified in DNS
  pointing to 172.16.10.32

• The actual hostname of the server is "accserver16". Using this in
  place of "dms-live" gives exactly the same result (401.2 error)

• Using FQDNs instead of the short hostnames gives the same result as
  well (401.2 error)

• The website has bindings to dms-live, accserver16 and 172.16.10.32,
  plus the associated FQDNs

• Our network has two domains in two forests, domain A in forest A and domain B in forest B. All users and workstations are in
  domain A, the IIS server is in domain B. There is two-way
  trust configured between domain A and domain B.

• Both domains have their own DNS servers

This problem only arose recently, after a single very specific DNS change was made. Originally the DNS zone for domain B was hosted by the domain B DNS server, and a Conditional Forwarders entry was set up to forward requests from the domain A DNS servers. In the interests of resiliency this was changed so that the domain A DNS servers had secondary Forward and Reverse zones set up for the domain B zones.

The secondary zone is loading fine, replicates changes properly and returns the correct DNS records to ping and nslookup requests.

One other odd bit of information – some PCs on the network are able to load http://dms-live/site without any problems. There doesn't seem to be much of a pattern for which PCs work and which don't.

It's not down to the user (the same user on two different PCs won't necessarily get the same result), browser (this has been tested with IE8 and IE9 and results are not consistent) or Operating System (problem seems independent of whether Windows XP or Windows 7 is being used).

Troubleshooting tried so far:

• Enabling a single form of authentication at a time
• Disabling Kernel-mode authentication
• Setting up SPN records for DMS-LIVE and the associated FQDN
• Clearing the IE cache between attempts
• Rebooting the PCs
• Flushing the DNS cache with ipconfig /flushdns
• Setting the "DisableStrictNameChecking" registry key to 1 and
  rebooting the server
• Adding ACCSERVER16$ and IIS_IUSRS to the NTFS permissions of the
  application directory

On a PC that fails to load the page Fiddler shows that a single request is made which returns a 401 error. On a PC that loads the page successfully Fiddler shows the same 401 error followed by a 302, followed by a 200.

If the Secondary zones are removed and the Conditional Forwarders entry put back in, the problem disappears.

It seems as though for some reason, when using a Secondary Zone instead of a Conditional Forwarder to resolve a hostname in a different forest, authentication fails for some PCs but not all – but why this would be the case I have no idea. Can anyone help?