Well, we finally appear to have resolved this issue in our environment. For the benefits of others, here's what we discovered and how we fixed the problem:
To try and gain further insight into what was occurring before/during/after the delays we used Wireshark on a client machine to capture/analyse network traffic whilst that client attempted to access a DFS share.
These captures showed something strange: whenever the delay occurred, in between the DFS request being sent from the client to a DC, and the referral to a DFS root server coming back from the DC to the client, the DC was sending out several broadcast name lookups to the network.
Firstly, the DC would broadcast a NetBIOS lookup for DOMAIN (where DOMAIN is our pre-Windows 2000 Active Directory domain name). A few seconds later, it would broadcast a LLMNR lookup for DOMAIN. This would be followed by yet another broadcast NetBios lookup for DOMAIN. After these three lookups had been broadcast (and I assume timed out) the DC would finally respond to the client with a (correct) referral to a DFS root server.
These broadcast name lookups for DOMAIN were only being sent when the long delay opening a DFS share occurred, and we could clearly see from the Wireshark capture that the DC wasn't returning a referral to a DFS root server until all three lookups been sent (and ~7 seconds passed). So, these broadcast name lookups were pretty obviously the cause of our delays.
Now that we knew what the problem was, we started trying to figure out why these broadcast name lookups were occurring. After a bit more Googling and some trial-and-error, we found our answer: we hadn't set the DfsDnsConfig registry key on our domain controllers to 1, as is required when using DFS in a DNS-only environment.
When we originally setup DFS in our enviroment we did read the various articles about how to configure DFS for a DNS-only environment (e.g. Microsoft KB244380 and others) and were aware of this registry key, but had misintepreted the instructions on when/how to use it.
KB244380 says:
The DFSDnsConfig registry key must be
added to each server that will
participate in the DFS namespace for
all computers to understand fully
qualified names.
We thought this meant that the registry key has to be set on the DFS namespace servers only, not realising that it was also required on the domain controllers. After we set DfsDnsConfig to 1 on our domain controllers (and restarted the "DFS Namespace" service), the problem vanished.
Obviously we're happy with this outcome, but I would add that I'm still not 100% convinced that this is our only problem - I wonder if adding DfsDnsConfig=1 to our DCs has only worked around the problem, rather than solving it. I can't figure out why the DCs would be trying to lookup DOMAIN (the domain name itself, rather than a server in the domain) during the DFS referral process, even in a non-DNS-only environment, and I also know I haven't set DfsDnsConfig=1 on domain controllers in other (admittedly much smaller / simpler) DNS-only environments and haven't had the same issue. Still, we've solved our problem so we are happy.
I hope this is helpful to the others who are experiencing a similar issue - and thanks again to those that offered suggestions along the way.
By default, targets in a client's site are preferred to any other referrals. Therefore, you can chose an ordering method for targets ousite of the client's site.
From the Microsoft website: http://technet.microsoft.com/en-us/library/cc782417%28v=ws.10%29.aspx
The domain controller determines the clients site and returns a list
of root targets. By default, the root targets in the clients site are
at the top of the list, followed by the remaining root targets in
random order. If least-expensive target selection is enabled, the
remaining root targets are ordered by lowest cost. If same-site target
selection is enabled, only root servers in the clients site are listed
in the referral.
Also, from the DFS help you have the procedure to change the priority of referrals:
A referral is an ordered list of targets that a client computer
receives from a domain controller or namespace server when the user
accesses a namespace root or folder with targets in the namespace.
Each target in a referral is ordered according to the ordering method
for the namespace root or folder. To refine how targets are ordered,
you can set priority on individual targets. For example, you can
specify that the target is first among all targets, last among all
targets, or first (or last) among all targets of equal cost.
To set target priority on a root target for a domain-based namespace
Click Start, point to Administrative Tools, and then click DFS Management.
In the console tree, under the Namespaces node, click the domain-based namespace for the root targets that you want to set
priority on.
In the details pane, on the Namespace Servers tab, right-click the root target with the priority that you want to change, and then click
Properties.
On the Advanced tab, click Override referral ordering, and then click the priority that you want.
- First among all targets Specifies that users should always be referred to this target if the target is available.
- Last among all targets Specifies that users should never be referred to this target unless all other targets are unavailable.
- First among targets of equal cost Specifies that users should be referred to this target before other targets of equal cost (which
usually means other targets in the same site).
- Last among targets of equal cost Specifies that users should never be referred to this target if there are other targets of equal cost
available (which usually means other targets in the same site).
To set target priority on a folder target
Click Start, point to Administrative Tools, and then click DFS Management.
In the console tree, under the Namespaces node, click the folder of the targets that you want to set priority on.
In the details pane, on the Folder Targets tab, right-click the folder target with the priority that you want to change, and then
click Properties.
On the Advanced tab, click Override referral ordering, and then click the priority that you want.
Best Answer
Have you seen this DFS Server Prioritization document? It sounds like even if both of your servers are in the same "site", you can still set a different "target priority" (according to that document). It sounds like this should prioritize the servers as you would expect from a client point of view.