How to troubleshoot excessive latency between client and server

latency

On a large private network there seems to be excessive latency between two boxes which I control. This is stopping the server providing a web service to the client as its causing timeouts.

Here's a slow running request logged in Charles

Timing  
Request Start Time  03/11/10 23:21:33
Request End Time    03/11/10 23:21:33
Response Start Time 03/11/10 23:21:42
Response End Time   03/11/10 23:21:42
Duration    8.99 sec
Request Duration    16 ms
Response Duration   0 ms
Latency 8.97 sec
Speed   1.30 KB/s
Response Speed  ∞ KB/s
Size    
Request Header Size 412 bytes
Response Header Size    151 bytes
Request Size    -
Response Size   11.17 KB (11436 bytes)
Total Size  11.72 KB (11999 bytes)
Request Compression -
Response Compression    -

As a test I just tried getting an 423k image file via a browser, but this downloads so slow it stalls.

How can I troubleshoot where the problem lies?

I tried using pingplotter
alt text

Looks like hops 3 and 4 are the culprits? Where do I go from here?

ANSWER

This turned out to be a problem with the client box running Windows Web Server 2008 R2 and a known network speeds problem if the router is not able to handle network TCP window scaling. Updated router software and problem was resolved.

Best Answer

Attempt to replicate the issue from as many different points as possible. You'll probably find something in common between them (be it the server they are using, a switch, etc). Collect traceroute data from each endpoint.

It's always better to use the onion approach: add/remove layers. You can start form the client and come closer and close to the server. Or the inverse.

Make sure to check the interface statistics for error counts. I find it very useful to fire up tcpdump/Wireshark on both ends and capture the packets from the TCP session, then compare both. Wireshark does a good job of pointing out the most obvious problems (like checksum errors or retransmissions).

Related Solutions

How to control the network latency

If it has to be windows.. its not going to be easy and free i think. If you got some budget for this, go Shunra. No budget, go linux and use Qdisc.

Ive tried some free linux WAN simulators to emulate 4 hostingsites with delays and bandwidthlimits.. all just crashed with my config so i ended up setting up Qdisc manually.

If you decide to go with the linux option i could assisst in the Qdisc config.

For more opinions youve got a link from Zoredache in you questionpost.

Networking – Typical Network Latency for East to West Coast USA

Speed of Light:
You are not going beat the speed of light as an interesting academic point. This link works out Stanford to Boston at ~40ms best possible time. When this person did the calculation he decided the internet operates at about "within a factor of two of the speed of light", so there is about ~85ms transfer time.

TCP Window Size:
If you are having transfer speed issues you may need to increase the receiving window tcp size. You might also need to enable window scaling if this is a high bandwidth connection with high latency (Called a "Long Fat Pipe"). So if you are transferring a large file, you need to have a big enough receiving window to fill the pipe without having to wait for window updates. I went into some detail on how to calculate that in my answer Tuning an Elephant.

Geography and Latency:
A failing point of some CDNs (Content Distribtuion Networks) is that they equate latency and geography. Google did a lot of research with their network and found flaws in this, they published the results in the white paper Moving Beyond End-to-End Path Information to Optimize CDN Performance:

First, even though most clients are served by a geographically nearby CDN node, a sizeable fraction of clients experience latencies several tens of milliseconds higher than other clients in the same region. Second, we find that queueing delays often override the benefits of a client interacting with a nearby server.

BGP Peerings:
Also if you start to study BGP (core internet routing protocol) and how ISPs choose peerings, you will find it is often more about finances and politics, so you might not always get the 'best' route to certain geographic locations depending on your ISP. You can look at how your IP is connected to other ISPs (Autonomous Systems) using a looking glass router. You can also use a special whois service:

whois -h v4-peer.whois.cymru.com "69.59.196.212"
PEER_AS | IP               | AS Name
25899   | 69.59.196.212    | LSNET - LS Networks
32869   | 69.59.196.212    | SILVERSTAR-NET - Silver Star Telecom, LLC

It also fun to explore these as peerings with a gui tool like linkrank, it gives you a picture of the internet around you.

Related Topic