Measuring network latency

latencynetworkingperformancewindows-server-2008

I want to measure performance of the network, especially latency. We are performing stress tests of a system that sends live audio / video data over the network. When there are multiple simultaneous connections (e.g. 100 connections) we are observing that it takes too long for the data to be sent from one party to the second party. Currently, I don't know if it is because of server performance or it is a network issue.

Server load is really low (about 15% cpu load), however it could still be server issue because of threads context switching and it is possible that data waits for the server to begin processing. My setup is as follows:

Client -> Load balancer -> Proxy Server -> Application Server -> Agent

I want to measure how long it takes for the data to be sent between every two elements (omitting only Load balancer), e.g. between Client and Proxy Server.

Servers are Windows Server 2008R2, client and agent are Windows XP/7.

I thought that I could install wireshark on every machine, synchronize clocks and manually check packets captured on every machine. However, I hope there is a better solution.

Best Answer

Psping from Russinovich does ICMP ping, TCP ping, bandwidth testing, and more.

http://technet.microsoft.com/en-us/sysinternals/jj729731.aspx

It has a server component and a client component so you have to run it in both machines for the more advanced tests.

Related Solutions

FreeBSD performance tuning: Sysctl parameter, loader.conf, kernel

I'd recommend against options IPFIREWALL_DEFAULT_TO_ACCEPT. The default is to Default to Deny. The firewall comes up with just one rule deny ip from any to any and stays that way until a script configures exactly what traffic should get through.

Follow-Up Note: RSA (one of the world's leading security technology companies) was hacked recently when part of their firewall was disabled during a maintenance window. This really underscores how quickly a system can be compromised given the right conditions.

If you insist on disabling the firewall until you explicitly block unwanted traffic, please consider using the sysctl available by adding net.inet.ip.fw.default_to_accept=1 to loader.conf. This has the added benefit of being easily modified (no recompiling the kernel) if you change your mind at some point in the future.

Networking – Typical Network Latency for East to West Coast USA

Speed of Light:
You are not going beat the speed of light as an interesting academic point. This link works out Stanford to Boston at ~40ms best possible time. When this person did the calculation he decided the internet operates at about "within a factor of two of the speed of light", so there is about ~85ms transfer time.

TCP Window Size:
If you are having transfer speed issues you may need to increase the receiving window tcp size. You might also need to enable window scaling if this is a high bandwidth connection with high latency (Called a "Long Fat Pipe"). So if you are transferring a large file, you need to have a big enough receiving window to fill the pipe without having to wait for window updates. I went into some detail on how to calculate that in my answer Tuning an Elephant.

Geography and Latency:
A failing point of some CDNs (Content Distribtuion Networks) is that they equate latency and geography. Google did a lot of research with their network and found flaws in this, they published the results in the white paper Moving Beyond End-to-End Path Information to Optimize CDN Performance:

First, even though most clients are served by a geographically nearby CDN node, a sizeable fraction of clients experience latencies several tens of milliseconds higher than other clients in the same region. Second, we find that queueing delays often override the benefits of a client interacting with a nearby server.

BGP Peerings:
Also if you start to study BGP (core internet routing protocol) and how ISPs choose peerings, you will find it is often more about finances and politics, so you might not always get the 'best' route to certain geographic locations depending on your ISP. You can look at how your IP is connected to other ISPs (Autonomous Systems) using a looking glass router. You can also use a special whois service:

whois -h v4-peer.whois.cymru.com "69.59.196.212"
PEER_AS | IP               | AS Name
25899   | 69.59.196.212    | LSNET - LS Networks
32869   | 69.59.196.212    | SILVERSTAR-NET - Silver Star Telecom, LLC

It also fun to explore these as peerings with a gui tool like linkrank, it gives you a picture of the internet around you.

Related Topic