I just had an argument with a colleague of mine and thought I'd just reach out to the experts on this. Here's the scenario. We were using a website that measures your connection's speed. We tested using a server that is far from us (We are in Malaysia and the server was in US). It was around 2 Mbps. Then we tried with a server in Singapore and it was much faster (around 15 Mbps). My colleague believed it's because of the physical distance while I don't think it matters. My understanding is once you have done the initial handshake and the data flow has started, it doesn't matter where the server is located and the result should be almost the same. Am I missing something here? How does it really work?
Tcp – Does physical distance affect download speed
bandwidthlatencytcp
Best Answer
Both of you were right at some point in history, but your understanding is mostly correct... today :). There are a few factors that changed between the older answer your friend gave, and the capabilities we have today.
The difference in the results you saw could have been affected by:
TCP Window Scaling: The bandwidth-delay effect
As your friend mentioned, older implementations of TCP suffered from the limits imposed by the original 16-bit receive window size in the TCP header (ref RFC 793: Section 3.1); RWIN controls how much unacknowledged data can be waiting in a single TCP socket. 16-bit RWIN values limited internet paths with high bandwidth-delay products (and many of today's high bandwidth internet connections would be limited by a 16-bit value).
For high RTT values, it's helpful to have a very large RWIN. For example, if your path RTT from Malaysia to the US is about 200ms, the original TCP RWIN would limit you to 2.6Mbps.
RFC 1323 defined several very useful TCP additions which helped overcome these limitations; one of those RFC 1323 TCP Options is "window scaling". It introduces a scale factor, which multiplies the original RWIN value, in order to get the full receive window value; using window scaling options allow a maximum RWIN of 1073725440 bytes. Applying the same calculations:
Keep in mind that TCP increases RWIN gradually over the duration of a transfer, as long as packet loss isn't a problem. To see really large transfer rates over a high-delay connection, you have to transfer a large file (so TCP has time to increase the window) and packet loss can't be a problem for the connection.
Packet Loss
Internet circuits across the Pacific Ocean get pretty congested at times. Part of my family lives overseas, and we have to cross the Pacific Ocean to talk to them… when congestion and packet loss get heavy, sometimes we run into video throughput issues when we use Google Talk. Even modest packet loss (such as 0.5% loss) can slow a high-bandwidth connection down.
It's also worth saying that not all packet drops are bad. TCP was engineered to intentionally drop packets. I mentioned above that TCP gradually opens the RWIN over time (as long as packet loss is not a problem). When a congested circuit drops a packet, TCP responds to the drops by slowing the connection down (and spread across enough TCP sockets, TCP's response helps mitigate circuit congestion). However, this kind of TCP tail drop behavior has some drawbacks…We will talk about another congestion avoidance scheme when we talk about RED.
Also related to the subject of intentional packet drops is Random Early Detection (RED). This is quite honestly a great feature to enable in your TCP traffic queues when you enable QoS / HQoS. It works by looking at the interface queue and randomly dropping a packet. The packet drops (on TCP connections) will slow sockets down; and it's widely considered to be a much better alternative than tail-dropping TCP. It's also worth noting that this feature only works on TCP... as you may be aware, many (non-multimedia) UDP protocols do not slow down in the face of packet loss.
Parallel TCP streams
FYI, some speed test websites use parallel TCP streams to increase throughput; this may be affecting the results you see, because parallel TCP streams dramatically increase throughput in case you have some packet-loss in the path. I have seen four parallel TCP streams completely saturate a 5Mbps cable modem that suffered from 1% constant packet loss. Normally 1% loss would lower the throughput of a single TCP stream.
Bonus Material: Host Buffer tuning
Many older OS implementations had sockets with limited buffers; with older OS (like Windows 2000), it didn't matter whether TCP allowed large amounts of data to be in-flight... their socket buffers were not tuned to take advantage of the large RWIN. There was a lot of research done to enable high performance on TCP transfers. Modern operating systems (for this answer, we can call Windows Vista and later "modern") include better buffer allocation mechanisms in their socket buffer implementations.