Tcp – Does Server Internet Bandwidth affect internet client download speeds

My colleague believed it's because of the physical distance while I don't think it matters. My understanding is once you have done the initial handshake and the data flow has started, it doesn't matter where the server is located and the result should be almost the same. Am I missing something here? How does it really work?

Both of you were right at some point in history, but your understanding is mostly correct... today :). There are a few factors that changed between the older answer your friend gave, and the capabilities we have today.

• TCP Window Scaling
• Host buffer tuning

The difference in the results you saw could have been affected by:

• Packet loss
• Parallel TCP transfers

TCP Window Scaling: The bandwidth-delay effect

As your friend mentioned, older implementations of TCP suffered from the limits imposed by the original 16-bit receive window size in the TCP header (ref RFC 793: Section 3.1); RWIN controls how much unacknowledged data can be waiting in a single TCP socket. 16-bit RWIN values limited internet paths with high bandwidth-delay products (and many of today's high bandwidth internet connections would be limited by a 16-bit value).

For high RTT values, it's helpful to have a very large RWIN. For example, if your path RTT from Malaysia to the US is about 200ms, the original TCP RWIN would limit you to 2.6Mbps.

Throughput_max = Rcv_Win / RTT
Throughput_max = 655358 / 0.200*
Throughput_max = 2.6Mbps

RFC 1323 defined several very useful TCP additions which helped overcome these limitations; one of those RFC 1323 TCP Options is "window scaling". It introduces a scale factor, which multiplies the original RWIN value, in order to get the full receive window value; using window scaling options allow a maximum RWIN of 1073725440 bytes. Applying the same calculations:

Throughput_max = Rcv_Win / RTT
Throughput_max = 10737254408 / 0.200*
Throughput_max = 42.96Gbps

Keep in mind that TCP increases RWIN gradually over the duration of a transfer, as long as packet loss isn't a problem. To see really large transfer rates over a high-delay connection, you have to transfer a large file (so TCP has time to increase the window) and packet loss can't be a problem for the connection.

Packet Loss

Internet circuits across the Pacific Ocean get pretty congested at times. Part of my family lives overseas, and we have to cross the Pacific Ocean to talk to them… when congestion and packet loss get heavy, sometimes we run into video throughput issues when we use Google Talk. Even modest packet loss (such as 0.5% loss) can slow a high-bandwidth connection down.

It's also worth saying that not all packet drops are bad. TCP was engineered to intentionally drop packets. I mentioned above that TCP gradually opens the RWIN over time (as long as packet loss is not a problem). When a congested circuit drops a packet, TCP responds to the drops by slowing the connection down (and spread across enough TCP sockets, TCP's response helps mitigate circuit congestion). However, this kind of TCP tail drop behavior has some drawbacks…We will talk about another congestion avoidance scheme when we talk about RED.

Also related to the subject of intentional packet drops is Random Early Detection (RED). This is quite honestly a great feature to enable in your TCP traffic queues when you enable QoS / HQoS. It works by looking at the interface queue and randomly dropping a packet. The packet drops (on TCP connections) will slow sockets down; and it's widely considered to be a much better alternative than tail-dropping TCP. It's also worth noting that this feature only works on TCP... as you may be aware, many (non-multimedia) UDP protocols do not slow down in the face of packet loss.

Parallel TCP streams

FYI, some speed test websites use parallel TCP streams to increase throughput; this may be affecting the results you see, because parallel TCP streams dramatically increase throughput in case you have some packet-loss in the path. I have seen four parallel TCP streams completely saturate a 5Mbps cable modem that suffered from 1% constant packet loss. Normally 1% loss would lower the throughput of a single TCP stream.

Bonus Material: Host Buffer tuning

Many older OS implementations had sockets with limited buffers; with older OS (like Windows 2000), it didn't matter whether TCP allowed large amounts of data to be in-flight... their socket buffers were not tuned to take advantage of the large RWIN. There was a lot of research done to enable high performance on TCP transfers. Modern operating systems (for this answer, we can call Windows Vista and later "modern") include better buffer allocation mechanisms in their socket buffer implementations.

A quick-and-dirty way to get the current utilization would be to poll these OIDs from OLD-CISCO-INTERFACES-MIB:

• locIfInBitsSec - 1.3.6.1.4.1.9.2.2.1.1.6.x
• locIfInPktsSec - 1.3.6.1.4.1.9.2.2.1.1.7.x
• locIfOutBitsSec - 1.3.6.1.4.1.9.2.2.1.1.8.x
• locIfOutPktsSec - 1.3.6.1.4.1.9.2.2.1.1.9.x

These give you the same numbers as the "input rate" and "output rate" lines from the output of show interface. Just replace the "x" with the interface index for Gig2/23 shown in show snmp mib ifmib ifindex. The MIB states that it's the "five minute exponentially-decayed moving average", but the actual rate matches whatever you've configured with load-interval on the interface (which defaults to 5 minutes).

These OIDs will give you a pretty good point-in-time measurement of the interface's utilization without requiring an external system to keep track of deltas between measurements. However, since it's an exponentially-decayed moving average, it may not be as accurate.