TCP Upload Speeds Slow to the Internet over LFN

linux-networkingtcp

I've got two Linux VMs separated by a 50 mbps WAN connection. There is about 70ms of RTT on this WAN link, so it's got some latency. I've been testing my upload throughput to the Internet from each of these VMs. The basic setup is that Site A has an Internet connection and also a WAN connection to Site B. Site B uses the Internet connection at Site A.

I've noticed that Site B uploads a fair bit slower to the Internet than Site A. I was just using some of those Internet speed testing sites to test my upload speeds. I used the same Internet speed test site and server from each site to do the testing. I also ran the tests many, many times.

I ran some Wireshark caps to see what was going on. I assumed that the Internet server wasn't opening a wide enough TCP window to account for my 70ms of additional latency from Site B. However, the TCP window is wide open and it's actually my server at Site B that stops transmitting, waiting for ACKs to come in before sending more data.

I've looked at a bunch of stuff: TCP timestamps, SACKs, and Window Scaling are all enabled.
I've increased my buffers as follows:

net.core.rmem_max = 67108864 
net.core.wmem_max = 67108864 
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432
net.core.netdev_max_backlog = 30000
net.ipv4.tcp_congestion_control=cubic

I've also increased by transmit queue length as follows: 

ifconfig eth0 txqueuelen 10000

Lastly, I've disabled the software TCP segment offloading on the VM (there is no hardware TOE on my server).

Still though, Wireshark shows me that I don't get more than about 11,000 bytes in flight. I do have some dropped packets near the beginning of the transfer, but by the time things really get going, I would have expected more data to be flowing in flight.

Can anyone shed some light on why the sender is holding back data when so little data is actually in flight?

Best Answer

What I see in your packet trace is congestion control reacting to packet loss.

The client starts out sending an initial 9 segments followed by slow start, where it send two more segments each time it receives an ACK packet.

The slow start algorithm continues until the first duplicate ACK from the server indicates that a packet has been lost. This happens at a point where there are 20820 bytes in flight. After this the client will grow the congestion window slower.

The second case of congestion happens only half a second into the transmission. After this the number of bytes in flight grows from around 15K and reaches 68012 bytes in transit by the time the third case of congestion happens which is 6 seconds into the transmission.

There is about 54KB in flight after the third case of congestion. This grows until it reaches 94384 bytes in flight, and the fourth case of congestion happens, this is 10 seconds into the transmission.

There are several more cases of congestion throughout the rest of the trace. The transmission might have been able to pick up speed, if it did not run into packet loss as often as it did. Having experienced the first packet loss as early as it did, it would take a long time reach full speed.

So what you need to figure out is why packets are lost that early during the TCP connection. It turns out that the packet lost at this point is one of the 9 packets send back-to-back at the very start of the connection. This indicates that the client is configured with a too high initcwnd setting. Judging from the one packet capture you provided, a reasonable setting would be 4.

The initcwnd setting is specified separately for each routing table entry. They can be viewed and changed using the ip route command.

Related Topic