Windows – Tuning Windows for High Latency and High Bandwidth TCP Transfers

bandwidthnetworkingwindows

How can I speed up my Windows 7 tcp file transfers over a 100 mbps link between Chicago and London?

Right now, I am transferring two 4GB files in two sftp sessions. Each file transfer is running at a rate of 500 KB/s (4 Mbits/sec). (B == bytes, b == bits, naturally). This is quite slow, as iperf tests show a rate of 16.8 Mbits/sec for a single TCP stream. That's 2 MB/sec.

Starting the second sftp session does not affect the speed of the first one iota. This makes sense to me as I noticed that my iperf tests were limited under TCP- each connection was 16 Mbps which leaves room on the wire for more connections. I was able to make iperf saturate my line by running many parallel connections, and it was a linear increase in bandwidth. That is, with 5 parallel connections I achieved bandwidth of about 5×16 == 80 Mbps.

I note that local file transfers using sftp are around 30 MB/s… quite good. So I don't think SSH channel encryption is the bottleneck.

I have reviewed What is the best way to transfer a single large file over a high-speed, high-latency WAN link? as well as other serverfault questions and there are a lot of interesting tools but what I am looking for is a way for me to adjust Windows' TCP stack to make sure I get my full 2 MB at least… and maybe more, since the iPerf tests were across untuned machines.

I notice, too, that when I drag-and-drop a file on a London machine from my Chicago fileserver, it seems to transfer at quite a nice clip (as shown by looking at the details during the transfer) but after a few minutes it drops off the cliff into the 10's of Kbps. It was very odd.

My latency is steady at around 88 ms, so I think the network infrastructure is good. We have a dedicated line; this is not a VPN over the Internet or anything like that.

========

I have a Linux VM that is "close" to my Windows desktop, network-wise. That is, they are on different subnets but both machines go into switches that both go into another switch that goes to London. Right now I am running an sftp from Linux to London and it is running at 8 MB/s while my desktop continues at 500 KB/s.

========
Ok, I put my Linux laptop on exactly the same switch as my Desktop. Where the desktop is performing an sftp at 500 KB/s at this moment, the Linux laptop is sending a 4Gig file at 7.6 MB/s. To the same server in London, of course.

========
Another test: I was using SecureFX by VanDyke Corp. I also have Cygwin on my PC, and I just went and used the CLI sftp command there. I just increased my throughput fivefold… I'm now getting 2.5 MB/s instead of 500k! :-\ This is on par with my iPerf results of 16 Mbps. Which I would have been content with… although now I notice that my laptop is three times as fast as that.

========
Another test: On my London machine, I started a Windows Explorer file copy from my fileserver in Chicago to the locally-mounted Downloads folder (that is, on an internal hard drive). The throughput was a rock-steady 5.49 MB/s for 12 minutes or so. That's close enough to fine for me! And again, copying via SecureCRT was miserably slow- a steady 500 KB/s.

The only thing I can conclude is that although there may be TCP tweaks that will help, the app you use can make a big difference! Furthermore, I don't know why my filecopy acted so strangely the other day. If it happens again I will attempt to debug.

Thanks for the suggestions and replies.

======================

Ok, I have followed Todd Wilcox' links and figured a few things out:

First, according to Microsoft, from https://technet.microsoft.com/en-us/library/cc957546.aspx?f=255&MSPPError=-2147217396, "TCP uses a receive window… For Ethernet networks, the default value of this entry is 0x4470 (17,520, or 12 segments of 1,460 bytes each)" Well I have a WAN with 80ms latency so this is much too small.
I have set both GlobalMaxTcpWindowSize and TcpWindowSize in the registry (the latter on my adapter) to 1 MiB, then went to the command line and used netsh to change the Receive Window Auto-Tuning Level from normal to disabled. My network transfer speeds did not change, so I rebooted my machine and again performed an sftp from my desktop to the London server.
Throughput plummeted to 700 KB/s. Needless to say, I set everything back and rebooted again. Now my throughput is back to 2.1 MB/s.
During the sftp, Wireshark shows a strange sawtooth pattern to the TCP window scaling, going from about 64000 to 62000 bytes then back up again. The cycle takes about 35 seconds.
Let's say that my average is 63000 bytes window size. Therefore, my Bps throughput should be (from http://bradhedlund.com/2008/12/19/how-to-calculate-tcp-throughput-for-long-distance-links/), Window-Size (bytes) / Latency (sec) == 700 KB/s. Strange that I'm getting 2.1 MB/s… that's higher than the theoretical! I'm sure there are no Wan accelerators in between or anything more complicated than a Cisco 4948. Maybe Wireshark's window scaling graph is not the same as the window size, I don't know, but Googling told me that that was where I'd find the TCP window size.
Things aren't adding up. Although I did find that my twiddling did screw things up, I don't know how to definitively tell what my TCP window sizes are. Furthermore, the oft-sited Brad Hedlund page gives some calculations that are not reflected in my experience.
Finally, and this is the greatest lesson I have learned: I think I'll leave well enough alone. If we find we need better throughput, maybe I'll recommend a WAN accelerator. But since we're going halfway across the world we oughtn't expect phenomenal bandwidth from a single TCP connection.

Best Answer

Formula to calculate optimal TCP window size (source):

Bandwidth-in-bits-per-second * Round-trip-latency-in-seconds = TCP window size in bits / 8 = TCP window size in bytes

In your case: 100 000 000 * .088 = 8 800 000 bits or 1 100 000 bytes

This is configurable in the Windows registry in the TcpWindowSize key in a valid range of 0–0x3FFFFFFF (1 073 741 823 decimal), so that figure is in the valid range.

The default value is the smallest of the following (Note: "Values greater than 64 KB can be achieved only when connecting to other systems that support RFC 1323 Window Scaling"):

0xFFFF

GlobalMaxTcpWindowSize (another registry parameter)

The larger of four times the MSS (Maximum Segment Size)

16384 rounded up to an even multiple of the MSS

The stack also tunes itself based on the media speed:

Below 1 Mbps: 8 KB

1 Mbps – 100 Mbps: 17 KB

Greater than 100 Mbps: 64 KB

Source (This link is now mostly dead - slightly alive, but only a miracle could bring it back)

Also see: http://bradhedlund.com/2008/12/19/how-to-calculate-tcp-throughput-for-long-distance-links/

And: https://technet.microsoft.com/en-us/library/cc938219.aspx

Related Solutions

How to find the bottleneck while transferring huge files between 2 hosts

your problem is that you are testing too many things at once:

disk read speed
SSH encryption
wireless
SSH decryption
disk write speed

Since you mentioned SSH I am going to assume this is a unix system...

You can rule out any problems with disk read speed with a simple

dd if=yourfile of=/dev/null #or
pv yourfile > /dev/null

on the receiving end you can do a simple disk write test

dd if=/dev/zero of=testfile bs=1M count=2000 # or
dd if=/dev/zero bs=1M count=2000 | pv > testfile

dd is not really a "benchmark" but since scp uses sequential IO, it is close enough

you can also test SSH by doing something like

dd if=/dev/zero bs=1M count=100 | ssh server dd of=/dev/null # or
dd if=/dev/zero bs=1M count=100 | pv | ssh server dd of=/dev/null

finally, to rule out SSH being the bottleneck, you can use nc to test the network performance

server$ nc -l 1234 > /dev/null
client$ dd if=/dev/zero bs=1M count=100 | pv | nc server 1234 # or
client$ dd if=/dev/zero bs=1M count=100 | nc server 1234

if you really want to properly test the network, install and use something like iperf, but nc is a good start.

I'd start with the nc test as that will rule out the most things. You should also definitely run the tests while not using wireless. 802.11n can easily max out a 100mbit port, but only if you have it properly setup.

(Ubuntu >= 12.04 defaults to netcat-openbsd. nc -l -p 1234 > /dev/null may be what you want if you're using netcat-traditional).

The best way to transfer a single large file over a high-speed, high-latency WAN link

Searching for "high latency file transfer" brings up a lot of interesting hits. Clearly, this is a problem that both the CompSci community and the commercial community has put thougth into.

A few commercial offerings that appear to fit the bill:

FileCatalyst has products that can stream data over high-latency networks either using UDP or multiple TCP streams. They've got a lot of other features, too (on-the-fly compression, delta transfers, etc).
The fasp file transfer "technology" from Aspera appears to fit the bill for what you're looking for, as well.

In the open-source world, the uftp project looks promising. You don't particularly need its multicast capabilities, but the basic idea of blasting out a file to receivers, receiving NAKs for missed blocks at the end of the transfer, and then blasting out the NAK'd blocks (lather, rinse, repeat) sounds like it would do what you need, since there's no ACK'ing (or NAK'ing) from the receiver until after the file transfer has completed once. Assuming the network is just latent, and not lossy, this might do what you need, too.

Best Answer

Formula to calculate optimal TCP window size (source):

Related Solutions

How to find the bottleneck while transferring huge files between 2 hosts

The best way to transfer a single large file over a high-speed, high-latency WAN link

Related Topic