Speed up SFTP uploads on high latency network

latencymac-osxmac-osx-servernetworkingsftp

I'm trying to transfer a set of large files internationally using SFTP, but I am finding my international partner can't get upload speeds above ~50k despite very good connections on either side. We can get multiple connections uploading at this speed (so not bandwidth?), but no single upload improves in speed, which is a problem as many files are several gb in size.

The SFTP is being hosted using the standard Apple OSX "Remote Login" SFTP system.

Is there a way to improve upload speeds, or is there a different SFTP host that would help? It's not clear to me if this is a configuration problem or an inherent limitation of the protocol.

(For security reasons I need to be using an end-to-end encrypted peer-to-peer connection — no cloud services).

Best Answer

With OpenSSH sftp client (which you seem to use), you can use:

  • -R switch to increase request queue length (default is 64)
  • -B switch to increase read/write request size (default is 32 KB)

For a start, try to double both:

sftp -R 128 -B 65536 user@host

It probably does not matter much, which of them you increase.

Increasing either should help to saturate your high-latency connection. With the above settings, it will keep 8 MB worth of data flowing in the pipe at any time (128*64K=8M).

Note that this helps with big file transfers only. It won't have any effect, when transferring a lot of small files.


For some background and a discussion about other (GUI) SFTP clients, see the "Network delay/latency" section of my answer to Why is FileZilla SFTP file transfer max capped at 1.3MiB/sec instead of saturating available bandwidth? rsync and WinSCP are even slower.