Vps – rsync/scp extremely slow transfer rate (<30kB/s) over fast connection

file-transferrsyncvps

I am attempting to perform a backup using rsync of some files on a Digital Ocean VPS running Ubuntu 14.04.4 LTS.

rsync is extremely slow, ranging from 20-50 kB/s, only 1% of what it should be. It is equally slow when using scp or when using rsync in daemon mode, so this does not seem to be a problem particular to rsync or ssh.

I also have access to another VPS (Linode), I get the expected transfer speeds of 2-3MB/s when transferring files using rsync from that VPS.

Also, when I transfer files using rsync directly from the Digital Ocean VPS to the Linode VPS I get normal inter-server transfer rates of 15MB/s. These factors combined indicate that ssh and rsync are working fine on all the machines.

For now I've circumvented the problem by tunneling the ssh connection via the Linode VPS, but I'm curious about what might be causing the very low transfer rates.

Best Answer

When you say "mainly 15Mb tarballs", how many smaller files are there? Do you see the slow rate constantly or does it look faster on the larger files? rsync (and almost any other filesystem operation) is slower at processing many small files rather than fewer large ones due to the filesystem overhead of opening and closing each file (and any directory read/modify operations needed too). This can be exacerbated by VPS providers enforcing fair(er) IO use by throttling IO operations per second for each VM to a fixed maximum value.

To illustrate this:

  1. Create a large (say ~100Mb) file with something like:
    dd if=/dev/urandom of=/tmp/test.file bs=1024 count=102400
    and pushing just that file over rsync
  2. then repeat with many small files, for example 10240 10Kb files created with:
    for i inseq 1 10240; do dd if=/dev/urandom of=test.file.$i bs=1024 count=10; done
  3. repeat with even smaller files, 102400x 1Kb files created with:
    for i inseq 1 102400; do dd if=/dev/urandom of=test.file.$i bs=1024 count=1; done

You should see the process being far slower for the smaller files even though the cumulated contents are the same size. This will be the case even if you do everything locally rather then over the network at all (though obviously the network will magnify the effect somewhat). On a little old machine here time rsync * /location/on/another/local/drive/ gives real 0m0.683s, user 0m0.596s, sys 0m0.220s for the 1*100MB case and real 0m38.793s, user 0m4.664s, sys 0m15.657s for 102,400*1Kb - a difference of ~57x.

Related Topic