Limit rsync bandwidth nfs copy

backupbandwidthnetworkingrsync

We're currently forced to limit the backup bandwidth to a nfs disk outside our network (thru VPN) because it fills up the modem cache and we are forced to reboot it to regain connectivity.

0 22 * * *   flock rsync_wan_lock -c "rsync --rsync-path=\"nice -n5 ionice -c2 -n3 rsync\" --bwlimit 2000 -avrPq --delete-after /var/data/ /mnt/somedrive"

But that is not sufficient. Even thought 2000 KiB it ought be 50% of transfer speed (we have 30Mbps), it still fills the buffer.

So, I've read that rsync bursts and then goes silent to honour the bwlimit and that metadata does still not honour the bwlimit. So I'm trying trickle now.

The problem is that every doc I could find uses trickle on ssh connections thru the -e option. I don't think -e will work if I'm not copying through ssh, but they give the reason that putting the trickle in the --rsync-path won't work due to the forking rsync does.

0 22 * * *   flock rsync_wan_lock -c "rsync --rsync-path=\"nice -n5 ionice -c2 -n3 trickle -s -u 1000 -d 10000 rsync\" -avrPq --delete-after /var/data/ /mnt/somedrive"

Any ideas/comments? And what's going on with this modem, since when it's so easy to overflow a modem? The previous firewall was capped at 10000 and didn't run into cache problems.

Best Answer

As far as your user-level rsync is concerned, there is no network between the source directory /var/data and the destination /mnt/somedrive (the network transfer to the NFS server happens behind the scenes). Therefore trickle cannot work in this use case. On the other hand, the --bwlimit qualifier does work on local transfers.

The ionice option should help, but as the --rsync-path option is ignored on local transfers there's no point trying to apply it there.

See how this works for you

nice -n5 ionice -c2 -n3 rsync --bwlimit 2000K -avP --delete-after /var/data/ /mnt/somedrive

Note that your --bwlimit 2000K is actually 20Mb/s, which is considerably more than 50% of your maximum bandwidth.

Related Solutions

How to Use Rsync Over FTP

You don't. rsync can't do that for you, it is a protocol of its own and doesn't work over FTP.

You might, however, want to try csync. IIRC it provides rsync-like behaviour over HTTP. I can't comment on whether it works over FTP, you'll have to try it.

Linux – Copying a large directory tree locally? cp or rsync

I would use rsync as it means that if it is interrupted for any reason, then you can restart it easily with very little cost. And being rsync, it can even restart part way through a large file. As others mention, it can exclude files easily. The simplest way to preserve most things is to use the -a flag – ‘archive.’ So:

rsync -a source dest

Although UID/GID and symlinks are preserved by -a (see -lpgo), your question implies you might want a full copy of the filesystem information; and -a doesn't include hard-links, extended attributes, or ACLs (on Linux) or the above nor resource forks (on OS X.) Thus, for a robust copy of a filesystem, you'll need to include those flags:

rsync -aHAX source dest # Linux
rsync -aHE source dest  # OS X

The default cp will start again, though the -u flag will "copy only when the SOURCE file is newer than the destination file or when the destination file is missing". And the -a (archive) flag will be recursive, not recopy files if you have to restart and preserve permissions. So:

cp -au source dest

Best Answer

Related Solutions

How to Use Rsync Over FTP

Linux – Copying a large directory tree locally? cp or rsync

Related Topic