Linux – rsync server side limit bandwidth/connection

linuxrsyncscriptingsynchronization

In a VOIP application, I have upto 3000 clients rsync audio files from there linux server in a daily, server is placed at a data center (10Mbps in/out bound), the server works as a VOIP sip server running FreeSWITCH (low ping latency should be ensured.)

Therefore I would like to have server side control of rsync which controls:

Limit total outbound bandwidth.
Limit total number of connections. (Reject clients while at max number of connection and let it retry after a specific time frame.)
OPTIONAL: list/kill individual connections.

Normally I would use ssh + rsync + pem_keys with some extra options, but above requirements are not feasible by simple command lines. Can anyone point me some direction. or show some scripts/tools? I would also probably integrate them and release on github. Thanks!

Best Answer

If I were in your situation, I would continue using stock tools purely because you can always keep them up-to-date and get all the newest bugfixes/features without having to patch them with your own code.

As a result, I would achieve what you wanted with this:

Use tc for traffic shaping - OpenVZ has quite a nice article on it.
Use iptables - Cyberciti to the rescue for this one. You would reject new connections over the threshold, so you would need to wrap rsync client-side to ensure it retries if it fails fast.
Probably the easiest way to achieve this is to have all your files under a single group, and a bunch of users part of that group. Then give every client their own user to connect as. From here, it should be relatively straightforward to kill the PIDs of the clients connected (I'm sure there are other ways you could identify the users uniquely though, this is just a suggestion.)

Related Solutions

How to Use Rsync Over FTP

You don't. rsync can't do that for you, it is a protocol of its own and doesn't work over FTP.

You might, however, want to try csync. IIRC it provides rsync-like behaviour over HTTP. I can't comment on whether it works over FTP, you'll have to try it.

Linux – Copying a large directory tree locally? cp or rsync

I would use rsync as it means that if it is interrupted for any reason, then you can restart it easily with very little cost. And being rsync, it can even restart part way through a large file. As others mention, it can exclude files easily. The simplest way to preserve most things is to use the -a flag – ‘archive.’ So:

rsync -a source dest

Although UID/GID and symlinks are preserved by -a (see -lpgo), your question implies you might want a full copy of the filesystem information; and -a doesn't include hard-links, extended attributes, or ACLs (on Linux) or the above nor resource forks (on OS X.) Thus, for a robust copy of a filesystem, you'll need to include those flags:

rsync -aHAX source dest # Linux
rsync -aHE source dest  # OS X

The default cp will start again, though the -u flag will "copy only when the SOURCE file is newer than the destination file or when the destination file is missing". And the -a (archive) flag will be recursive, not recopy files if you have to restart and preserve permissions. So:

cp -au source dest

Best Answer

Related Solutions

How to Use Rsync Over FTP

Linux – Copying a large directory tree locally? cp or rsync

Related Topic