Linux – rsync : Read input from a file and sync accordingly

Unfortunately not directly. rsync requires a clean link with a shell that will allow it to start the remote copy of rsync, when run this way.

If you have some way of running long-lived listening processes on the host you could try starting rsync manually listening for connections on a non-privileged port, but most techniques for doing that would require proper shell access via SSH too, and it relies on the hosts firewall arrangements letting connections in on the port you chose (and the host having rsync installed in the first place). Running rsync as a publicly addressable service (rather than indirectly via SSH or similar) is not generally recommended for non-public data though.

If you host allows scripting in PHP or similar and does not have it locked down so extra processes can not be execed by user scripts, then you could try starting rsync in listening mode that way. If your end is connectible (you are running SSH accessible to the outside world) you could try this in reverse - have a script run rsync on the server but instead of listening for incoming connections have it contact your local service and sync that way. This still relies on rsync actually being installed on the host which is not a given, or that you can upload a working copy, but does not have the security implications of running an rsync daemon in a publicly addressable fashion and talking to it over an unencrypted channel.

Messing around as described above may be against the hosts policies though, even if it works at all, and could get you kicked off. You are better off asking if a full shell can be enabled for that account and either abandoning rsync for that host or abandoning that host and moving elsewhere if they will not do that.

It looks like you've only transferred 2GB of incremental changes. Remember that rsync still has to read in one whole file and checksum it so it has to read 80GB of data. Check your server stats during the rsync. Are you cpu or IO bound during the operation? How fast can you read the 80GB file off disk? That will be near your absolute minimum transfer time.

Also of note is that rsync makes a copy of the file while transferring then moves the final file into place in an atomic operation. You can see this by seeing a similiar filename with a random suffix during the transfer in the destination directory. This means that you have to read 160GB of data (80GB each for each source and destination) and write out 80GB on the destination side. Have you looked at the --inplace option? It may be beneficial here.

In short you may only have 2GB of changes but rsync is doing LOTS of work. You're probably IO bound as all that reading and writing on the same disk would could cause lots of contention and slowdown.