Force rsync timeout on stalled connection

rsync

I have a problem with a defective round robin data mover node I'm rsync'ing data from. One of the nodes keeps dropping the data connection to the file system my data is coming from and needs to reconnect, which can take up to several minutes. During this time, the rsync on that particular file does not abort, it just hangs, for up to several minutes, until the download continues. Because this is time critical data, that is not a viable option, but instead should drop the connection asap when it stalls, and reconnect (which would then be load balanced to the next node anyway).

Can anyone think of a method to tell rsync to abort a transfer that's been stalled for more than e.g. 20 seconds?

The operator of the broken data mover nodes are slow to react to this problem, they're aware of it, but not doing anything at the moment, so I need to find a solution at my end.

I've tried using the –timeout option, but that doesn't seem to work when a data connection is stalled.

Best Answer

You can run rsync with --timeout=SECONDS to exit after stalled file transfers. More recent versions of rsync have --contimeout=SECONDS for connection timeouts.

# timeout after 20 seconds
rsync --timeout=20 source destination

# timeout after 3 seconds of no connectivity
rsync --contimeout=3 source destination

More helpful example: https://explainshell.com/explain?cmd=rsync+--timeout%3D60+-aP+--append-verify+--stats+user%40host%3A%2Ffolder+%2Fcygdrive%2Ff%2Fbackup

Related Solutions

How to Use Rsync Over FTP

You don't. rsync can't do that for you, it is a protocol of its own and doesn't work over FTP.

You might, however, want to try csync. IIRC it provides rsync-like behaviour over HTTP. I can't comment on whether it works over FTP, you'll have to try it.

Linux – Copying a large directory tree locally? cp or rsync

I would use rsync as it means that if it is interrupted for any reason, then you can restart it easily with very little cost. And being rsync, it can even restart part way through a large file. As others mention, it can exclude files easily. The simplest way to preserve most things is to use the -a flag – ‘archive.’ So:

rsync -a source dest

Although UID/GID and symlinks are preserved by -a (see -lpgo), your question implies you might want a full copy of the filesystem information; and -a doesn't include hard-links, extended attributes, or ACLs (on Linux) or the above nor resource forks (on OS X.) Thus, for a robust copy of a filesystem, you'll need to include those flags:

rsync -aHAX source dest # Linux
rsync -aHE source dest  # OS X

The default cp will start again, though the -u flag will "copy only when the SOURCE file is newer than the destination file or when the destination file is missing". And the -a (archive) flag will be recursive, not recopy files if you have to restart and preserve permissions. So:

cp -au source dest

Best Answer

Related Solutions

How to Use Rsync Over FTP

Linux – Copying a large directory tree locally? cp or rsync

Related Topic