Ssh – How to fix corrupt packet error for with rsync for (relatively) large files

corruptionpacketlossrsyncssh

Trying to update files on a server, with the rsync command:

rsync -ravq -e "ssh -o ConnectTimeout=2 -o ServerAliveInterval=2 -ServerAliveCountMax=2" --delete ./local_dir user@$SERVER:/dest_dir

corrupt packet errors keep getting thrown, specifically:

rsync: writefd_unbuffered failed to write 4092 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (11337 bytes received so far) [sender]
rsync error: unexplained error (code 255) at /home/lapo/package/rsync-3.0.9-1/src/rsync-3.0.9/io.c(605) [sender=3.0.9]

This is probably related to a ssh timeout, as it seems to happen with large(r) files. Also, I keep getting timeouts using WinSCP. This is happening only to me; several of the people I work with that use this server do not have the same problem.

Using rsync from a Cygwin terminal in Windows 7, against a Centos 6.3 server.

I am not sure what other information might be useful or how to obtain it. I will update the question or add comments as per any suggestions.

How should I solve this?

Thanks very much!

Best Answer

I'm not sure what might cause the corrupt packet problem that drops your connection, but you might find rsync's --partial or --partial-dir option helpful when transferring large files so that when you restart the transfer it will continue where the transfer left off instead of having to start over transferring the whole file again:

--partial-dir=.rsync-partial

So you can modify your original command like this:

rsync -rav --progress --partial -e "ssh -o ConnectTimeout=2 -o ServerAliveInterval=2 -ServerAliveCountMax=2" --delete ./local_dir user@$SERVER:/dest_dir

or

rsync -rav --progress --partial-dir=.rsync-partial -e "ssh -o ConnectTimeout=2 -o ServerAliveInterval=2 -ServerAliveCountMax=2" --delete ./local_dir user@$SERVER:/dest_dir

Note that for this example I removed the -q (--quiet) option and added the --progress option in the first example and --partial-dir=.rsync-partial in the second example.

The difference between --partial and --partial-dir=.rsync-partial is that the later creates a directory that keeps the partial files separate from the fully-transferred files if that is important to you on the receiving (server) side.

The rsync manpage will explain this in further detail, though I'll also point out an important security note from the manpage:

IMPORTANT: the --partial-dir should not be writable by other users or it is a security risk. E.g. AVOID "/tmp".

Related Topic