Linux – Rsync to NETGEAR ReadyNAS fails on one particular file

linuxreadynasrsync

I currently have a server that nightly backs up its data to a NETGEAR ReadyNAS device using Rsync. This has been working fine for the past few months. However recently, I started to get the following error:

rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Connection reset by peer (104)
inflate (token) returned -5
rsync error: error in rsync protocol data stream (code 12) at token.c(604) [receiver=3.0.6]
rsync: connection unexpectedly closed (229743 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.7]

I've managed to track it down to one particular file that is causing the problem (by using Rsync to copy just that file). The file itself is quite large – about 975Mb. However, there are also other files in the same directory, some as large as 3Gb, and these backup fine.

I'm using the following Rsync command:

 rsync -avz --password-file=/root/rsync.secret <sourcedir> username@readynas::backup

Rsyncing to an Ubuntu Linux host seems to work, so it doesn't look like there's a problem with the rsync client on the server.

I've Googled around but found nothing. A lot of people seem to be saying that adjusting the timeout at the Rsync server end solved similar problems, but if this were my problem I can't understand why it works with the larger files.

Does anyone have any suggestions how to solve this?

Best Answer

I have seen this happen (rarely) before when doing compressed rsync between machines. In all cases where I have seen it the data itself was already compressed to begin with (e.g. video-files or zip's). Most files come through OK. Once in w while a specific file has this problem.

Try it without compression. (no -z flag on rsync.) There could be something about the transferred compressed data that, in rare cases, causes the other end to barf.

Besides: Without compression might actually be faster depening on the the speed of the network and the CPU/RAM capacity available on source and destination. Most (low/mid-end) NAS boxes don't really have the power for it.

Related Solutions

SSH SFTP – Is It Possible to Use rsync Over SFTP Without an SSH Shell?

Unfortunately not directly. rsync requires a clean link with a shell that will allow it to start the remote copy of rsync, when run this way.

If you have some way of running long-lived listening processes on the host you could try starting rsync manually listening for connections on a non-privileged port, but most techniques for doing that would require proper shell access via SSH too, and it relies on the hosts firewall arrangements letting connections in on the port you chose (and the host having rsync installed in the first place). Running rsync as a publicly addressable service (rather than indirectly via SSH or similar) is not generally recommended for non-public data though.

If you host allows scripting in PHP or similar and does not have it locked down so extra processes can not be execed by user scripts, then you could try starting rsync in listening mode that way. If your end is connectible (you are running SSH accessible to the outside world) you could try this in reverse - have a script run rsync on the server but instead of listening for incoming connections have it contact your local service and sync that way. This still relies on rsync actually being installed on the host which is not a given, or that you can upload a working copy, but does not have the security implications of running an rsync daemon in a publicly addressable fashion and talking to it over an unencrypted channel.

Messing around as described above may be against the hosts policies though, even if it works at all, and could get you kicked off. You are better off asking if a full shell can be enabled for that account and either abandoning rsync for that host or abandoning that host and moving elsewhere if they will not do that.

Ssh – rsync connection closing right around an hour

Idle network connection during rsync transfert:

Yes, rsync could stay quiet during disk read and while they have no packet to send...

You said. our firewall is configured to close idle connections that have opened for more than an hour but nothing about the continuous idle duration.

rsync process on both ends, reading whole files to search for difference between packets, based on checksums previously computed.

While there are no difference between files on both ends, they may be no exchange between both ends until differences are found.

There is a little graphic representing a single rsync run between my desktop and his backup:

Trace rsync network traffic

In which we may see some seconds elapsed without datas:

TIME                     BLOCKS           BYTES
                        IN  OUT      IN     OUT
...
Jul 28 15:00:07 2013   449   41   23443 1526403
Jul 28 15:00:08 2013     0    0       0       0
Jul 28 15:00:09 2013     0    0       0       0
Jul 28 15:00:10 2013   143  182   21348 1123986
Jul 28 15:00:11 2013   112  142    6966  523326
Jul 28 15:00:12 2013    31   30    1942    1890
Jul 28 15:00:13 2013    30   28    1853   88038
Jul 28 15:00:14 2013     0    0       0       0
Jul 28 15:00:15 2013    57   66    3954   38301
...

Workaround:

As a workaround, you may use socket based ssh sharing:

First run a plain ssh for hanging socket and connection

ssh -M -o ControlPath=$HOME/.ssh/socket_test destHost ping -n localhost

Than in another window:

rsync -e 'ssh -e none -S $HOME/.ssh/socket_test' /src/file destHost:/dst/file

In this way, only one connection will be used for both ssh sessions. The ping will ensure (a lot of) continuous traffic, than rsync could work quietely, using same network connection.