Ssh – rsync over ssh backup failing after relocation of server

freebsdrsyncssh

I've got two FreeBSD machines set up; one serves video data and the other is the backup for the first. At this point I've got around 4TB of data. I add files to the video server a few at a time, and was planning to use rsync over ssh to keep the backup machine up to date.

I did the initial, large backup with both machines hooked up to the same subnet at the lab with no problems using rsync. Then, when I moved the backup machine off-site (but still on the university network), I attempted a sync without changing anything other than the IP (as the machine is now on a different subnet) and got the following error:

2010/03/22 15:55:21 [1260] rsync: connection unexpectedly closed (6340840244 bytes received so far) [receiver]
2010/03/22 15:55:21 [1260] rsync error: error in rsync protocol data stream (code 12) at io.c(601) [receiver=3.0.7]
2010/03/22 15:55:21 [1258] rsync: connection unexpectedly closed (60 bytes received so far) [generator]
2010/03/22 15:55:21 [1258] rsync error: unexplained error (code 255) at io.c(601) [generator=3.0.7]

The script that handles the backup hasn't been changed, nor has the crontab that invokes it. Does anyone have any ideas about what might be causing the hiccup? I was under the impression that it might have something to do with the ssh connection timing out or something along those lines, but am not entirely clear on how to diagnose the cause of the problem.

Best Answer

It appears like your connection has been established and data has been transmitted so it doesn't seem like the issue is related to your known_hosts or establishing your connection.

If there was a NAT between the 2 computers and rsync needed to calculate the checksum for a large file then it seems like it would be possible that the time it takes to calculate the checksum would exceed the length of time that the translation for your connection is saved in the state table. I would be tempted to add an option like this ServerAliveInterval 60 on the machine acting as the ssh client to force keep alives to be sent.

Idle network connection during rsync transfert:

Yes, rsync could stay quiet during disk read and while they have no packet to send...

You said. our firewall is configured to close idle connections that have opened for more than an hour but nothing about the continuous idle duration.

rsync process on both ends, reading whole files to search for difference between packets, based on checksums previously computed.

While there are no difference between files on both ends, they may be no exchange between both ends until differences are found.

There is a little graphic representing a single rsync run between my desktop and his backup:

Trace rsync network traffic

In which we may see some seconds elapsed without datas:

TIME                     BLOCKS           BYTES
                        IN  OUT      IN     OUT
...
Jul 28 15:00:07 2013   449   41   23443 1526403
Jul 28 15:00:08 2013     0    0       0       0
Jul 28 15:00:09 2013     0    0       0       0
Jul 28 15:00:10 2013   143  182   21348 1123986
Jul 28 15:00:11 2013   112  142    6966  523326
Jul 28 15:00:12 2013    31   30    1942    1890
Jul 28 15:00:13 2013    30   28    1853   88038
Jul 28 15:00:14 2013     0    0       0       0
Jul 28 15:00:15 2013    57   66    3954   38301
...

Workaround:

As a workaround, you may use socket based ssh sharing:

First run a plain ssh for hanging socket and connection

ssh -M -o ControlPath=$HOME/.ssh/socket_test destHost ping -n localhost

Than in another window:

rsync -e 'ssh -e none -S $HOME/.ssh/socket_test' /src/file destHost:/dst/file

In this way, only one connection will be used for both ssh sessions. The ping will ensure (a lot of) continuous traffic, than rsync could work quietely, using same network connection.

Best Answer

Related Solutions

Ubuntu – Rsync process is killed abruptly during backup over SSH

Ssh – rsync connection closing right around an hour

Idle network connection during rsync transfert:

Workaround:

Related Topic