Idle network connection during rsync transfert:
Yes, rsync
could stay quiet during disk read and while they have no packet to send...
You said. our firewall is configured to close idle connections that have opened for more than an hour but nothing about the continuous idle duration.
rsync
process on both ends, reading whole files to search for difference between packets, based on checksums previously computed.
While there are no difference between files on both ends, they may be no exchange between both ends until differences are found.
There is a little graphic representing a single rsync run between my desktop and his backup:
In which we may see some seconds elapsed without datas:
TIME BLOCKS BYTES
IN OUT IN OUT
...
Jul 28 15:00:07 2013 449 41 23443 1526403
Jul 28 15:00:08 2013 0 0 0 0
Jul 28 15:00:09 2013 0 0 0 0
Jul 28 15:00:10 2013 143 182 21348 1123986
Jul 28 15:00:11 2013 112 142 6966 523326
Jul 28 15:00:12 2013 31 30 1942 1890
Jul 28 15:00:13 2013 30 28 1853 88038
Jul 28 15:00:14 2013 0 0 0 0
Jul 28 15:00:15 2013 57 66 3954 38301
...
Workaround:
As a workaround, you may use socket based ssh sharing:
First run a plain ssh for hanging socket and connection
ssh -M -o ControlPath=$HOME/.ssh/socket_test destHost ping -n localhost
Than in another window:
rsync -e 'ssh -e none -S $HOME/.ssh/socket_test' /src/file destHost:/dst/file
In this way, only one connection will be used for both ssh sessions. The ping
will ensure (a lot of) continuous traffic, than rsync
could work quietely, using same network connection.
Best Answer
Firstly, make sure that the parent directory on the target exists. I.e.
root@1.1.2.2:/var/www/uploads
should exist. I think with your formulation, the trailing.
refers to a directory which rsync tries to create, and it can't do so unless the parent directory already exists. ie the parent isroot@1.1.2.2:/var/www/uploads/$NOW
.Secondly, realise that the behaviour of rsync is subtly different to cp in various ways with the trailing '/' on the file name. I find the safest and most intuitive way to do things is to use a trailing
/
on the end of all directories. Like so:Unlike cp, rsync will reliably copy the content of the directory in the source argument to the content of the target directory, creating the target directory if necessary (though it's parent must exist), and not putting the source directory (ie the parent) inside the target directory if the target already exists.
This is slightly different to what you were doing though in that the way you did it would exclude files with a name starting with
.
in the source directory, and would fail if the list of files being copied was too long (bash expansion capped at a total command line length of around 32K characters if memory serves me right).