Rsync – Transfer Files Between Two Remote Servers

rsync

I have 3 servers. Call them A, B, and C. B and C are remote and only have A in their hosts.allow.

What I need to do is log into A, and then rsync over SSH the files from B to C. I want to use rsync because many of the files are similarly present on both B and C.

I could try to rsync from B -> A and then from A -> C except I don't have nearly enough disk space on A, so I'd have to find some way of "chunking" the files and I don't see an obvious way of grouping them, and even if I did, I'd have to babysit it too much, with deleting files off A's disk, restarting the process with the next chunk etc.

So is it somehow possible for me to rsync between the two servers? I don't think I will be able to get anyone to change the hosts.allow for me 🙁

Best Answer

You should be able to set up an SSH tunnel from A->B and from A-C, and then rsync directly from B->C by going across that tunnel.

It's going to be a bit complicated, but basically you do the following:

Set up an rsync daemon on a high port (say 20000) on B
Set up a LocalForward A:20000 to B:20000 with ssh
Set up a RemoteForward from C:20000 to A:20000 with ssh
On C, rsync -av /path rsync://localhost:20000/path

Bear in mind that all this traffic will be going both in and out the network on A and being encrypted, so it will probably be pretty slow. it might be worth seeing if B is actually firewalled from C, you might be able to run the rsync server on a high port and connect directly.

Related Solutions

Linux – Copying a large directory tree locally? cp or rsync

I would use rsync as it means that if it is interrupted for any reason, then you can restart it easily with very little cost. And being rsync, it can even restart part way through a large file. As others mention, it can exclude files easily. The simplest way to preserve most things is to use the -a flag – ‘archive.’ So:

rsync -a source dest

Although UID/GID and symlinks are preserved by -a (see -lpgo), your question implies you might want a full copy of the filesystem information; and -a doesn't include hard-links, extended attributes, or ACLs (on Linux) or the above nor resource forks (on OS X.) Thus, for a robust copy of a filesystem, you'll need to include those flags:

rsync -aHAX source dest # Linux
rsync -aHE source dest  # OS X

The default cp will start again, though the -u flag will "copy only when the SOURCE file is newer than the destination file or when the destination file is missing". And the -a (archive) flag will be recursive, not recopy files if you have to restart and preserve permissions. So:

cp -au source dest

Rsync –files-from equivalent

I suspect that xargs is your friend in this instance.

Completely untested:

 # cat file | xargs -IREPLACE rsync -v REPLACE server:/path

xargs will take data from stdin and add it to the commandline one parameter per input line until it fills the command line length, and then runs a separate command until all the input lines have been used up. This will mean that rsync could well run more than once. This shouldn't be a problem in this case, but could be with other commands.

Usually xargs just sticks the arguments on the end of the line, but with rsync you'd want to have the destination last, so we use the -I option to replace a string with the arguments. (This is the bit that's untested). I'm also a little unsure what rsync will do with the paths, so please test this carefully before using it in anger.

If you do have a problem with xargs, you could use something like:

for i in $(cat file); do rsync -v $i server:/path; done

If your file is larger, you probably want to use a while/read loop instead. This will be much slower as it will do a single file at a time.

Best Answer

Related Solutions

Linux – Copying a large directory tree locally? cp or rsync

Rsync –files-from equivalent

Related Topic