Rsync over SSHFS hangs

rackspace-cloud-sitesrsyncsshfs

I have sshfs setup to connect to another system (Rackspace Cloud Sites) that I don't have ssh access to (but for some reason I can use sshfs? Go figure). I'm attempting to rsync files from the sshfs mount to my local disk. It's several thousand small files (1k-200k). Sometimes, the rsync will just pause and hang for a while on files that are very small. It will pause on, say a 10k text file for like 5 minutes, then it will continue.

Is there anyplace I can look on my machine to determine why rsync would be hanging like this? Or is there a good chance it's simply a problem on the other end that I can't do anything about?

My rsync options are simply -avrP.

Best Answer

Alright, I'm going to take a stab at this, because I think my idea makes sense.

You are dealing with multiple caches in this case, and that's what is tripping you up, I think.

The first thing that rsync does is to determine which files it needs to transfer. It usually does this by spawning an instance of rsync at the remote side, reading the metadata for each of the files in the directory on the source, while at the same time reading metadata for the local files, and then the two metadata sets are compared. Anything newer (or different, depending on the rsync options) gets transferred.

You don't have a "remote side", according to rsync. You're working "locally", so it will iterate over both directories, the source and the destination.

This is very disk intensive, particularly with a ton of small files - the more files, the more discrete disk operations. This causes a lot of disk thrashing, plus it fills the cache with the metadata from those files.

Notice that this happens all the way down the stack. Your local machine caches metadata from the FUSE filesystem you've got mounted over ssh AND the local directory. The remote machine caches metadata from the local disk mount. And the VM host that your remote machine is running on is almost certainly overcommitted and giving you ballooned memory.

I suspect that it's very likely that you're crossing thresholds when it freezes, and everything has to catch up and either decache or swap.

I would be very interested to see if this happens when you do rsync over ssh without the disk mount.

Related Solutions

How to Use Rsync Over FTP

You don't. rsync can't do that for you, it is a protocol of its own and doesn't work over FTP.

You might, however, want to try csync. IIRC it provides rsync-like behaviour over HTTP. I can't comment on whether it works over FTP, you'll have to try it.

Linux – Copying a large directory tree locally? cp or rsync

I would use rsync as it means that if it is interrupted for any reason, then you can restart it easily with very little cost. And being rsync, it can even restart part way through a large file. As others mention, it can exclude files easily. The simplest way to preserve most things is to use the -a flag – ‘archive.’ So:

rsync -a source dest

Although UID/GID and symlinks are preserved by -a (see -lpgo), your question implies you might want a full copy of the filesystem information; and -a doesn't include hard-links, extended attributes, or ACLs (on Linux) or the above nor resource forks (on OS X.) Thus, for a robust copy of a filesystem, you'll need to include those flags:

rsync -aHAX source dest # Linux
rsync -aHE source dest  # OS X

The default cp will start again, though the -u flag will "copy only when the SOURCE file is newer than the destination file or when the destination file is missing". And the -a (archive) flag will be recursive, not recopy files if you have to restart and preserve permissions. So:

cp -au source dest

Best Answer

Related Solutions

How to Use Rsync Over FTP

Linux – Copying a large directory tree locally? cp or rsync

Related Topic