Cygwin’s RSYNC for large data transfer


I'm using rsync from Cygwin to do a large scale data transfer from an aging HP MSA 1000 to a new DAS attached to a different server. I have a daemon running on the remote server in read only mode and a local copy writing the files to disk.

One of my servers is an image repository with over a million files spread across about 300 directories. Each file averages only a couple hundred kilobytes. More so than any other box this one is proving problematic.

The rsync process will work for a while – some times 20 minutes, some times an hour – and then it simply quits and sits idle at a given file name.

I have verified that the file isn't corrupt on the remote server and that the file is successfully created on the local drive. I ran the rsync client in -vv mode, which returns nothing. I checked out the logs created by the daemon. I looked at the network utilization on the interface, which is sitting idle. I looked at the AV settings to see if anything could pose a problem there. I even updated to the latest release of Cygwin.

What do I need to in order to keep this connection up?


The client system is using the command

rsync.exe server::Drives/f/Repo/ /cygdrive/T/Repo --archive -P  -vv

The server is using the command
rsync.exe –daemon –no-detach –config "rsyncd.conf"

The contents of rsyncd.conf:

use chroot = false
strict modes = false
hosts allow =
log file = c:/rsyncd.log

path = /cygdrive
read only = yes


The file server is 2003, the disk type on the array is GPT and the size is of the array is about 4 TB.


Stranger.. It looks like the process is reliably erroring out at about 175,000 files. Rsync runs fine when I pick the same directory it has problems with one at a time.


rsync  version 3.0.9  protocol version 30
Copyright (C) 1996-2011 by Andrew Tridgell, Wayne Davison, and others.
Web site:
    64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
    no socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes

A similar failure occurred when going from the same set of files with Cygwin to a Linux install. It didn't happen until several hours later than normal however.

Best Answer

You don't say the total data volume, so I can't be sure, but I have used rsync before on comparable sized repositories. So far, I haven't run into this particular issue. The vast majority of my use is on Linux/Unix devices though, with only occasional Windows runs.

To figure out what's actually going on, you might want to try running rsync with additional -v arguments, see if you can get enough detail to see what it's doing when it hangs. If that doesn't help, you might look into running rsync under strace or ltrace.

As a workaround, you could try --include-from=FILE to specify a list of directories to include, and then list out the 300 directories you mentioned. I'm not sure if that would run into the same issue you're seeing, or not.

If that doesn't work, you could loop through the directory listing and rsync each one individually with something like:

for DIR in $(cat file_of_dirs); \
do rsync.exe server::Drives/f/Repo/$DIR /cygdrive/T/Repo --archive -P -vv; \

(Written for bash/bourne shell, since you said you have Cygwin installed. Could be done one line without the trailing slashes at the end of the lines.)

Obviously this doesn't answer the question of why rsync is hanging for you, but it should get your files syncing again.