Each night I copy several virtual disks with rsync from one Linux Debian computer to another Linux Debian.
Most files are raw images with "holes" into them: some parts were never written to, so remain unallocated on disk.
rsync hangs on one file, always the same. The hang occurs after 50 Gb transfered, and each time. I'm not sure if this is always at the exact same point, but ls -sh displays 50 Gb.
This is a 800 Gb file containing 151 Gb (so 649 Gb are unallocated). Some other virtual disks have similar figures, and rsync works well on them.
I have the exact same behaviour if I use rsync to update the file locally, without any network involvement (with --no-whole-file, this is a requirement, see later).
Once rsync is stalled, it uses one CPU core to 100% and zero disk activity on the receiving side (this is a pull request, so rsync is ran from this side) and zero CPU and zero disk on the sending side.
I let it ran during several hours.
Ctrl+c immediately stops rsync.
When ran to copy locally, once stalled I also have one CPU core to 100% and zero disk activity.
The only exception I found is when I rsync this file to a new location (ie the destination file does not exist). So I suspect the problem is related to the comparison between old and new data.
I use --inplace to limit writes to the destination disk, because snapshots are then used after each backup. So this option is a requirement, and rsync too, except if I found a tool able to update only changed parts of these files.
Best Answer
rsync is known to have this kind of problem on huge files because of an inefficient algorithm for the internal hashes buffer.
You have to use the
--block-sizeoption with a big value. But current versions doesn't allow to use more than 128 kB. Some webpages tell to use--block-size=10485760 --protocol=29but in my case it is refused by rsync.