How to increase the disk write buffer for rsync on Mac OS X

disk-cachemac-osxrsync

I have to copy some TB of data from one external drive to another (mostly ~400 MB files).
The source is NTFS-formatted USB2 and the best read speed I can get from it is 7MB/s.
The target is HFS+ USB3 and the write speed is considerably higher than that.

The NTFS filesystem isn't OK, and only NTFS-3g (slow, userspace) can read all the files in it, not even Windows can do it or I'd use that.

I'm using rsync -va /Volumes/source /Volumes/target to perform the copy.

Is there a way, in rsync or OS X, to cache the writes, so that the target doesn't need to be permanently writing at a slow speed for days? Something like filling 1GB or 2GB or even 4GB of RAM before writing? Thus the target drive would spend a few days doing

write - rest - rest - rest - rest - rest
write - rest - rest - rest - rest - rest
write - rest - rest - rest - rest - rest
...

instead of spending all those days continuously writing at a slow speed.

Another possibility, given that most files are large, would be to have rsync only write any given file after it's finished reading it, but I found no way to do that either.

Or is it better to just spend the whole time continuously writing at a slow speed?

Best Answer

First, you're going to face an eternally long and difficult process reading from NTFS, without a good driver config.

Try http://www.macupdate.com/app/mac/26288/ntfs

That said, you're quite likely having issues with the original drive, if you're running that slowly.

There is no good way to change the cache process on official rsync

you might find that cp -rvn is a faster option in this situation.

Related Topic