Optimizing rsync for transferring huge files quickly

filesrsync

This is what I need:

  • Rsync daemon running on source server. Destination server does a rsync pull as below: 
    rsync -ptv rsync://source_ip:document/source_path/*.abc destination_path/
  • Huge binary files (3 GB to 5 GB) are copied from source machine to destination over a LAN. So no encryption/decryption required. Hence not using SSH. Destination machine directly connects to rsync daemon.
  • The destination folder will always be empty before rsync pull. So I don't have to worry about conditionally updating some files. Rsync pull will always create new files in destination (not update). Hence I don't need rsync to calculate checksums.

I am not satisfied with the amount of CPU consumed by rsync at the receiving end? What rsync options will help me optimize rsync usage in above explained case. What I am concerned about is that rsync might be doing more than what I need. rsync checksums files. I don't need that.

Also would like to see speed improvements.

Changing block-size, disable checksum etc will help?

Best Answer

It seems like you might want to take a look at the --whole-file or -W switch. This is enabled by default if rsync is doing local filesystem copies, but I think the docs recommend its usage if the LAN speed is high and syncing over the network.

Basically, it disables the rsync delta algorithm and just transfers the whole file if it believes it's different.

You shouldn't have to worry about checksumming unless you force -c. By default, it just looks at the last modified time and the file size, and assumes the files are different if those values are different.

Related Topic