A way to speed up rsync in two phase process

rsync

Here is what I do:

rsync everything
enter maintenance
rsync changes since first rsync
leave maintenance

The first rsync is syncing the mayor changes without locking the system. It can run a long time and that is fine.

But the second rsync should finish as quickly as possible and normally finds none or only a few changes. Yet since this are so many files it takes quite long.

Is there a trick I can use because I know I synced it just before?

Here the rsync flags I use:

rsync --partial --progress --delete --archive --verbose --compress --links --times

Best Answer

Some tips:

  • when synching many files, the uid-to-username mapping process can take a significant amount of time; so, try adding the --numeric-ids option to your rsync invocation
  • on both sides, schedule a find <path> > /dev/null some time before the maintenance rsync; this will preload the metadata caches on both nodes, greatly speeding up rsync execution
Related Topic