My application use rsync to copy files from network driver to local as cache to improve performance.
But since my application might be run in parallel, so as the rsync, and there could be a case that 2 instance of rsync being run in parallel against the same source and directory.
I've run into a problem that the destination doesn't sync completely (leaving one sub directory empty). so I doubt it is a problem of a rsync, so
- Is rsync safe to copy files from same source to the same destination in parallel?
- Is there any option available to ensure the correctness of such operation.
The options I am using is:
rsync -a --copy-unsafe-links --delete
Best Answer
It's probably safe to run rsyn in parallel, but not efficient, since most likely the files will be transfered twice since rsync makes up the list of files to transfer at the beginning. There is of course always the slim chance that the same file is transfered simultaneously by both rsyncs, but the contents should be the same afterwards.
To prevent double runs, usually I write a script that checks for, and otherwise
touches a lockfile which is removed at the end. As an option, you can use a trap in your script to clean up the lockfile if you break off the script or when an error occurs (use 'help trap' within your shell). As far as I know, there is no such locking mechanism in rsync itself (which would be hard to implement anwyay).