Using rsync as a queue

rsync

Is it possible to use rsync (or similar) in a way that lets me queue requests?

I have a script that processes photos (creates jpegs of various sizes) then uploads them to a server.
Version 1 of the script did all the processing then rsynced everything at the end.
This wasn't very efficient so I'm trying to parallelise it.

So I have a version 2 script that processes a single photo then rsyncs just the jpegs it has created in the background.
The problem with that script is that it is spawning a new rsync process for each photo it has processed.
It's fine if the processing takes the same time as an upload but if the upload takes longer I'll end up with dozens of rsync processes slowing to a crawl.

Is there a way to pass the upload requests to as single rsync process so it can queue them somehow?
Or is there another application that can do similar?

I'm running this from a OS X client and uploading to a Linux server.
My script that controls it all is written in Ruby.

Thanks all.

Best Answer

Use two threads:

thread A just processes your photos. send a signal when finished.
thread B does rsync in a loop, maybe waiting a few seconds (or a few photos) between finishing one and starting the next. Don't start two rsync's in parallel. Finish the loop after the first rsync that started after the signal from A was received.

This way you're getting advantage of the fact that rsync transfers only changes. All but the last iterations will send at least one partial file; but it's no permanent damage, because it will be completed on the next iteration.

Be sure that the last rsync iteration is started after the last photo was processed.

Related Solutions

Using rsync to quickly upload a file that is similar to another file

I'm putting together a deployment script which tars up a directory of my code, names the tar file after the current date and time, pushes that up to the server, untars it in a directory of the same name and then swaps a "current" symlink to point at the new directory.

Personally, I think you should skip using tar, and instead look at using the --link-dest or --copy-dest feature of rsync. The link-dest function is pretty cool it will know to look at the previous sync of the directory, and if the files where identical it will hardlink them together skipping the need to retransfer the file each time.

mkdir -p /srv/codebackup/2009-10-12 \
         /srv/codebackup/2009-10-13

# first backup on 10-12
rsync -a sourcehost:/sourcepath/ \
         /srv/codebackup/2009-10-12/

# second backup made on 10-13
rsync -a --link-dest=/srv/codebackup/2009-10-12/
         sourcehost:/sourcepath/ \
         /srv/codebackup/2009-10-13/

Your second run of rsync will only transfer changed files. Identical files will be hard linked together. You can delete the older tree and the new backup will still be 100% complete. You will save a lot of storage space since you will not be keeping multiple copies of identical files.

Archive mode in rsync

It's all of these:

-r, --recursive recurse into directories

-l, --links copy symlinks as symlinks

-p, --perms preserve permissions

-t, --times preserve modification times

-g, --group preserve group

-o, --owner preserve owner (super-user only)

-D same as --devices --specials

--devices preserve device files (super-user only)

--specials preserve special files

It excludes:

-H, --hard-links preserve hard links

-A, --acls preserve ACLs (implies -p)

-X, --xattrs preserve extended attributes

It's perfect for backups. My "default" set of switches is -avzP - archive mode, be verbose, use compression, preserve partial files, display progress.

Note: Invariably when the descriptions say "preserve", it means make the destination be like the source.

Best Answer

Related Solutions

Using rsync to quickly upload a file that is similar to another file

Archive mode in rsync

Related Topic