Is there an ftp server that behaves as a 'distribution front end' to multiple other servers? So that when I upload a file, it accepts the contents, puts them on all of a list of other ftp servers and (importantly) does not confirm success of the upload until it's on all of the other servers?
Alternatively, if it could wait until (say) rsync had replicated the uploaded file to all the other servers before returning success (or, more generically, wait for some external command to complete before returning success).
Background:
We have an app that uploads files to a repository (using ftp or sftp), then immediately instructs a device to download the file (via http).
We need the repository to be load-balanced/highly-available/resilient. Our corporate hosting standards do not permit shared storage.
What we do with other related apps is have several ftp/http servers, and manually upload files to all of them before telling the app (and then the device) to use them. A load balancer distrbutes download requests. This works becasue those apps do not do the uploading, instead we configure them to use the URL of the previously uploaded files. The problem app doesn't do this, it does the upload itself.
We could use rsync or similar to replicate the files uploaded by the problem app to the multiple servers, but the use of these files is immediate, so they may not have replicated to the other servers when a request for them is received. The app cannot be configured to have a delay here.
But if the ftp server didn't return until the file had been replicated (either by the server itself doing all the replication/upload to other servers, or by it waiting for an external command to complete), then the app wouldn't tell the device to use the files until we knew they were everywhere. And it would all work.
Any pointers to suitable servers? Other ideas for solving the problem? (altering the app isn't possible in the timescales, unfortunately)
Best Answer
If you need to use FTP, you could write a script (perhaps a Python program, or in any language that offers a convenient FTP library) that your upload program runs immediately after completing the upload to the 'master' server. This script would scan the FTP sites that are supposed to be replicated to, and won't exit until it sees those files. On the master server, you would have another script that monitors the filesystem (such as using Linux's inotify) and when it sees new or modified files, it uploads them to the slave servers.
Alternately, you could use a replicated filesystem. This moves the problem from a homebrewed set of scripts at the application layer to a layer designed to deal with replicating files. Check out Tahoe-LAFS. I quote the relevant sentence: