I created a sub-domain for static content to be able to serve it more efficiently by several load-balanced web servers. This static content is updated automatically, at a rate of ~1k files/day.
Right now I use rsync to update the servers in a master/slave way but since the content has a growing number of 100k+ files, it takes more and more time and puts an increasing I/O load on both the master and the slaves.
I cannot use the solution I proposed on the Improve rsync performance question since I cannot know which files are modified without stat-ing them all, and that wouldn't solve the increasing I/O cost. I also have to handle file deletion.
I thought about using something like a r/o NFS on the slaves but that could somewhat defeat the load-balancing effect and put a gratuitous SPOF.
Btw, the servers are running AIX, but I'm also interested in a solution in a more generic context.
Best Answer
Why don't you just use reverse proxy, such as Squid?