Nginx – Is this a sensible way to scale Nginx for static content serving

nginxscalability

I need to set up some VPSs for serving static content (many small files). I plan on using Nginx for this, and would like to set it up so that we are able to scale out relatively easily. The requirements are:

  • Many files (at least hundreds of thousands).
  • Small file sizes (less than 10KB).
  • Files are being added constantly by an application on neighbouring servers.
  • New files must be immediately available to all Nginx servers.

My current plan is:

  • Have a 'master' server with an NFS share containing all of the files.
  • Application producing new files interacts only with master.
  • Have multiple Nginx servers mounting this NFS share.
  • Load balance across Nginx instances.

One obvious problem with this is that the 'master' server is a single point of failure (any remedy for this?). Are there other potential issues which I have overlooked? Are there elements here which will not scale well in this way? Would anyone suggest an alternative approach?

Regarding memory requirements, I'm assuming I should give each Nginx server as much as possible so that hot files can be cached (by OS? Nginx?) an not have to be reqested form the NFS share constantly.

Lastly, am I crazy not to use a CDN?

Best Answer

NFS does not scale. It adds latency to every request and will eventually become too big a bottleneck. We have a similar issue at work, but with photos (so, much larger files) and wrote our own software to shard and distribute them. For a few GB of files like you have, you might be able to get away with the upload process doing an HTTP PUT to all servers and doing resyncs when servers have been offline.

Or tackle it another way: have a (set of) central server(s) with all files and caching reverse proxies (squid, pound, varnish) that actually serve the files to customers.

And you're not crazy not to use a CDN. You're crazy if you don't investigate whether it's worthwhile though :-)

Related Topic