I'm preparing a server setup for a dating website who need to be able to process around 5'000 – 10'000 req/sec to the main site.
My idea was to do it this way:
Server for static content (css, js, img) :: Varnish cache => nginx webserver
Server for member photos :: [1] Varnish cache => [n] nginx webserver
Server for member videos and streaming :: nginx webserver with Erlyvideo or Wowza (only paid member)
Server for web app :: [1] nginx (as cache if needed) => [n] HipHop webserver or Apache mod_php nolog
is this OK or is there a better way?
We developed the web app with a custom framewrok and optimized as much possible, the result is that the execution time per site don't take more then 0.05 sec (no cache) 0.0009 sec (with apc or memcached) on a 3 yr old development webserver with apache and mysql.
I'm not sure how much Server we will need for the web app and for the DB to can handle this amount of requests.
Best Answer
put a load balancer in front of it all so you can pull servers in and out, spread the load [preferably for all types of content - both static and dynamic]. [you can use nginx or varnish as reverse proxy for that and if that's not enough - put cluster of LVS or haproxy servers to spread the load across multiple reverse proxies].
make sure your generators of dynamic content are stateless or at least would allow partitioning with sticky sessions so you can easily increase capacity with liner investment.
maybe it makes sense to offload static hosting to some cdn?