PHP5-FPM and ‘ondemand’

I suspect your varnish cache is not caching anywhere near enough of the hits

here's what I would do in your situation:

Lower php max children to 100 or even 50 (if varnish does its job properly you don't need them) also remove the max requests line to allow the php processes not to respawn too quickly and thus prevent APC from being cleared too quickly which is also bad

also IF is not good according to nginx - http://wiki.nginx.org/IfIsEvil

I would change this line:

            if (!-e $request_filename) {
                    rewrite ^(.+)$ /index.php?q=$1 last;
            }

to:

try_files $uri $uri/ /index.php?$args;

If your version of nginx supports it (pretty certain if your nginx version is > 0.7.51 then it supports it)

you should also look at inserting the w3tc nginx rules direct into your vhost file to enabled proper disk enhanced caching of pages (which is faster than APC caching with nginx)

Take a look at the following varnish vcl which I use for sites - you will need to read through and edit a few things for your website - it also assumes that its only WP sites on the server and only 1 site on the server, it can easily be modified for more sites (take a look at the cookie section)

generic vcl: https://gist.github.com/b7332971a848bcb7ecef

With this config I would argue to remove fastcgi_cache to prevent any possible issues with a cache-chain occurring whereby trying locate any stray stale cache entries is more difficult

also tell w3tc that varnish is at 127.0.0.1 and it will purge it for you ;)

I deployed this to a server on Wednesday evening (with a few domain specific modifications) that was handling 2500 active site visitors it reduced load to less than 1 and the approx number of running php children was around 10-20 (this number does depend on number of logged in users and other factors like cookies) this server did have much more ram but the principle is the same, you should be able to easily handle the number of visitors you get at peaks

It's highly unlikely you have enough RAM to have 500 PHP processes running. On my boxes, they typically take up 40-80MB each depending on what they happen to have been doing.

So what happens here is, when you start PHP, it tries to start 400 copies of itself, as that's what you've specified. But you would need at least twice as much RAM as you currently have! So the machine very quickly starts swapping, and grinds to a halt.

Drop these numbers dramatically, until you are well within the bounds of your memory. You don't need 400 simultaneous processes running to handle 5-15 pageviews per second.

You also should set a value for pm.max_requests in case of memory leaks in PHP or another PHP module that you might be using. These are unfortunately far too common.

To get your server back under control, try starting with:

pm.max_children = 50
pm.start_servers = 10
pm.min_spare_servers = 10
pm.max_spare_servers = 50
pm.max_requests = 500

This gives you plenty of room to process both normal traffic and spikes of up to 5x your normal load. You can raise pm.max_children if you start getting much busier and see log entries about running out of children, but check your free RAM before you do; if you ever get to that point you'll probably have to upgrade the server.