We have an Apache webserver running on CentOS 5.6 (installed via yum). Today we have had an expected influx of new users — however unexpectedly we seem to have hit a limit of 300 accesses per second — I have a graph (which Stack overflow is not leting me upload), but it shows that three times today (when our site almost ground to a halt) the "accesses / second" hit 300 and stayed for a while. As I type the current figure is 292.23 and the max today is 300.41.
We are running PHP inside the apache process, and that is connecting to a MySQL server (different physical box). However we're happy that MySQL is responding fine.
We've upped our limits in the httpd.conf from the default 256, however this hasn't had an effect on the magic 300 figure. We've searched both the php.ini and httpd.conf file for 300 and cannot find it. Below is the prefork config we are now using:
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 400
MaxClients 400
MaxRequestsPerChild 4000
</IfModule>
If anybody has any clue to what is causing this 300 limit I'd greatly appreciate it.
Best Answer
There are all sorts of system-performance-related possibilities, but the fact that the limit is such a nice, round number is rather suspicious -- you wouldn't expect such a neat limit if the bottleneck were, say, CPU or disk I/O (or even network). What's even more suspicious is that it's per second -- if it were a limit to Apache or MySQL, you'd expect a limit on concurrent connections or requests, but a limit of 300 per second sounds like something else.
What is sitting upstream from you, network wise? If there's any firewalls, load balancers, DDoS mitigation systems, or other L3+ devices up there, I'd take a poke at them to see if there's anything in there that could be causing it.
Do you know what anyone getting above your 300/second limit sees? (You might have to hammer the site yourself a bit to see what happens) Are there log messages anywhere that might give a clue?