Php – What to do after hitting the dreaded 256 max connections Apache Limit

apache-2.2httplampPHP

After scratching my head trying to figure out why my site was responding so slowly even though the server resources are fine, I finally checked the Apache status and found:

78 requests/sec - 0.7 MB/second - 8.5 kB/request
256 requests currently being processed, 0 idle workers

It appears that my apache is literally maxed out with connections. Anyone trying to visit my site gets put on a "waiting list" until Apache is free again.

It seems I have two options.

A) Raise the max connections limit above 256. Although according to this article it's not so easy:

By default, the MaxClients parameter has a compiled in hard limit of
256. This can be changed by recompiling Apache however. Some distributions, or hosting companies raise this limit to a very high
value, such as 512 or even 1024 in order to cope with large loads.

B) Locate scripts that are taking up too much time. This seems a lot more tricky to me, since most apache processes just appear and then disappear again. Also, my sites PHP scripts are optimized pretty well…and once again, server resources are fine:

Server load 2.69 (8 CPUs)   
Memory Used 25.33% (2,039,108 of 8,048,804) 
Swap Used   1.32% (54,156 of 4,095,992)

Which option (if either of them) should I chose and how should I do it?

EDIT

More information:
Server Version: Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0-fips DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635

HTTP Conf: http://pastebin.com/yBeLt6mP

Parital Request sample: http://pastebin.com/vzUVDMPR

Toggle Text-Wrap if the paste-bins show up weird.

Best Answer

That article is inaccurate; MaxClients can be raised above 256 when using the prefork MPM (which is what I assume you're using currently based on your description of the problem). From the documentation:

For non-threaded servers (i.e., prefork), MaxClients translates into the maximum number of child processes that will be launched to serve requests. The default value is 256; to increase it, you must also raise ServerLimit.

ServerLimit is the one that has the hard-compiled limit, but it's way past where you should ever reach without your server running into some other bottleneck. Documentation:

There is a hard limit of ServerLimit 20000 compiled into the server (for the prefork MPM 200000). This is intended to avoid nasty effects caused by typos.

So, if you want to raise your client limit to something like 512, then:

MaxClients 512
ServerLimit 512

You should also take a look at which MPM you're using, as MPMs other than prefork are better for scale. See here for more information.