Ubuntu – Apache2: Server reached MaxClients setting, consider raising the MaxClients setting

I'm encountering issues on a dedicated Ubuntu 12.04 LTS (Apache/2.2.22), PHP 5.3 with APC.

Every few hours I see this happen:

[Fri Aug 17 15:36:51 2012] [error] server reached MaxClients setting, consider raising the MaxClients setting

This causes the server to drop/refuse all connections.

When I go to restart apache:

[Fri Aug 17 15:38:14 2012] [warn] child process 8946 still did not exit, sending a SIGTERM
[Fri Aug 17 15:38:14 2012] [warn] child process 7496 still did not exit, sending a SIGTERM
[Fri Aug 17 15:38:16 2012] [warn] child process 8946 still did not exit, sending a SIGTERM
[Fri Aug 17 15:38:16 2012] [warn] child process 7496 still did not exit, sending a SIGTERM
[Fri Aug 17 15:38:18 2012] [warn] child process 8946 still did not exit, sending a SIGTERM
[Fri Aug 17 15:38:18 2012] [warn] child process 7496 still did not exit, sending a SIGTERM
[Fri Aug 17 15:38:20 2012] [error] child process 8946 still did not exit, sending a SIGKILL
[Fri Aug 17 15:38:20 2012] [error] child process 7496 still did not exit, sending a SIGKILL
[Fri Aug 17 15:38:21 2012] [notice] caught SIGTERM, shutting down
[Fri Aug 17 15:38:21 2012] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
[Fri Aug 17 15:38:22 2012] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.2 with Suhosin-Patch mod_ssl/2.2.22 OpenSSL/1.0.1 configured -- resuming normal operations

Since this started happening, i upped the configs for apache (it's a 6 core machine with 16Gb of ram, so I figured I'd up it a lot).

<IfModule mpm_prefork_module>
    StartServers          8
    MinSpareServers       5
    MaxSpareServers      10
    ServerLimit         500
    MaxClients          500
    MaxRequestsPerChild 1000
</IfModule>

The server serves nothing but PHP (no static files), and page loads are never more than a few seconds…

I've been watching mod_status (just activated it), yet I see nothing weird…

Server uptime: 20 minutes 46 seconds
Total accesses: 9206 - Total Traffic: 641 kB
CPU Usage: u147.52 s17.21 cu0 cs0 - 13.2% CPU load
7.39 requests/sec - 526 B/second - 71 B/request
2 requests currently being processed, 8 idle workers

___...W__..._W.....__...........................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
....................................................

I even tried running some stress tests with apache ab to see if I could reproduce it… no luck. 🙁

EDIT: I've just made a big find:

DEBUG: WARNING - require(): GC cache entry '/var/www/common/revisions/1061/library/core/bootstrap.php' (dev=64513 ino=1577080) was on gc-list for 3601 seconds /var/www/common/revisions/1061/library/core/cache/disk/dao.php (16)

This error is what is causing the problem, I believe it is related to APC.