My Environment
- CentOS 6.4 X86_64
- Apache 2.4.4
- PHP 5.4.16 (FPM)
- 2 Intel Xeon E5-2620 @ 2.00GHz (8 core, 16 threads in each processor)
- 48GB RAM registered memory.
- 3 Hard Disk 15RPM 145GB in RAID0 (by BIO
Interesting Variables
<IfModule mpm_event_module>
StartServers 2
ThreadLimit 196
MinSpareThreads 96
MaxSpareThreads 192
ThreadsPerChild 96
MaxRequestWorkers 192
MaxConnectionsPerChild 96
</IfModule>
Apache Server Status
Server Version: Apache/2.2.4 (Unix) OpenSSL/1.0.1e mod_fastcgi/mod-fastcgi-SNAP-0910052141
Server Built: May 24 2013 16:48:07
Current Time: Monday, 17-Jun-2013 09:48:11 COT
Restart Time: Monday, 17-Jun-2013 08:35:14 COT
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 1 hour 12 minutes 57 seconds
Server load: 0.05 0.10 0.09
Total accesses: 14144 – Total Traffic: 349.7 MB
CPU Usage: u.28 s.25 cu0 cs0 – .0121% CPU load
3.23 requests/sec – 81.8 kB/second – 25.3 kB/request
1 requests currently being processed, 191 idle workersPID | Connections | Threads | Async connections | total | accepting | busy | idle | keep-alive | closing ============================================================== 18997 | 3 | yes | 1 | 95 | 0 | 3 18485 | 0 | yes | 0 | 96 | 0 | 0 ============================================================== Sum | 3 | | 1 | 191 | 0 | 3
Error Log
The error message is
[Mon Jun 17 09:32:45.680842 2013] [mpm_event:error] [pid 8574:tid 140185091581760] AH00485: scoreboard is full, not at MaxRequestWorkers
This appears every few seconds. I don’t understand it. How can I fix it?
Best Answer
We had the same problem on Apache 2.4.6. After monitoring the server and adjusting the setting for several hours it appears to us that Apache may have a bug. What appears to happen is that the server processes occasionally goes into the
G
state (Gracefully finishing) and restarts to accept new requests, that's normal. What is not normal is that for some reason this can take up to a few minutes to restart. If you only have a few server process running and they all go into theG
state at the same time then your scoreboard fills up and you won't be able to server any more requests.What we did was increase the number of servers so there is a less of a chance that they will all go into the
G
state at the same time. Also make sure you allocate at least 25 threads (MaxRequestWorkers
) for each server process because that appears to be the default (i.e. if 5Servers
x 25ThreadsPerChild
= 125MaxRequestWorkers
). You can changeThreadsPerChild
if you like, we left it at default. If you don't allocate enough threads the additional servers will not start. We leftMinSpareThreads
at the default value which is 25 and the default forMaxSpareThreads
which is 75. If you do modify these settings, the value forMaxSpareThreads
must be greater than or equal to the sum ofMinSpareThreads
andThreadsPerChild
. AlsoMaxRequestWorkers
must be equal to or less than theServerLimit
.Here is what worked for us but it might not be the best configuration for you.
Edit: This is a confirmed bug in httpd's mpm_event module which might not be fixable through configuration.
The linked bugtracker entry has a presumed patch and more discussion about how to fix this until a new version of the event module is officially released.