We recently switched to apache 2.4.18 on our production server and went all https/ssl (HTTP/2 with http2_module) with our website with about 100.000 visits per day.
We're using the worker MPM and PHP 5.6 via PHP-FPM (proxy_fcgi) since event MPM seems not yet to be working properly when using https.
Settings are:
<IfModule mpm_worker_module>
ServerLimit 16
StartServers 8
MinSpareThreads 64
MaxSpareThreads 256
ThreadLimit 64
ThreadsPerChild 64
MaxRequestWorkers 1024
MaxConnectionsPerChild 10000
</IfModule>
After the server is restarted, the number of threads on our Debian Jessie installation increases to several thousands after a few hours and apache error log fills up with these messages:
AH00288: scoreboard is full, not at MaxRequestWorkers
/server-status seems to indicate some weird "W sending reply" threads which don't do anything but have a very long running time:
W___KRK___KKC__C___KR__K__KKKRKKKKK___K__CC__KR____K__KK________
__KWR_KK_K_C__KC_KKKKKKK__KK__C_KKKKCK_KKK_C_K____WRKK_K_KKR_K_K
KK_CRKKKKRKKKKK__KK_WKCKK_KKKK_KWKKKWKKRRR__KK__K____KKKCK_KKKKK
_KK___K_W__K___CK_C___K____CRK____CKK_______KKKKKKCK___KCKK_KKK_
KK_KK_KK_KKKKK_K_KKK_KRKK_C_CKKR_C_KW__K_KWRK_KK_R_KR_K__KKK__K_
..............................C.........................W.......
...............................W..............................WW
............W.........W.........................................
.........W..........W....................W......................
.................W.......W......................................
.........................................................W......
CKKKKK_KK_KKKK_CKKKRK_RKRKKKKRRKKRKK_KC_KRKKK_KKKKKRKRRKKKKR_RK_
.............................................................W..
................................................................
................................................................
................................................................
All "W" threads look like this: Running for 1453710297 seconds which can't be true since the server was just restarted yesterday and nothing else.
5-0 21872 0/0/0 C 0.00 1453710297 0 0.0 0.00 0.00
5-0 21872 0/0/0 W 0.00 1453710297 0 0.0 0.00 0.00
6-1 17300 0/0/0 W 0.00 1453710297 0 0.0 0.00 0.00
6-1 17300 0/0/0 W 0.00 1453710297 0 0.0 0.00 0.00
6-1 17300 0/0/0 W 0.00 1453710297 0 0.0 0.00 0.00
7-1 4298 0/0/0 W 0.00 1453710297 0 0.0 0.00 0.00
7-1 4298 0/0/0 W 0.00 1453710297 0 0.0 0.00 0.00
8-1 4539 0/0/0 W 0.00 1453710297 0 0.0 0.00 0.00
8-1 4539 0/0/0 W 0.00 1453710297 0 0.0 0.00 0.00
8-1 4539 0/0/0 W 0.00 1453710297 0 0.0 0.00 0.00
9-1 21393 0/0/0 W 0.00 1453710297 0 0.0 0.00 0.00
9-1 21393 0/0/0 W 0.00 1453710297 0 0.0 0.00 0.00
10-1 25117 0/0/0 W 0.00 1453710297 0 0.0 0.00 0.00
No indication what the request was in the last column.
loaded modules are:
apache2ctl -M
Loaded Modules:
core_module (static)
so_module (static)
watchdog_module (static)
http_module (static)
log_config_module (static)
logio_module (static)
version_module (static)
unixd_module (static)
access_compat_module (shared)
alias_module (shared)
auth_basic_module (shared)
authn_core_module (shared)
authn_file_module (shared)
authz_core_module (shared)
authz_groupfile_module (shared)
authz_host_module (shared)
authz_user_module (shared)
deflate_module (shared)
dir_module (shared)
env_module (shared)
evasive20_module (shared)
expires_module (shared)
filter_module (shared)
headers_module (shared)
http2_module (shared)
mime_module (shared)
mpm_worker_module (shared)
negotiation_module (shared)
pagespeed_module (shared)
proxy_module (shared)
proxy_fcgi_module (shared)
reqtimeout_module (shared)
rewrite_module (shared)
setenvif_module (shared)
socache_shmcb_module (shared)
ssl_module (shared)
status_module (shared)
I've searched the web for days now and can't seem to find any solution. Help is greatly appreciated. If you need more info, details, logs, please let me know and point me to it.
Best Answer
There were a few issues like this and upgrading to the latest version of mod_h2 seems to fix it.
For example: https://github.com/icing/mod_h2/issues/72
I would also add that the mod_h2 developer (icing) is very responsive to issues like this raised on that Git project so if upgrading to latest version doesn't fix it then raise an issue there. As this module is still experimental that is the best place to get support for now.