Ubuntu – Slow Apache2 restart with SSL

We run apache2 server with multiple sites (each has own domain or subdomain) using SSL (also multiple certificates on same IP but mostly an asterisk certificate for subdomains). Now we are having a problem that when there are more than 20-30 individual sites per server, restart takes over 20 seconds.

Logs don't show anything and I am not sure how to figure out what takes so long to restart. This may be connected – running apache2ctl -S also takes roughly same amount of time (When I run one restart after another or one -S after another its fast <1s, if I wait a minute or so its again slow).

How can I trouble shoot this issue? How to determine what causes those slow restarts (we need to restart when we add new sites and its slowly becoming unmanageable).

— Update —

So, after all this time, this is still a problem.

There has been some changes that may point me in a right direction:

One of the servers all of the sudden started to work as fast as I would expect it too. Not sure why it happened as I cant pinpoint when exactly this happened. I compared apache configs and they are pretty much identical on all servers but one now restart fast and other two are still slow (>2min).

Now, just recently when troubleshooting something else I came across some comments about ipv6 setup slowing down certain certificate issuance etc. I have tested ipv6 connectivity and only discrepancy I found is that on a fast server when I try this:

wget -6 https://google.com/

I get this:

--2017-06-09 07:49:32-- https://google.com/
Resolving google.com (google.com)... 2a00:1450:4009:809::200e
Connecting to google.com (google.com)|2a00:1450:4009:809::200e|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://ipv6.google.com/sorry/index?continue=https://google.com/&q=EhAqAX4AAAAAAPA8kf_-iVh9GIym6ckFIhkA8aeDS9iurm4ysYfhKSPR5hfHhPY5mBEsMgNyY24 [following]
--2017-06-09 07:49:33-- https://ipv6.google.com/sorry/index?continue=https://google.com/&q=EhAqAX4AAAAAAPA8kf_-iVh9GIym6ckFIhkA8aeDS9iurm4ysYfhKSPR5hfHhPY5mBEsMgNyY24
Resolving ipv6.google.com (ipv6.google.com)... 2a00:1450:401b:802::200e
Connecting to ipv6.google.com (ipv6.google.com)|2a00:1450:401b:802::200e|:443... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2017-06-09 07:49:33 ERROR 503: Service Unavailable.

While on the slow servers same request produces this:

--2017-06-09 07:50:37--  https://google.com/
Resolving google.com (google.com)... 2404:6800:4003:802::200e
Connecting to google.com (google.com)|2404:6800:4003:802::200e|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.google.com/ [following]
--2017-06-09 07:50:37--  https://www.google.com/
Resolving www.google.com (www.google.com)... 2404:6800:4001:80b::2004
Connecting to www.google.com (www.google.com)|2404:6800:4001:80b::2004|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'index.html'
[ <=>    ] 10,382      --.-K/s   in 0.001s

2017-06-09 07:50:37 (18.4 MB/s) - 'index.html' saved [10382]

So in other words there appears to be some issue with ipv6 that possibly causes timeout when trying to reboot apache?

I also figured out the following – its a shutdown that takes all the time. If I shutdown apache and then try to start it starting is fast.

To answer previous questions:

Ciphers are optimised (all sites getting A on ssl test).
Traffic is not great. Here is the server-status dump before a 2 min restart:

Apache Server Status for s3.example.com (via 0.0.0.0)

Server Version: Apache/2.4.7 (Ubuntu) PHP/5.5.9-1ubuntu4.21 OpenSSL/1.0.1f
Server MPM: prefork
Server Built: May 9 2017 16:14:10

Current Time: Friday, 09-Jun-2017 08:05:46 UTC
Restart Time: Friday, 09-Jun-2017 07:57:35 UTC
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 8 minutes 10 seconds
Server load: 0.00 0.00 0.00
Total accesses: 15 - Total Traffic: 23 kB
CPU Usage: u0 s0 cu0 cs0
.0306 requests/sec - 48 B/second - 1570 B/request
1 requests currently being processed, 7 idle workers

____W___........................................................
................................................................
......................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

Srv PID Acc M   CPU     SS  Req Conn    Child   Slot    Client  VHost   Request
0-0 11047   0/2/2   _   0.00    456 0   0.0 0.00    0.00    77.0.0.0    s3.example.com:80   NULL
2-0 11049   0/3/3   _   0.00    219 0   0.0 0.00    0.00    127.0.0.1   s3.example.com:80   GET /server-status?auto HTTP/1.1
4-0 11051   0/7/7   W   0.00    0   0   0.0 0.01    0.01    77.0.0.0    s3.example.com:443  GET /server-status HTTP/1.1
6-0 11166   0/2/2   _   0.00    35  0   0.0 0.00    0.00    127.0.0.1   s3.example.com:443  \x16\x03\x01
7-0 11167   0/1/1   _   0.00    333 1   0.0 0.00    0.00    77.0.0.0    s3.example.com:443  NULL
Srv Child Server number - generation
PID OS process ID
Acc Number of accesses this connection / this child / this slot
M   Mode of operation
CPU CPU usage, number of seconds
SS  Seconds since beginning of most recent request
Req Milliseconds required to process most recent request
Conn    Kilobytes transferred this connection
Child   Megabytes transferred this child
Slot    Total megabytes transferred this slot
SSL/TLS Session Cache Status:
cache type: SHMCB, shared memory: 512000 bytes, current entries: 0
subcaches: 32, indexes per subcache: 88
index usage: 0%, cache usage: 0%
total entries stored since starting: 0
total entries replaced since starting: 0
total entries expired since starting: 0
total (pre-expiry) entries scrolled out of the cache: 0
total retrieves since starting: 0 hit, 1 miss
total removes since starting: 0 hit, 0 miss
Apache/2.4.7 (Ubuntu) Server at s3.example.com Port 443

Please advise…