Ubuntu – Apache2 crashes at 7 am every day with “reached MaxClients”. Runs fine the rest of the day

If you're looking for a challenge… you've found it.

Every day at between 07:01:30 – 07:03:00 my apache installation crashes. Just at that time. It runs fine any other time of the day.
I've had this problem for about a year now, Googled on this for hours and tried a lot of things.
My guess is something is scheduled around this time that causes it. So far I have not been able to find it.

Now you're going to say, oh, look at this link you lazy person! Just up the maxclients setting!
Trust me, I have, but unfortunately, none of the solutions on this site or any other site have worked.
Now to add to this, to me, mystery, the server runs absolutely fine all day. Just at ~07:01 it crashes with below error.

The error I get is:
[Sun Jul 28 07:01:30 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Mon Jul 29 07:03:09 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Tue Jul 30 07:01:54 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting

Notice monday is slightly off. This is because, at 7:01:00 I did a service apache2 restart. It just made this crash come a little bit later. Huh?
Maybe a fluke… I don't understand this at all.

I've used apachetop to check traffic. Just before getting this error, there aren't many users on the site, perhaps 10-20.
During evenings, however, we can get hunderds of visitors at the same time. I've seen Apache handle up to 100 requests/sec!
The problem is not at all that the server reached MaxClients because 7am is a really quiet time for the site.
Access logs show nothing special. A few clicks here and there.

So in my eyes, it can't be a keepalive issue. This wouldn't cause a crash at nearly exactly the same time each day.
It can't be an attack, because why would this be at 7am every day for a year? Also, the logs don't show any kind of traffic really.

This is what I've tried and checked:
– Upping the maxclients setting (as high as 512. Yes, also ServerLimit). Making it much lower. Nothing works.
– Reduced keepalive time in Apache config.
– Moved all my resources (js, css, images) to another server with nginx using haproxy. It's not the amount of requests!
– Upgraded Ubuntu to 12.04.
– Tried every kind of tool to check my apache config
– Moved daily cronjob an hour earlier in /etc/crontab. Still ~7:00.
– Checked other users for crontabs. Nothing.
– mod_status is not showing an abnormal amount of connections.

There is a haproxy installation before Apache. But I have tried without this, and Apache still crashes.
Haproxy logs state that the server went down (I'm using httpcheck). I tried switching off this check, this just causes the site to not load at all (it times out).

Version info:
I'm running Ubuntu 12.04 but also had this problem running 11.10.
I'm using prefork.
Server version: Apache/2.2.22 (Ubuntu)
Server built: Jul 12 2013 13:37:10

No crontabs except for root:
for user in $(cut -f1 -d: /etc/passwd); do crontab -u $user -l; done1

Server has plenty of resources, with mostly gigabytes of RAM free.

Obviously I'm missing something here.
Where else can Ubuntu schedule stuff?
Is Apache doing something special around 7am that is not in the crontab?

Thanks!

~~~~ I can't answer my own question but it's solved! ~~~~

Reply:
Aaaaaah I found it. Janne Pikkarainen got it right the first time.
Apparently I was not using my eyes. Probably not the correct command with netstat. Because I just saw loads of connections… (sure, it's a busy time now, but that doens't explain 10 times more connections suddenly).

Without further ado, it was the database server. It's running a mysqldump every day at 7am. Because it's a different server it didn't even come up to me that I should check that one too.

I just tried to run that backup and soon enough requests start piling up. And bam! Apache throws error and stops responding. Stupid developer (me) is stupid 🙂
So the fix was never service apache2 restart, the backup just ended and the server responded again. So actually, it did not crash at all.

Thanks everyone for all the help. Not such a mystery… I knew I missed something somewhere.
I hope this answer helps someone, some day 🙂