So I'm sure this has probably been asked before, but I want to make sure that I'm getting the right info.
I'm running a 360 Linode box with Debian 5 and Apache 2.2. I've compiled everything myself (no apt-get). Every now and then (once every few weeks?), my server would randomly "crash": it would shoot up to 100% CPU (really 400%, but you know what I mean) and hand the box. You couldn't SSH in to see what was the problem and the server itself stopped accepting connections. The only way to fix it was to reboot the box.
Recently, it's started happening with relative frequency: 24 hours, 12 hours, 10 hours, 8, 6, 4. Finally, I managed to catch a glimpse right before it locked up about two days ago. I noticed that the disk IO was elevated and that there was barely any RAM left! Also, there was a boat load of httpd processes running at 3-4% RAM. And by boat load, I mean when I did a ps -ef, they took up the entire screen. If you scrolled up, they took up the entire buffer for my SSH client.
So I made some tweaks to my code thinking that something wasn't closing properly. I fixed memory issues in my PHP, I turned on more granular error logging and fixed a bunch of errors, and it seemed to help to some degree. The crashes went back to about every 24 hours.
I'm convinced that this is caused because there is too little memory, and the volume of hits I receive is kicking my server into the swap. Since there are so many requests hitting the swap, the disk IO shoots through the roof, causing my CPU usage to shoot through the roof, causing my server to lock up.
Here's what I did to try to fix it: I did some research and found that I should probably be using prefork. I looked around in my config and couldn't find any ServerLimit or MaxClients or anything of the like, so I added some "default" values in and my server refused to accept /any/ incoming connections. Effectively, the prefork values blocked all inbound http traffic by choking off the connection (maybe because my server isn't capable of handling prefork? idk).
The way I see it, what my server "used to do" was fine, except all of those apache process kept hanging up and leaking memory. Is there any way to set a timeout on Apache processes or put a limit on how many of them there are? It seems pretty dumb that the best solution is to use prefork; I have to imagine that there's a better way.
Thanks guys
Best Answer
It seems you are learning the hard way that PHP is a memory hog and not particularly scalable.
Some suggestions, in no particular order:
If you still suspect a memory leak set MaxRequestsPerChild to really low value.
Consider buying more memory, 360 megs is really not much nowadays.
Try finding an average size of httpd process by running ps or top and then setting MaxClients so that everything always fits in memory. Swapping is a death spiral, the slower you are processing the requests because of it, the more processes apache needs to fork, using even more memory.
If you laod php as a module in apache it gets loaded for every request, be it a script or a static file (an image or css or .js or whatnot). Consider serving static content from a separate server or using fastcgi or a reverse proxy like nginx to have apache serve only php to limit the number of fat php instances you have to keep in memory.