Ubuntu – 100% apache cpu usage after upgrade

apache-2.2Ubuntu

Greetings,

I run LAMP on an Ubuntu 8.04 server. I recently completed a "sudo apt-get upgrade" and upgraded all components found.

After this, my Apache frequently uses way too much cpu which in turn causes the website its running to become inaccessible. Before the upgrade it never exceeded around 2-3%.

Doing a "sudo /etc/init.d/apache2 restart" does not restart Apache, it has to be killed using "killall apache".

I am running

PHP 5.2.4-2ubuntu5.7 with Suhosin-Patch 0.9.6.2 (cli) (built: Aug 21 2009 19:52:39)
Copyright (c) 1997-2007 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2007 Zend Technologies

Server version: Apache/2.2.8 (Ubuntu)
Server built:   Aug 18 2009 14:18:10

Here is a screenshot taken of "htop": http://bayimg.com/image/hadoeaack.jpg

And here is one of "munin": bayimg.com/image/hadobaack.jpg

I would be eternally grateful for any help as I am loosing users by the minute! Thank you!

Edit: Maybe there is some correlation here:

1144 [Wed Sep 02 03:29:11 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting
1145 [Wed Sep 02 10:07:48 2009] [notice] caught SIGTERM, shutting down
1146 [Wed Sep 02 10:08:22 2009] [notice] Apache/2.2.8 (Ubuntu) mod_ssl/2.2.8 OpenSSL/0.9.8g configured -- resuming normal operations

-------------------

1162 [Wed Sep 02 16:10:14 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting
1163 [Wed Sep 02 19:03:53 2009] [notice] caught SIGWINCH, shutting down gracefully
1164 [Wed Sep 02 19:04:13 2009] [notice] Apache/2.2.8 (Ubuntu) mod_ssl/2.2.8 OpenSSL/0.9.8g configured -- resuming normal operations
1165 [Wed Sep 02 19:05:52 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting

But reaching MaxClients should not cause Apache to crash…? Surely?

Best Answer

I ended up solving it this way:

sudo dpkg --get-selections | grep apache
sudo dpkg --get-selections | grep php

to find all of the apache/php packages, and then

sudo apt-get remove --purge apache2 apache2-mpm-worker apache2-threaded-dev apache2-utils apache2.2-common libapache2-mod-python libapache2-mod-python-doc libapache2-mod-wsgi

to completely remove them.

Then reinstalled the same packages using

sudo apt-get install apache2 apache2-mpm-worker apache2-threaded-dev apache2-utils apache2.2-common libapache2-mod-python libapache2-mod-python-doc libapache2-mod-wsgi

This gives you a completely fresh install of Apache and PHP.

Thanks to http://dancingpenguinsoflight.com/2009/02/how-to-completely-reset-an-apache-instance-in-ubuntu/

Related Solutions

Linux – Apache server keeps crashing regularly

I'm not saying this is what's happening but based on my own experience as a CentOS admin, it's most likely runaway apache/php processes taking down the server. I've seen this numerous times on CentOS 5. It's frustrating because there's usually not a trace of what happened in the log files. The machine just grinds to a halt due to physical memory and swap being sucked up by apache/php processes. You would think linux memory management or some daemon would jump in and say "hey stop" but it doesn't. It'll let apache grind your system to a halt.

Having said that, to see what's happening you'll need something that can monitor and log resource usage. I like to use a program called atop. Atop is a lot like the top program but it also takes a snapshot of resource usage at defined intervals. It's pretty simple to install.

wget http://www.atcomputing.nl/Tools/atop/packages/atop-1.23.tar.gz 
tar -zxvf atop-1.23.tar.gz
cd atop-1.23 && make install

Open /etc/atop/atop.daily with a text editor and change INTERVAL=600 to INTERVAL=60

Run the command /etc/atop/atop.daily from a command prompt to start it. Wait a few minutes and run atop -r /var/log/atop/atop_20091118 with the correct date of course.

Hit the t key to go forward in time and T to go back. Next time your server crashes do this and check the MEM free and SWP free lines. If you have memory problems these will be in red. Also look for numerous httpd lines under CMD. If apache/php is your problem there'll be a bunch of them.

If this is the case, I recommend looking at you're MaxClients setting in httpd.conf. If set too high, apache will gladly eat all of your memory causing your machine to crash. Apache/php can easily eat 40-50MB/process. If you multiply 40mb x MaxClients you'll get a rough idea of how much memory apache can potentially use. MaxClients usually defaults to 150 on CentOS so apache can potentially use 6GB of memory by default. This doesn't include memory your system needs for itself and other processes to run. Try setting it to a more realistic value based on the amount of memory you have like 40 if you have 2G of memory and see if that helps. Also if you have KeepAlive On, set KeepAliveTimeout to a low number like 2 or 3.

In my opinion CentOS's apache/php compilation is a real pos that should never have seen the light of day. It's buggy and crash prone. If you run a serious site, I highly recommend compiling your own version of apache/php or even using one of the newer high performance webservers like lighttpd or nginx with fgci php.

100% CPU usage while uploading large files

It is natural that when you remove one bottleneck (disk IO), the system will speed up to the point of hitting another bottleneck (in this case, apparently, CPU). You should improve the efficiency of your code (so that it utilises a single core better), parallelise your code (so that it can utilise multiple cores simultaneously), obtain better CPUs, or learn to live with it. Is your site running slower than is acceptable for your situation?

Best Answer

Related Solutions

Linux – Apache server keeps crashing regularly

100% CPU usage while uploading large files

Related Topic