Apache – Troubleshooting Sudden Slow Performance

apache-2.2apache-2.4lampUbuntuubuntu-12.04

Apache on my webserver is suddenly very slow.
I have a 8 GB Ram, 4 CPU server at digitalocean with LAMP installed.
What can I do to fix this?

This is the top output:

top - 10:37:15 up 93 days, 20:20,  1 user,  load average: 0.22, 0.22, 0.28
Tasks: 471 total,   3 running, 468 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.4 us,  0.6 sy,  0.0 ni, 97.4 id,  0.1 wa,  0.0 hi,  0.5 si,  0.0 st
KiB Mem:   8176832 total,  6816036 used,  1360796 free,   184160 buffers
KiB Swap:        0 total,        0 used,        0 free.  5057196 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                
 6463 mysql     20   0 2420696  88740   3808 S   0.7  1.1 348:23.30 mysqld                                                                                                 
16207 www-data  20   0  385076  11560   4376 S   0.7  0.1   0:00.18 apache2                                                                                                
    7 root      20   0       0      0      0 S   0.3  0.0 818:55.76 rcu_sched                                                                                              
    8 root      20   0       0      0      0 S   0.3  0.0 464:04.83 rcuos/0                                                                                                
 1053 root      20   0  100708   2624    708 S   0.3  0.0   9:19.16 sendmail-mta                                                                                           
16168 www-data  20   0  384812  11420   4424 S   0.3  0.1   0:00.17 apache2                                                                                                
16194 www-data  20   0  385076  11620   4384 S   0.3  0.1   0:00.21 apache2                                                                                                
16202 www-data  20   0  384820  11432   4428 S   0.3  0.1   0:00.22 apache2                                                                                                
16276 www-data  20   0  385076  11588   4440 S   0.3  0.1   0:00.15 apache2                                                                                                
16287 www-data  20   0  386696  13136   4436 S   0.3  0.2   0:00.18 apache2                                                                                                
16309 www-data  20   0  385068  11592   4404 S   0.3  0.1   0:00.17 apache2                                                                                                
16333 www-data  20   0  384812  11392   4396 S   0.3  0.1   0:00.14 apache2                                                                                                
16340 www-data  20   0  386348  12608   4416 R   0.3  0.2   0:00.11 apache2                                                                                                
16353 www-data  20   0  386696  13036   4404 S   0.3  0.2   0:00.22 apache2                                                                                                
16385 www-data  20   0  385076  11640   4404 S   0.3  0.1   0:00.21 apache2                                                                                                
16412 www-data  20   0  386696  13116   4436 S   0.3  0.2   0:00.40 apache2                                                                                                
16446 www-data  20   0  386672  13080   4404 S   0.3  0.2   0:00.14 apache2                                                                                                
16454 www-data  20   0  386696  13100   4424 S   0.3  0.2   0:00.17 apache2                                                                                                
16479 www-data  20   0  386412  12852   4396 S   0.3  0.2   0:00.14 apache2                                                                                                
16540 root      20   0   25212   1988   1176 R   0.3  0.0   0:00.33 top                                                                                                    
    1 root      20   0   33500   2688   1328 S   0.0  0.0   0:44.63 init                                                                                                   
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.15 kthreadd                                                                                               
    3 root      20   0       0      0      0 S   0.0  0.0 202:28.30 ksoftirqd/0                                                                                            
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H                                                                                           
    6 root      20   0       0      0      0 S   0.0  0.0 157:44.10 kworker/u8:0                                                                                           
    9 root      20   0       0      0      0 R   0.0  0.0 164:29.67 rcuos/1                                                                                                
   10 root      20   0       0      0      0 S   0.0  0.0 152:37.93 rcuos/2                                                                                                
   11 root      20   0       0      0      0 S   0.0  0.0  68:24.97 rcuos/3                                                                                                
   12 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh                                                                                                 
   13 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/0                                                                                                
   14 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/1                                                                                                
   15 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/2

This is the apache error log:

[Tue Dec 29 10:32:11.628945 2015] [:error] [pid 16404] [client 61.178.71.8:37626] PHP Notice:  Undefined variable: config_location in /var/www/html/a/lib/common.php on line 32, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:11.727455 2015] [:error] [pid 16447] [client 87.168.254.145:63259] PHP Notice:  Undefined variable: countryname in /var/www/html/c/arth.php(33) : eval()'d code on line 1, referer: http://eywow.com/c/arth.php?slug=balls
[Tue Dec 29 10:32:11.727566 2015] [:error] [pid 16447] [client 87.168.254.145:63259] PHP Notice:  Undefined variable: city in /var/www/html/c/arth.php(33) : eval()'d code on line 1, referer: http://eywow.com/c/arth.php?slug=balls
[Tue Dec 29 10:32:11.728356 2015] [:error] [pid 16447] [client 87.168.254.145:63259] PHP Notice:  Undefined offset: -1 in /var/www/html/c/arth.php on line 323, referer: http://eywow.com/c/arth.php?slug=balls
[Tue Dec 29 10:32:12.077442 2015] [:error] [pid 16326] [client 138.0.68.53:2349] PHP Notice:  A session had already been started - ignoring session_start() in /var/www/html/a/lib/common.php on line 3, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:12.077514 2015] [:error] [pid 16326] [client 138.0.68.53:2349] PHP Notice:  Undefined index: domain in /var/www/html/a/lib/common.php on line 14, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:12.077950 2015] [:error] [pid 16326] [client 138.0.68.53:2349] PHP Notice:  Undefined variable: config_location in /var/www/html/a/lib/common.php on line 32, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:12.213301 2015] [:error] [pid 16220] [client 187.111.210.17:4461] PHP Notice:  A session had already been started - ignoring session_start() in /var/www/html/a/lib/common.php on line 3, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:12.213353 2015] [:error] [pid 16220] [client 187.111.210.17:4461] PHP Notice:  Undefined index: domain in /var/www/html/a/lib/common.php on line 14, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:12.213812 2015] [:error] [pid 16220] [client 187.111.210.17:4461] PHP Notice:  Undefined variable: config_location in /var/www/html/a/lib/common.php on line 32, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:12.902652 2015] [:error] [pid 16256] [client 162.156.92.161:57060] PHP Notice:  Undefined variable: countryname in /var/www/html/c/arth.php(33) : eval()'d code on line 1, referer: http://eywow.com/c/arth.php?slug=ufo
[Tue Dec 29 10:32:12.902836 2015] [:error] [pid 16256] [client 162.156.92.161:57060] PHP Notice:  Undefined variable: city in /var/www/html/c/arth.php(33) : eval()'d code on line 1, referer: http://eywow.com/c/arth.php?slug=ufo
[Tue Dec 29 10:32:13.002648 2015] [:error] [pid 16427] [client 41.190.198.114:50583] PHP Notice:  A session had already been started - ignoring session_start() in /var/www/html/a/lib/common.php on line 3, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:13.002704 2015] [:error] [pid 16427] [client 41.190.198.114:50583] PHP Notice:  Undefined index: domain in /var/www/html/a/lib/common.php on line 14, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:13.003456 2015] [:error] [pid 16427] [client 41.190.198.114:50583] PHP Notice:  Undefined variable: config_location in /var/www/html/a/lib/common.php on line 32, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:13.136681 2015] [:error] [pid 16256] [client 119.93.37.113:38315] PHP Notice:  A session had already been started - ignoring session_start() in /var/www/html/a/lib/common.php on line 3, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:13.136740 2015] [:error] [pid 16256] [client 119.93.37.113:38315] PHP Notice:  Undefined index: domain in /var/www/html/a/lib/common.php on line 14, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:13.137251 2015] [:error] [pid 16256] [client 119.93.37.113:38315] PHP Notice:  Undefined variable: config_location in /var/www/html/a/lib/common.php on line 32, referer: http://hi-techno.xyz/clicknety.html

My site example page is very slow, this is the server status page of apache My guess is that the iptables are slowing it down, as there is no reason from the hardware side, there are enough free resources. I have 8 GB Ram and 4 CPUs.

Best Answer

63.2 requests/sec - 1.4 MB/second - 23.4 kB/request
375 requests currently being processed, 0 idle workers

You've maxed out your available Apache processes. You need to increase maxclients. Check your Apache error logs for more verbose information about the problem.

Related Solutions

Linux – Apache server keeps crashing regularly

I'm not saying this is what's happening but based on my own experience as a CentOS admin, it's most likely runaway apache/php processes taking down the server. I've seen this numerous times on CentOS 5. It's frustrating because there's usually not a trace of what happened in the log files. The machine just grinds to a halt due to physical memory and swap being sucked up by apache/php processes. You would think linux memory management or some daemon would jump in and say "hey stop" but it doesn't. It'll let apache grind your system to a halt.

Having said that, to see what's happening you'll need something that can monitor and log resource usage. I like to use a program called atop. Atop is a lot like the top program but it also takes a snapshot of resource usage at defined intervals. It's pretty simple to install.

wget http://www.atcomputing.nl/Tools/atop/packages/atop-1.23.tar.gz 
tar -zxvf atop-1.23.tar.gz
cd atop-1.23 && make install

Open /etc/atop/atop.daily with a text editor and change INTERVAL=600 to INTERVAL=60

Run the command /etc/atop/atop.daily from a command prompt to start it. Wait a few minutes and run atop -r /var/log/atop/atop_20091118 with the correct date of course.

Hit the t key to go forward in time and T to go back. Next time your server crashes do this and check the MEM free and SWP free lines. If you have memory problems these will be in red. Also look for numerous httpd lines under CMD. If apache/php is your problem there'll be a bunch of them.

If this is the case, I recommend looking at you're MaxClients setting in httpd.conf. If set too high, apache will gladly eat all of your memory causing your machine to crash. Apache/php can easily eat 40-50MB/process. If you multiply 40mb x MaxClients you'll get a rough idea of how much memory apache can potentially use. MaxClients usually defaults to 150 on CentOS so apache can potentially use 6GB of memory by default. This doesn't include memory your system needs for itself and other processes to run. Try setting it to a more realistic value based on the amount of memory you have like 40 if you have 2G of memory and see if that helps. Also if you have KeepAlive On, set KeepAliveTimeout to a low number like 2 or 3.

In my opinion CentOS's apache/php compilation is a real pos that should never have seen the light of day. It's buggy and crash prone. If you run a serious site, I highly recommend compiling your own version of apache/php or even using one of the newer high performance webservers like lighttpd or nginx with fgci php.

Mod_fcgid, perl script output going to apache error_log

Your configuration is telling Apache to serve all CGI scripts under FastCGI, which isn't possible. FastCGI isn't directly compatible with CGI scripts.

You should move the AddHandler directive into your client's VirtualHost, so it doesn't apply to the whole configuration, e.g.

<VirtualHost *:80>
    ServerName clienthost.com
    ...
    AddHandler fcgid-script .fcgi .cgi
</VirtualHost>

If your client's scripts are on the same VirtualHost as your other cgi scripts, you can restrict the FastCGI handling to a specific Location or Directory, e.g.

<Location /clientarea>
    AddHandler fcgid-script .fcgi .cgi
</Location>

Best Answer

Related Solutions

Linux – Apache server keeps crashing regularly

Mod_fcgid, perl script output going to apache error_log

Related Topic