Confusing result on Apache server status results

apache-2.2

I have a CentOS 5.5 Final with Apache 2.2.3 and checking if there are some misconfigurations I settled up server-status page and querying a certain page I obtained this result:

Apache Server Status for 192.168.3.23

Server Version: Apache/2.2.3 (CentOS)
Server Built: Aug 30 2010 12:32:08

Current Time: Tuesday, 07-Aug-2012 16:27:17 CEST
Restart Time: Tuesday, 07-Aug-2012 15:35:02 CEST
Parent Server Generation: 0
Server uptime: 52 minutes 14 seconds
Total accesses: 6263 - Total Traffic: 12.2 MB
CPU Usage: u1.85 s.77 cu0 cs0 - .0836% CPU load
2 requests/sec - 4083 B/second - 2043 B/request
1 requests currently being processed, 4 idle workers

_..__W_.........................................................
................................................................
................................................................
................................................................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

Srv PID     Acc         M   CPU     SS    Req   Conn    Child   Slot    Client          VHost       Request
0-0 29399   0/14/875    _   0.03    1     27    0.0     0.03    1.58    X.X.X.X         127.0.0.1   GET /messaggi_xml.php?id_srv=2381&id_dev=1233&last_event=1&i
1-0 -       0/0/908     .   0.40    43    25    0.0     0.00    1.81    X.X.X.X         127.0.0.1   GET /messaggi_xml.php?id_srv=2381&id_dev=1233&last_event=1&i
2-0 -       0/0/859     .   0.45    3     23    0.0     0.00    1.67    X.X.X.X         127.0.0.1   GET /messaggi_xml.php?id_srv=2381&id_dev=1233&last_event=1&i
3-0 29379   0/160/857   _   0.34    0     25    0.0     0.30    1.68    X.X.X.X         127.0.0.1   GET /messaggi_xml.php?id_srv=2381&id_dev=1233&last_event=1&i
4-0 29380   0/132/804   _   0.31    1     15    0.0     0.27    1.61    X.X.X.X         127.0.0.1   GET /check_realtime.php?id_srv=2381&id_dev=1233 HTTP/1.1
5-0 29383   0/116/851   W   0.24    0     0     0.0     0.27    1.68    X.X.X.X         127.0.0.1   GET /server-status HTTP/1.1
6-0 29385   0/114/751   _   0.23    0     16    0.0     0.17    1.32    X.X.X.X         127.0.0.1   GET /check_realtime.php?id_srv=2381&id_dev=1233 HTTP/1.1
7-0 -       0/0/178     .   0.26    2410  22    0.0     0.00    0.48    X.X.X.X         127.0.0.1   GET /messaggi_xml.php?id_srv=2381&id_dev=1233&last_event=1&i
8-0 -       0/0/180     .   0.36    2362  12    0.0     0.00    0.38    X.X.X.X         127.0.0.1   GET /check_realtime.php?id_srv=2381&id_dev=1233 HTTP/1.1

Since I see an open slot with no current process associated with some pages, would that mean there is misconfiguration in Apache and do you have an hint to check a possible parameter to avoid that? If you need more infos, like configurations, I can provide…
Plus I have to say I don't see any errors in /var/log/messages or /var/log/httpd/error_log or /var/log/httpd/ssl_error_log and neither in /tmp/php_error_log… plus if I do ps aux I don't see those open slot listed…

Cheers,
Luigi

Best Answer

The open slots with no current processes are just display place holders for future threads. On lightly used systems, you won't see all of the slots being used by threads. Compare your output with this user's scoreboard. Almost all of their slots are in a waiting for connection state.

Try using one of these HTTP Benchmark tools and watch how your scoreboard changes.

Related Solutions

Centos – Tomcat with virtual hosts – 404

It smells like maybe it's running into a mod_rewrite or alias before it's getting to your virtual host definition. Everything you've posted looks correct, but you can try adding a few extra options:

JkWorkersFile  /explicit/path/to/workers.properties
JkLogFile      /var/log/mod_jk.log
JkLogLevel     debug

That first setting will make sure you're using the actual properties file you think you are, maybe something in your multi-tomcat environment is loading the wrong one. You might also check this specific instance's TOMCAT_HOME to ensure it's loading the right server.xml; you can actually pass that as well as a parameter to $TOMCAT_HOME with -config:

cd $TOMCAT_HOME
./bin/startup.sh -config /direct/path/to/server.xml

See where that takes you...

Linux – Apache server keeps crashing regularly

I'm not saying this is what's happening but based on my own experience as a CentOS admin, it's most likely runaway apache/php processes taking down the server. I've seen this numerous times on CentOS 5. It's frustrating because there's usually not a trace of what happened in the log files. The machine just grinds to a halt due to physical memory and swap being sucked up by apache/php processes. You would think linux memory management or some daemon would jump in and say "hey stop" but it doesn't. It'll let apache grind your system to a halt.

Having said that, to see what's happening you'll need something that can monitor and log resource usage. I like to use a program called atop. Atop is a lot like the top program but it also takes a snapshot of resource usage at defined intervals. It's pretty simple to install.

wget http://www.atcomputing.nl/Tools/atop/packages/atop-1.23.tar.gz 
tar -zxvf atop-1.23.tar.gz
cd atop-1.23 && make install

Open /etc/atop/atop.daily with a text editor and change INTERVAL=600 to INTERVAL=60

Run the command /etc/atop/atop.daily from a command prompt to start it. Wait a few minutes and run atop -r /var/log/atop/atop_20091118 with the correct date of course.

Hit the t key to go forward in time and T to go back. Next time your server crashes do this and check the MEM free and SWP free lines. If you have memory problems these will be in red. Also look for numerous httpd lines under CMD. If apache/php is your problem there'll be a bunch of them.

If this is the case, I recommend looking at you're MaxClients setting in httpd.conf. If set too high, apache will gladly eat all of your memory causing your machine to crash. Apache/php can easily eat 40-50MB/process. If you multiply 40mb x MaxClients you'll get a rough idea of how much memory apache can potentially use. MaxClients usually defaults to 150 on CentOS so apache can potentially use 6GB of memory by default. This doesn't include memory your system needs for itself and other processes to run. Try setting it to a more realistic value based on the amount of memory you have like 40 if you have 2G of memory and see if that helps. Also if you have KeepAlive On, set KeepAliveTimeout to a low number like 2 or 3.

In my opinion CentOS's apache/php compilation is a real pos that should never have seen the light of day. It's buggy and crash prone. If you run a serious site, I highly recommend compiling your own version of apache/php or even using one of the newer high performance webservers like lighttpd or nginx with fgci php.

Best Answer

Related Solutions

Centos – Tomcat with virtual hosts – 404

Linux – Apache server keeps crashing regularly

Related Topic