How to trace high CPU Apache process to particular virtualhost

I appreciate that it doesn't always suit to have mod_status available and on all of the time, but it and apachetop are the best ways to diagnose these problems. However there are many ways to skin a cat.

This trick is useful in a number of circumstances and isn't just Apache specific. It does depend on a number of factors however, and you need to know what it's doing to know it's limitations.

for pid in `pgrep -u www-data`; do find /proc/${pid}/cwd -printf "%l\n" ; done

Let's break it down:

• pgrep -u www-data gives you the list of pids running under user www-data. That's the default on Debian / Ubuntu, change to suit your own system (RedHat based systems tend to use httpd, for example, as the user). For systems without pgrep, you can use ps axuwww | grep user | awk '{print $2}'
• the *for; do; ... done * loop means we loop over every entry running the command(s) within the do part of the loop.
• find /proc/${pid}/cwd -printf "%l\n" simply searches /proc for each of those PIDs and spits out the current working directory for that process. Apache will chdir() to the VirtualHost by default when serving files from that VirtualHost. /proc/PID/cwd is a symbolic link to the directory that apache process is running in. the printf "%l\n" prints the endpoint to that link. See find(1) for more info on that.

There are two major caveats to that trick:

1) If something running under the same context as the Apache process does a chdir()'s outside of the VirtualHost directory, you'd be hard pushed to find that out.

e.g. a PHP script running under mod_php (a CGI will be different as Apache fork's a separate process, but I'm presuming CGI's aren't a problem or you'd be able to track them easier).

2) If you have Apache instances which are very very quickly serving pages (e.g. a small static HTML page). This normally isn't a problem, but it may be possible. If you're getting a lot of "No such file or directory" errors, this is basically a manifestation of it. I would expect some, but not the majority unless they fit this particular case. Basically this is because the Apache processes you've scanned with ps have already exited by the time you've checked /proc. Obviously this means they are serving pages very very quickly.

Regarding memory bound Apache processes, I use ps_mem.py to calculate memory usage on my webservers. If you've got large Apache (in terms of resident memory size) processes and they are exiting quickly, that is roughly the equivalent of asking a big fat guy to keep running 100m sprints. If your webserver isn't a shared one, those "No such file or directory" errors are normally good candidates to move some content onto a smaller lightweight webserver (e.g. nginx / lighttpd) or start heavily caching content (e.g. varnish / squid).

The CPU usage from Apache's server-status page is the average usage since Apache was started so it won't show spikes like this. When you get these load spikes you can check the server-status page to see what pages/clients are being server (ExtendedStatus must be on).

You can also use netstat to see what clients are currently accessing your machine:

 netstat -an | grep ESTABLISHED

If you run this over multiple hours and traffic spikes you may be able to spot a reoccuring IP address and potentially trace to a specific robot/crawler. If this does turn out to be the case you can look into using robots.txt to limit how well behaving robots should crawl your site.

Edit: On a busy server the above netstat command should show some entries like:

tcp        0      0 10.2.212.13:80              216.146.52.21:24979         ESTABLISHED
tcp        0      0 10.2.212.13:80              86.174.113.138:54901        ESTABLISHED
tcp        0      0 10.2.212.13:80              94.1.216.253:51204          ESTABLISHED
tcp        0      0 10.2.212.13:80              24.9.61.204:62936           ESTABLISHED

The client's IP address will be the one on the right. If you only see 1 or 2 lines it just means that at that moment there is just your ssh connection. Check again when your load increases. You can also remove the grep to list all connections although this will include a large number of old TIME_WAIT.

I would start with the extended server-status and see if that can reveal any obvious crawlers during traffic peaks.