I'm not normally a sysadmin, but I've got a production server under heavy load (serving some basic php pages, and some php redirect files that have some sql queries, and no images) that keeps crashing. Specifically, the load gets up to about 20 and requests time out. There's nothing in the apache access log or error log indicating unusual activity but the disk IO chart shows heavy read/write spikes that correlate with our downtime.
I know it's some combination of these pages and a few hundred thousand hits an hour, but I'm stumped, and I don't know which tools to use. I need to see A) How many hits per second/minute/hour these pages are getting and B) How long it's taking to serve each page. What's available to profile a live server under load? What's best?
The server is apache2, php5, ubuntu hardy. Any advice at all is greatly appreciated.
EDIT:
Thanks for the ideas. I could edit the PHP, but these are pages that designers are changing often, they like to copy/paste/delete things, and I was hoping to find something better than ducttape for this because it's a recurring issue on a lot of our servers.
Are there really no software packages for monitoring server load per-file on production servers? Do I have to resort to debugging tools and per-code-segment profiling? If my server's already choking on hits, wouldn't adding XDebug royally F*!#-up my S@^&?
Best Answer
You can try Apache module modlogslow to get measures of the time period used for handling each request by the current process.