Windows Load Average – How to Get Load Average in Windows

can it be that mysql is using temporary tables? can you add some munin charts to with io stat..? io figures in provided log seems unbelievably low.

what is your working set - does the data fits comfortably in the memory [seems so]? do you do from time to time plenty of writes to sql [from log it seems - none at all]?

is it possible that suddenly you have spike in number of concurrent requests [ sql or to your custom server ]? what cat /proc/net/ip_conntrack|wc -l says? what does it show during load spikes?

can you turn on mysql slow query logging - eg everything > 1 or 2 sec?

are your disks directly attached to the server or maybe it's iscsi / nfs? can you check health status of disks [smart] / raid status? maybe one of drives is failing... or maybe you can run simple io disk benchmark off-peak to make sure you get decent read/write speeds.

or maybe something ugly shows in dmesg?

edit: check if netstat |wc -l is correlated with load

check if ps axms|wc -l is correlated with load

check if lsof |wc -l is correlated with load

[ preferably hack small munin plugins for those to get them on the charts ].

You might want to enable Apache's mod_status ( http://httpd.apache.org/docs/2.0/mod/mod_status.html ) so you can see exactly what's happening inside your webserver. Specifically, you'll be get numbers on per-request CPU consumption.

A few snapshots from vmstat/iostat wouldn't hurt, either.

Also, are you using MyISAM or InnoDB tables? When you get one of these load spikes, what do you get from "SHOW FULL PROCESSLIST\G" in MySQL? I have a feeling you're getting lock/query contention in MySQL which is blowing up the length of your kernel run queue.