Debian – Which tool provides per-process (or per-cgroup) resource (RAM/CPU/Disk) monitoring

can it be that mysql is using temporary tables? can you add some munin charts to with io stat..? io figures in provided log seems unbelievably low.

what is your working set - does the data fits comfortably in the memory [seems so]? do you do from time to time plenty of writes to sql [from log it seems - none at all]?

is it possible that suddenly you have spike in number of concurrent requests [ sql or to your custom server ]? what cat /proc/net/ip_conntrack|wc -l says? what does it show during load spikes?

can you turn on mysql slow query logging - eg everything > 1 or 2 sec?

are your disks directly attached to the server or maybe it's iscsi / nfs? can you check health status of disks [smart] / raid status? maybe one of drives is failing... or maybe you can run simple io disk benchmark off-peak to make sure you get decent read/write speeds.

or maybe something ugly shows in dmesg?

edit: check if netstat |wc -l is correlated with load

check if ps axms|wc -l is correlated with load

check if lsof |wc -l is correlated with load

[ preferably hack small munin plugins for those to get them on the charts ].

It depends what the server is doing really. For example, I know my Exchange 2007 servers will use a lot of memory, that’s what Exchange does, it grabs as much as it can, so monitoring this server for High Ram use would keep me awake all night, however I want to know if my disk space is getting low on here, as Exchange is prone to stop working with low disk space. On the other hand, I'm not really that concerned about the disk usage on my print server.

Really you need to look at your servers and determine what you need to know about them, what’s important to them running correctly, what’s nice to know for historical or tracking purposes, and what is superfluous. When you've determined what’s critical, then you really should have alarms or triggers setup for these events, what’s the point in monitoring something if you don't know when it goes wrong?