Linux: Identify process causing high *system* cpu usage

Ouch.. that's quite a task.

Software like Munin, cacti or zabbix would probably be most suited and would easily cover the 1st and 3rd points. I suspect Munin is possibly the easiest of the three for those points. The 2nd and 4th are slightly trickier. I'm not sure of any monitoring system that might provide you with total CPU time for a period. Usually system metric collection is focussed around the idea of "Whats going wrong" rather than system accounting. Hopefully someone else might have a suggestion.

As a starting point for point 4, you need to enable the process accounting features, something that's been integral to Linux for a while. There is some good detail here: http://shibuvarkala.blogspot.com/2009/04/howto-enable-process-accounting-in.html that will get you started with enabling process accounting and reading information it returns.

One reason for suggesting Munin is that it's relatively trivial to write plugins for, and I suspect you'll need to for what you're trying to achieve.

Wish I could be more help.

iotop (link) for starter ;) I haven't seen you posting an output of it.

1: I have experienced almost the same situation with a logging filesystem and atime - however with more writes.

Try to remount with noatime and turn off filesystem logging (later for testing only) in order to see if it's filesystem based and as said, iotop if it's process based.

2: I guess this partition is not part of a just-rebuilding raid array, is it?

3: If you are having a lot of very small files (a lot smaller than the actual block device blocksize and/or the filesystem blocksize), and you are reading those small files, you end up reading entire blocks from the system, and most of those blocks will be read for nothing.

4: If nothing helps above, you can always get the list of files accessed by executing

echo 1 > /proc/sys/vm/block_dump

please note that it degrades system performance by a lot. Instructions are available in my previous post here