Linux – Simple Option for Graphing CPU Usage on Remote Server

Is the load balancing completely round robin or is it doing stickiness based on IP or cookie? You could have some kind of user traffic that sticks to one server and moves upon restart - especially if another one of your servers is calling an app on the cluster. So cross check it against actual hits to the server.

You may also have a race condition in the app that certain operations get it in a loop. For that you could take a thread dump (kill -3 pid) and pull it out of your stdout log and run something like Samurai on it to see what up.

I would also turn on garbage collection logging and see if GC times correlate with perceived lag times.

Disk IOs per device (IOs/second)

With traditional hard drives this is a very important number. I/O operation is a read or write operation to disk. With rotational spindles you can get around from dozens to perhaps 200 IOPS per second, depending on the disk speed and its usage pattern.

This is not all to it: modern operating systems do have I/O schedulers which try to merge several I/O requests as one and make things faster that way. Also the RAID controllers and so on do perform some smart I/O request reordering.

Disk latency per device (Average IO wait)

How long it took from performing the I/O request to an individual disk to actually receive the data from there. If this hovers around couple of milliseconds, you are OK, if it's dozens of ms, then you are starting to see your disk subsystem sweating, if it's hundreds of more ms, you are in big trouble, or at least have a very, very slow system.

IO Service Time

How your disk subsystem (possibly containing lots of disks) is performing overall.

IOStat (blocks/second read/written)

How many disk blocks were read/written per second. Look for spikes and also the average. If average starts to near the maximum throughput of your disk subsystem, it's time to plan for performance upgrade. Actually, plan that way before that point.

Available entropy (bytes)

Some applications do want to get "true" random data. Kernel gathers that 'true' randomness from several sources, such as keyboard and mouse activity, a random number generator found in many motherboards, or even from video/music files (video-entropyd and audio-entropyd can do that).

If your system runs out of entropy, the applications wanting that data stall until they get their data. Personally in the past I've seen this happening with Cyrus IMAP daemon and its POP3 service; it generated a long random string before each login, and on a busy server that consumed the entropy pool very quickly.

One way to get rid of that problem is to switch the applications to use only semi-random data (/dev/urandom), but that's not among this topic anymore.

VMStat (running/I/O sleep processes)

Not thought about this one before, but I would think that this tells you about per-process I/O statistics, or mainly if they are running some I/O or not, and if that I/O is blocking I/O activity or not.

Disk throughput per device (bytes/second read/written)

This is purely bytes read/written per second, and more often this is more human-readable form than blocks, which may vary. Block size may differ because of the disks used, file system (and its settings) used, and so on. Sometimes the block size might be 512 bytes, other times 4096 bytes, sometimes something else.

inode table usage

With file systems having dynamic inodes (such as XFS), nothing. With file systems having static inodes maps (such as ext3), everything. If you have combination of static inodes, a huge file system and huge number of directories and small files, you might encounter a situation where you cannot create more files on that partition, even though in theory there would be lots of free space left. No free inodes == bad.