It's the process for managing multi-CPU threading. See SMP Affinity and Proper Interrupt Handling in Linux for the answers you seek on how to lower it down, but basically its the way the system handles masking the interrupts the are being caused by the CPU's needing to be fed more thread streams or memory and only CPU0 being able to take on the interrupts.
The higher the value, the better the threading of the system is happening, and this is a good thing.
Did you pull the Munin graphs before or after you rebooted the system? If after, the part after the blank section is likely AFTER you rebooted, and is irrelevant. I would guess it's after, because your swap use has dramatically dropped...
In your question you are ignoring the blank section... You say "the graph doesn't show memory usage going up", but what they really show is no data during the time when memory was likely going up. munin is a great tool, but it is terrible at reporting instances like this, because it only reports information every 5 minutes and if the system is busy it may not report anything at all.
Have you done the memory math for the number of instances of Apache you can run? By this I mean do "ps awwlx --sort=rss | grep apache" and look at how much memory each Apache instance is using. For example:
root@theobromine:~# ps awwlx --sort=rss | grep apache
0 0 18497 18485 20 0 1788 528 - S+ pts/0 0:00 grep apache
5 33 18458 5384 20 0 28468 6700 - S ? 0:00 /usr/sbin/apache2 -k start
5 33 18470 5384 20 0 28468 6700 - S ? 0:00 /usr/sbin/apache2 -k start
5 33 18480 5384 20 0 28468 6700 - S ? 0:00 /usr/sbin/apache2 -k start
5 33 18481 5384 20 0 28468 6700 - S ? 0:00 /usr/sbin/apache2 -k start
5 33 18457 5384 20 0 28468 6708 - S ? 0:00 /usr/sbin/apache2 -k start
5 0 5384 1 20 0 28336 11796 - Ss ? 0:16 /usr/sbin/apache2 -k start
It is that 8th column we're looking at. In this case it is using 6.7MB for each instance, which is actually fairly small. But now I look at how much memory I have:
root@theobromine:~# free
total used free shared buffers cached
Mem: 775196 643848 131348 0 77964 268788
-/+ buffers/cache: 297096 478100
Swap: 1148636 3368 1145268
So I have 800MB of RAM... Now, I can do the math and say that in the best case I can run 800/6.7 = 119 instances of Apache. But that doesn't leave any space for any other applications or the OS or cache, etc...
But actually you have 478MB (second column under "free") at most, minus the amount of currently running Apaches (6.7*6 -- I only had 6 Apache instances running above), leaving around 520MB of RAM (if leaving you with no cache, of course). So the max I can really run is more like 77 instances.
So how many am I actually running?
root@theobromine:~# grep MaxClients /etc/apache2/apache2.conf
# MaxClients: maximum number of server processes allowed to start
MaxClients 150
# MaxClients: maximum number of simultaneous client connections
MaxClients 150
Ah, Apache isn't limiting me to less memory than I have. So, if more than 77 clients connect to my web server at once, I'm likely to start thrashing.
I see this quite frequently: "I need to be able to handle 500 simultaneous web connections." But then you look at their Apache instances and they are using 60MB (not an uncommonly large size), but then they freak out when you say they need to upgrade their VPS to 32BG of RAM. :-)
Best Answer
this can happen because of two things:
1. Overcommitted CPUs or RAM
2. Time drifts
You should make sure you are not overloading the host, not assigning more CPU cores or RAM tha n the host can provide, and also set up ntp, and possibly implement some kernel boot options specific to your virtualization platform