Did you pull the Munin graphs before or after you rebooted the system? If after, the part after the blank section is likely AFTER you rebooted, and is irrelevant. I would guess it's after, because your swap use has dramatically dropped...
In your question you are ignoring the blank section... You say "the graph doesn't show memory usage going up", but what they really show is no data during the time when memory was likely going up. munin is a great tool, but it is terrible at reporting instances like this, because it only reports information every 5 minutes and if the system is busy it may not report anything at all.
Have you done the memory math for the number of instances of Apache you can run? By this I mean do "ps awwlx --sort=rss | grep apache" and look at how much memory each Apache instance is using. For example:
root@theobromine:~# ps awwlx --sort=rss | grep apache
0 0 18497 18485 20 0 1788 528 - S+ pts/0 0:00 grep apache
5 33 18458 5384 20 0 28468 6700 - S ? 0:00 /usr/sbin/apache2 -k start
5 33 18470 5384 20 0 28468 6700 - S ? 0:00 /usr/sbin/apache2 -k start
5 33 18480 5384 20 0 28468 6700 - S ? 0:00 /usr/sbin/apache2 -k start
5 33 18481 5384 20 0 28468 6700 - S ? 0:00 /usr/sbin/apache2 -k start
5 33 18457 5384 20 0 28468 6708 - S ? 0:00 /usr/sbin/apache2 -k start
5 0 5384 1 20 0 28336 11796 - Ss ? 0:16 /usr/sbin/apache2 -k start
It is that 8th column we're looking at. In this case it is using 6.7MB for each instance, which is actually fairly small. But now I look at how much memory I have:
root@theobromine:~# free
total used free shared buffers cached
Mem: 775196 643848 131348 0 77964 268788
-/+ buffers/cache: 297096 478100
Swap: 1148636 3368 1145268
So I have 800MB of RAM... Now, I can do the math and say that in the best case I can run 800/6.7 = 119 instances of Apache. But that doesn't leave any space for any other applications or the OS or cache, etc...
But actually you have 478MB (second column under "free") at most, minus the amount of currently running Apaches (6.7*6 -- I only had 6 Apache instances running above), leaving around 520MB of RAM (if leaving you with no cache, of course). So the max I can really run is more like 77 instances.
So how many am I actually running?
root@theobromine:~# grep MaxClients /etc/apache2/apache2.conf
# MaxClients: maximum number of server processes allowed to start
MaxClients 150
# MaxClients: maximum number of simultaneous client connections
MaxClients 150
Ah, Apache isn't limiting me to less memory than I have. So, if more than 77 clients connect to my web server at once, I'm likely to start thrashing.
I see this quite frequently: "I need to be able to handle 500 simultaneous web connections." But then you look at their Apache instances and they are using 60MB (not an uncommonly large size), but then they freak out when you say they need to upgrade their VPS to 32BG of RAM. :-)
Best Answer
If anyone stumbles across the same phenomenon, here's what I found out (spoiler: I'm stupid):
Of course, what should have immediately raised an alarm was the fact that I wrote
openwsmand
when thexapi
process is actually responsible for handling connection attempts from XenCenter and the likes. What happened here was that I just did a simplenetstat -plant
and noticed thatopenwsmand
was listening at port 443, which I knew was the port that XenCenter tried to connect to, so I assumed this was the right process.But of course, XenCenter can't connect to
openwsmand
because this is a tool of the OpenManage Suite by Dell. I suppose that when I first installed OpenManage,openwsmand
tried to bind to port 443, but this was already in use byxapi
, so it gave up and I was still able to connect.When I restarted the machine, however, the
openwsmand
service was started beforexapi
, so it occupied port 443, leaving me with no possibility to connect using XenCenter.A simple
fixed the problem. Currently running VMs won't be affected by this operation.