I am not that strong in linux / unix so asking for any help on problem I have. I am running site on amazon ec2. I am running apache tomcat mysql on one large instance 8gb of RAM. Database is quite large but I don't have problems with performance and there is around 3gb of free memory on the box. I noticed that about once in a hour box becomes virtually frozen, so site is not responding and even ssh shell is very slow. I enabled performance monitoring and noticed that amazon reports that each hour or so there are huge disk i/o spikes.
I am running UBUNTU with kernel release 2.6.32-305-ec2 So I started monitor with vmstat and noticed that in normal operation box reports around 1.5gb free memory and 2gb of cache and cache is gradually grows. but at some point cache starts growing takes whole ram leaving like 1mb free or something like that. And then I see lots of processes waiting for disk and everything depening on drives i/o freezes. It lasts from 3 to 5 minutes usually. My understanding that system is dumping cache to the drive and freeing memory and that is what causing all that mess. I did not have this problem before I think it started recently when I loaded much more data in mysql. But again there are plenty of free memory it just management of the memory and cache by linux causing it. I read that it should be seamless and it is good thing to have buffer cache but having site down for 5 minutes each hour is definitely bad thing for me. Please advice what options I have with it. I could not find anything on internet.
That is what vmstat outputs
r b swpd free buff cache si so bi bo in cs us sy id wa
0 30 0 45512 134300 4062404 0 0 437 685 31 40 11 4 74 6
5 31 0 47200 134328 4057136 0 0 28188 15508 1682 2255 13 9 2 66
3 33 0 46904 134328 4057048 0 0 16761 8785 1099 1523 4 3 0 89
0 35 0 48440 134344 4054908 0 0 23688 11572 1453 1782 6 3 0 85
Best Answer
Take a look at Greg Smith's A Linux write cache mystery. You may need to decrease dirty_ratio and dirty_background_ratio.