Linux – How to turn linux to keep ‘cache’ high

linuxmemoryperformance-tuning

We are running a heavy service on Debian. Our service generates lots of file i/o and its performance relies on how well Linux kernel caches files.

Our system has 64GB memory. Total file size is 500GB (lots of small files). These files are served via reiserfs on raid. The kernel is 3.2.9 now. (this is not a new issue. We have this issue for several years.)

Usually, most of file content are cached and our system runs smoothly. free(1) command shows 14G memory cache, 10G memory buffer, and 8G memory are free.

However, in our peak hour, I found linux kernel released most of cache/buffer and kept them in free state. In my opinion, it is a huge waste to keeps memory in free state and I guess it is the reason that our service overloads in busy hours. free(1) command show something like 2G cache, 2G buffer, and 28G free.

My question is, in what situation linux kernel want to release memory instead of caching/buffering. And how to avoid this behavior? Is there any configuration we can tune?

(Let me know if you need more information about our hardware/system/service)

Best Answer

There are two keys reasons this would happen.

One, that I hope you can rule out, is some person or script commanding it to do so. For example, someone sending a command to /sys/vm/drop_caches.

The other is if that memory is required for some other purpose. If an application requires a lot of memory, the kernel will give the application that memory, reducing the cache as needed. When the application is done with that memory, it will become free. It will only fill back up with cached data over time.

There may be a few things you can do if this is the case. For example, if the memory usage is happening because you're trying to process four requests at once, you may wish to reconfigure to do two requests at once maximum. If this lets you keep more data in cache, it may be a performance win overall.

Related Solutions

Linux RAM Cache – Caching and Preloading Files into RAM

vmtouch seems like a good tool for the job.

Highlights:

query how much of a directory is cached
query how much of a file is cached (also which pages, graphical representation)
load file into cache
remove file from cache
lock files in cache
run as daemon

vmtouch manual

EDIT: Usage as asked in the question is listed in example 5 on vmtouch Hompage

Example 5

Daemonise and lock all files in a directory into physical memory:

vmtouch -dl /var/www/htdocs/critical/

EDIT2: As noted in the comments, there is now a git repository available.

Linux memory usage

Since I see vmware-guestd running, I'll assume that this is a VM. Assuming ESX, go into the VIC and right-click the machine. Click Edit Settings... and select the Resources tab. Click Memory and ensure that Unlimited is check under Limit.

What is likely happening is that you have a limit on the memory, and when the machine uses lots of memory (approaches or exceeds the limit), the vmware guest tools keep the memory "used" rather than releasing it back to ESX, in case it's needed again.

UPDATE:

"The server allows you to power on a virtual machine only if the CPU and memory reservation is available... When resources are not used, the ESX Server host makes them available to other virtual machines."

http://www.vmware.com/pdf/vi3_esx_resource_mgmt.pdf

It does this by way of the "balloon" driver (vmmemctl) which is part of the guest tools. Say your machine is using 100MB of ram, then you run some program and it jumps up to 500MB used. Now you stop that program and expect the RAM to go back down to 100MB. However, it doesn't. This is because, in order for ESXi to reclaim the memory back from the guest OS (which it does even when there is a reservation) it must use the balloon driver. To make the OS "act" as though it has less memory, the balloon driver "uses" the memory that ESXi wants, so that the OS can't use it.

IOW, even with a reservation the guest is only using as much host RAM as it is using. When it uses more, ESX allocates more host RAM which the guest now thinks it always has available. To "convince" the guest that the new ram is gone again, the balloon driver uses it up. The reservation simply means "Don't start the VM unless the physical machine has 1GB free to load the guest into" and not "Give the guest 1GB whether it uses it or not."

Best Answer

Related Solutions

Linux RAM Cache – Caching and Preloading Files into RAM

Linux memory usage

Related Topic