Linux – how to find out what is causing huge dentry_cache usage

linuxmemory

Note that inode_cache & ext3_inode_cache slabs are very small compared to dentry_cache.
What happens is that slowly and steadily the within a week dentry_cache grows from 1M to ~5-6G
Then I need to run
echo 2 > /proc/sys/vm/drop_caches && echo 0 > /proc/sys/vm/drop_caches
This started to happening one day on all servers hosting some web code – the developers are saying that they have not changed anything related to filesystem access pattern around the time then the problem started.

The system is centos5 with 2.6.18 kernel so I don't have any instrumentation features available th newer kernels.
Any I idea how I can debug the problem? maybe with systemtap? This is a ec2 instance – so not even sure that systemtap will work there.

Thanks
Alex

Best Answer

Late, but maybe useful for others who come upon this.

If you are using the AWS SDK on that EC2 instance, it is highly likely that curl is causing the dentry bloat. While I haven't seen this trigger OOM, it is known to impact the performance of the server, due to the additional work required by the OS to reclaim SLAB.

If you can confirm that curl is being used by your developers to hit https (many of the AWS SDK do this), then the solution is to upgrade the nss-softokn library to at least v3.16.0 and set the environment variable, NSS_SDB_USE_CACHE (YES and NO are valid values, you may have to benchmark to see which performs curl requests more efficiently) for the process which is using libcurl.

I recently ran into this myself and wrote a blog entry (old blog entry link and upstream bug report) with some diagnostics & more detailed information, in case that helps.

Related Solutions

Linux – Find out symbolic link target via command line

Use the -f flag to print the canonicalized version. For example:

readlink -f /root/Public/myothertextfile.txt

From man readlink:

-f, --canonicalize
      canonicalize by following every symlink in every component of the given name recursively; all but the last component must exist

Linux Huge Pages Usage Accounting

This is because HugePages_rsvd is essentially read from HugePages_Free. Meaning, out of 596 huge pages which are free, 594 are already reserved by some application for use. That is kernel has committed that those 594 huge pages are available for the application.

If there is a request for 3 huge pages now, then it will fail as only 2 are available to be reserved. Think of it as a malloc() call, when you reserve memory virtual pages to account for the VSZ for the process but when the process actually uses them, it becomes the RSZ (running set) of the process.

As huge pages are always resident on main memory, when an app requests them kernel decrements it from free pool and increase the Rsvd counter.

This is from the kernel source. https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt

where:
HugePages_Total is the size of the pool of huge pages.
HugePages_Free  is the number of huge pages in the pool that are not yet
                allocated.
HugePages_Rsvd  is short for "reserved," and is the number of huge pages for
                which a commitment to allocate from the pool has been made,
                but no allocation has yet been made.  Reserved huge pages
                guarantee that an application will be able to allocate a
                huge page from the pool of huge pages at fault time.
HugePages_Surp  is short for "surplus," and is the number of huge pages in
                the pool above the value in /proc/sys/vm/nr_hugepages. The
                maximum number of surplus huge pages is controlled by
                /proc/sys/vm/nr_overcommit_hugepages.

Best Answer

Related Solutions

Linux – Find out symbolic link target via command line

Linux Huge Pages Usage Accounting

Related Topic