Linux – tuning linux cache settings for inode caching

cacheinodelinux

I am dealing with hundred million files in a filesystem (distributed among a lot of subdirectories), and I need to be able to list them very quickly, particularly in order to rsync them efficiently.

On a other hand, I don't really need to have the actual content of the file kept in cache.

I am constantly adding and removing files, but not that frequently (something like ten times per second).

Is there a way I can tell the OS (2.6.18-194.el5) to use the 24GB available RAM more on inode caching than on file caching? I already looked at /proc/etc/vm/vfs_cache_pressure but it doesn't seem to be exactly what I am looking for…

Best Answer

How about lowering the value for vfs_cache_pressure? According to Documentation for /proc/sys/vm/* this should do what you want:

vfs_cache_pressure

This percentage value controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects.

At the default value of vfs_cache_pressure=100 the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. When vfs_cache_pressure=0, the kernel will never reclaim dentries and inodes due to memory pressure and this can easily lead to out-of-memory conditions. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes.

Increasing vfs_cache_pressure significantly beyond 100 may have negative performance impact. Reclaim code needs to take various locks to find freeable directory and inode objects. With vfs_cache_pressure=1000, it will look for ten times more freeable objects than there are.

Related Solutions

Linux – Caching/preloading files on Linux into RAM

vmtouch seems like a good tool for the job.

Highlights:

query how much of a directory is cached
query how much of a file is cached (also which pages, graphical representation)
load file into cache
remove file from cache
lock files in cache
run as daemon

vmtouch manual

EDIT: Usage as asked in the question is listed in example 5 on vmtouch Hompage

Example 5

Daemonise and lock all files in a directory into physical memory:

vmtouch -dl /var/www/htdocs/critical/

EDIT2: As noted in the comments, there is now a git repository available.

Linux – Inode too big. Linux Filesystem

Check is your version of e2fsprogs (e.g. e2fsck -V). There was apparently a bug prior to version 1.40.2 that could cause e2fsck (which is called by fsck for EXT2/EXT3 filesystems) to incorrectly report filesystem corruption of this type:

Fix a bug in e2fsck which caused it to incorrectly salvange directories when the last entry's rec_len is bogusly too big. This resulted in a nonsense filesystem corruption to be reported, and required a second run of e2fsck to fully fix up the directory.

The RHEL 4 servers that I have access to have version 1.35 on them, so this could be the real cause of your problem. It might be worth installing a newer version and running fsck again to be sure.

If that doesn't help, it may indeed be a filesystem problem. According to this thread, it appears that this can happen if you have a directory containing a very large number of files (millions or more). "Fixing" it with fsck will result in the inode being truncated, as you saw. In the case of a directory inode, truncation will cause the directory to "lose" files, and those orphaned files will end up in the filesystem's lost+found directory. They will be named after their inode number, not their original names, so although you'll still have the files, it may be difficult to tell them apart.

Best Answer

vfs_cache_pressure

Related Solutions

Linux – Caching/preloading files on Linux into RAM

Linux – Inode too big. Linux Filesystem

Related Topic