Linux – Clearing out deleted memory mapped files quicker

fileslinuxlsof

We use Tika for PDF text content extraction for searching. I'm seeing heavy use of temporary files that seem to now run us out of file handles.

We're on CentOS 5.5. For our search system, we had previously bumped up open file handles (/etc/sysctl.conf) fs.file-max = 65535

When looking at lsof for my process, I see lots of files marked as DEL (deleted memory mapped files). If I stop submitting new documents to be indexed, these will get cleared out after several minutes.

java 11105 root DEL REG 104,2 1278402 /tmp/+~JF4155000471009101661.tmp

Can I tune the kernel to clear out these DEL files sooner, or should I increase the number of file handles? If so, how high can we safely go? I have sufficient memory to be able to give more to the kernel.

Thanks

David

Best Answer

I couldn't find a good reference for how increasing fs.file-max impacts the system. Presumably that makes your kernel consume more memory as the structure to track open files gets larger and larger. I suggest increasing to a larger number like 128K and keeping an eye on /proc/sys/fs/file-nr to see how many files are actually used.

Related Solutions

How to Avoid Linux Out-of-Memory Application Teardown

By default Linux has a somewhat brain-damaged concept of memory management: it lets you allocate more memory than your system has, then randomly terminates a process when it gets in trouble. (The actual semantics of what gets killed are more complex than that - Google "Linux OOM Killer" for lots of details and arguments about whether it's a good or bad thing).

To restore some semblance of sanity to your memory management:

Disable the OOM Killer (Put vm.oom-kill = 0 in /etc/sysctl.conf)
Disable memory overcommit (Put vm.overcommit_memory = 2 in /etc/sysctl.conf)
Note that this is a trinary value: 0 = "estimate if we have enough RAM", 1 = "Always say yes", 2 = "say no if we don't have the memory")

These settings will make Linux behave in the traditional way (if a process requests more memory than is available malloc() will fail and the process requesting the memory is expected to cope with that failure).

Reboot your machine to make it reload /etc/sysctl.conf, or use the proc file system to enable right away, without reboot:

echo 2 > /proc/sys/vm/overcommit_memory

Linux – Any way to determine how files were deleted on Linux/Apache server

By default, this isn't logged (imagine how big the logs would get). You can add logging with auditctl or inotify/iwatch (interesting options, check them out), or your control panel logging or ftp logs (usually all in /var/log somewhere) or bash history might help you track it down a little. Check your HTTP access logs in case some GET request managed to do this (unlikely, but possible)

If your history isn't there, it has been cleared. This is either done manually, or a command in your .bash_logout or .logout files.

Also, it's SSH not SSL :)

Best Answer

Related Solutions

How to Avoid Linux Out-of-Memory Application Teardown

Linux – Any way to determine how files were deleted on Linux/Apache server

Related Topic