Linux – How to determine which processes have most inodes opened

inodelinuxmunin

Here is my problem, visibile in a munin chart:

munin inode chart

My used/opened inodes are "suddenly" constantly increasing.

Is there a way to determine which process is keeping the most currently opened inodes?
I used the approach of https://unix.stackexchange.com/questions/117093/find-where-inodes-are-being-used and found and cleaned some folders with mails and logs I could get rid of…
Still, if the inodes are OPEN, there must be some process keeping them in use, right? It might not necessarily be the folder with the most files where the increase is coming from – or am I wrong there?

So I would like to see who is keeping them open and then track the usage to see where the increase is coming from

Update

Based on Andrew's script, I created a version of it that also shows the process name. As I have some nginx/apache processes running that may respawn, I'd like to see the results on process name.

for dir in /proc/*/fd;
do
    echo -n "$dir ";
    pid=`expr "$dir" : '\/proc\/\(.*\)\/.*'`; # extract the pid
    pname=`ps -p $pid -o comm=`; # grab process name
    echo -n "$pname ";
    ls $dir 2>/dev/null | wc -l;
done | sort -n -k 3

Sample output:

/proc/4612/fd sshd 49
/proc/46470/fd node 60
/proc/5655/fd nginx 66
/proc/6656/fd nginx 76
/proc/7654/fd nginx 81
/proc/8578/fd dovecot 107
/proc/9657/fd nginx 117
/proc/3495/fd java 146
/proc/4785/fd mysqld 382

So the next test would be logging the distribution over time to see what changes and how it correlates with the number of /proc/sys/fs/inode-nr that Morgan mentioned

One year later…

Some time has passed, here is a new chart
munin open inodes

And guess what, end of September is the point where a faulty drive was replaced.
So it looks like the whole mess was generated by a disk error.
Nevertheless, the scripts still are useful!

Best Answer

Count the number of entries in each /proc/[PID]/fd directory. That will give you the number of file descriptors each process has open. While it will take a while to enumerate all processes, missing processes that start or stop while your counting is in progress shouldn't be a problem as you're looking for a long-lived process with a lot of open file descriptors.

Something like this:

for dir in /proc/*/fd;
do
    echo -n "$dir "; #need a space to get real columns for the sort
    ls $dir 2>/dev/null | wc -l;
done | sort -n -k 2

The last lines of output will show the /proc/[PID]/fd directory with a count of open file descriptors for each process. The culprit process(es) should be there near the bottom.

Note that each entry in /proc/[PID]/fd is technically a file descriptor and not a separate open inode, each separate open inode must have at least one separate file descriptor in a /proc/[PID]/fd directory somewhere.

Related Topic