Linux – How to debug inode usage

Don't expect this to run quickly...

cd to a directory where you suspect there might be a subdirectory with lots of inodes. If this script takes a huge amount of time, you've likely found where in the filesystem to look. /var is a good start...

Otherwise, if you change to the top directory in that filesystem and run this and wait for it to finish, you'll find the directory with all the inodes.

find . -type d | 
while 
  read line  
do 
  echo "$( find "$line" -maxdepth 1 | wc -l) $line"  
done | 
sort -rn | less

I'm not worried about the cost of sorting. I ran a test and sorting through the unsorted output of that against 350,000 directories took 8 seconds. The initial find took . The real cost is opening all these directories in the while loop. (the loop itself takes 22 seconds). (The test data was run on a subdirectory with 350,000 directories, one of which had a million files, the rest had between 1 and 15 directories).

Various people had pointed out that ls is not great at that because it sorts the output. I had tried echo, but that is also not great. Someone else had pointed out that stat gives this info (number of directory entries) but that it isn't portable. It turns out that find -maxdepth is really fast at opening directories and counts .files, so... here it is.. points for everyone!

You can't create a link to it, but you can get it back. Let's do an experiment:

$ echo blurfl >myfile.txt
$ tail -f myfile.txt &
$ rm myfile.txt

myfile.txt is now gone, but the inode is kept alive by the tail command. To get your file back, first find the PID of the process keeping the inode:

$ ps auxw | grep tail
sunny      409  0.0  0.0   8532   824 pts/5    S    18:07   0:00 tail -f myfile.txt

The PID is 409. chdir to /proc/409/fd/ and list the contents:

dr-x------ 2 sunny sunny  0 2009-07-24 18:07:18 .
dr-xr-xr-x 7 sunny sunny  0 2009-07-24 18:07:17 ..
lrwx------ 1 sunny sunny 64 2009-07-24 18:07:33 0 -> /dev/pts/5
lrwx------ 1 sunny sunny 64 2009-07-24 18:07:33 1 -> /dev/pts/5
lrwx------ 1 sunny sunny 64 2009-07-24 18:07:18 2 -> /dev/pts/5
lr-x------ 1 sunny sunny 64 2009-07-24 18:07:33 3 -> /home/sunny/tmp/myfile.txt (deleted)

The /proc/[PID]/fd/ directories contain symlinks to file descriptors of all files the process uses. In this case the symlink "3" points to the deleted file. So, to restore the file, copy the contents to a new file:

$ cat 3 >/home/mydir/saved_file.txt