Inode Table Usage Spiking Every Morning at 8am

inodemunin

I installed munin on my ubuntu server. It's showing my 'inode table usage' spiking every morning at 8am. It then rapidly curves down and settles over the course of several hours. What might cause this?

enter image description here

I thought it might be something running in /etc/cron.daily but this was set to run at 6a.m. and I've changed it to 4am. The spike remains at 8am. I also enabled cron logging, but can't see anything getting launched at 8a.m.

It's a virtual server hosted by memset. Could it be caused by something happening on the virtual host?

Best Answer

On a default ubuntu/debian set-up, some things will run via 'anacron'. The timing of this can depend on a number of factors, particularly when the machine started up, but on a server machine it will trigger at 7:30 a.m. every morning by default.

This is configured in the file /etc/cron.d/anacron , so a simple solution is to edit that file and and set the configured time to something more convenient (e.g. middle of the night if you are concerned about the tasks impacting day time performance) There is no need to restart anything.

The cron log which I enabled, was in fact showing the initiation of anacron at 7:30. Anacron delays for a little while before doing anything and also presumably queued up several jobs, the result being that one job was starting up at about 8a.m. each day and causing a lot of inode table usage (people have suggested the mlocate db re-index) I don't know that it was causing any performance problems, but it seems healthier now to see it doing this in the middle of the night.

There is some brief documentation specific to ubuntu here: https://help.ubuntu.com/community/CronHowto#How_Anacron_is_Set_Up

...and much more detailed explanation of the interplay between cron and anacron here: https://linuxformat.com//tuxradarchive/content/automate-linux-cron-and-anacron

Related Solutions

Linux – Determine Location of Inode Usage

Don't expect this to run quickly...

cd to a directory where you suspect there might be a subdirectory with lots of inodes. If this script takes a huge amount of time, you've likely found where in the filesystem to look. /var is a good start...

Otherwise, if you change to the top directory in that filesystem and run this and wait for it to finish, you'll find the directory with all the inodes.

find . -type d | 
while 
  read line  
do 
  echo "$( find "$line" -maxdepth 1 | wc -l) $line"  
done | 
sort -rn | less

I'm not worried about the cost of sorting. I ran a test and sorting through the unsorted output of that against 350,000 directories took 8 seconds. The initial find took . The real cost is opening all these directories in the while loop. (the loop itself takes 22 seconds). (The test data was run on a subdirectory with 350,000 directories, one of which had a million files, the rest had between 1 and 15 directories).

Various people had pointed out that ls is not great at that because it sorts the output. I had tried echo, but that is also not great. Someone else had pointed out that stat gives this info (number of directory entries) but that it isn't portable. It turns out that find -maxdepth is really fast at opening directories and counts .files, so... here it is.. points for everyone!

Linux – What does it mean if inode usage is high

The file-system uses inodes to track file locations on disk. If you don't have any more inodes, you can't write more files to the file-system until there are more available. It is best to plan what will live on the file-system before formatting it as with many file-systems, you can choose the inode default size appropriate to the size of file that will live there (i.e. larger inode size for larger files, smaller for smaller files) and thus maximize the number of files to file-system space.

Best Answer

Related Solutions

Linux – Determine Location of Inode Usage

Linux – What does it mean if inode usage is high

Related Topic