I`ve installed icinga2 on a virtual machine for monitoring 5 other ones with graphite support for the graphics and icingaweb2 as an interface. Recently on my web interface I got the follow:
Fatal error: Uncaught exception 'ErrorException' with message 'session_start(): open(/var/lib/php5/sess_3aabfsu76vcdj2hd2u3teofta5dvanpn, O_RDWR) failed: No space left on device (28)' in /usr/share/php/Icinga/Web/Session/PhpSession.php:105 Stack trace: #0 [internal function]: Icinga\Application\ApplicationBootstrap->Icinga\Application\{closure}(2, 'session_start()...', '/usr/share/php/...', 105, Array) #1 /usr/share/php/Icinga/Web/Session/PhpSession.php(105): session_start() #2 /usr/share/php/Icinga/Web/Session/PhpSession.php(215): Icinga\Web\Session\PhpSession->open() #3 /usr/share/php/Icinga/Authentication/Auth.php(344): Icinga\Web\Session\PhpSession->refreshId() #4 /usr/share/php/Icinga/Authentication/Auth.php(166): Icinga\Authentication\Auth->persistCurrentUser() #5 /usr/share/php/Icinga/Authentication/Auth.php(262): Icinga\Authentication\Auth->setAuthenticated(Object(Icinga\User)) #6 /usr/share/php/Icinga/Authentication/Auth.php(93): Icinga\Authentication\Auth->authExternal() #7 /usr/share/php/Icinga/Application/Web. in /usr/share/php/Icinga/Web/Session/PhpSession.php on line 105
Fatal error: Uncaught exception 'ErrorException' with message 'Unknown: open(/var/lib/php5/sess_3aabfsu76vcdj2hd2u3teofta5dvanpn, O_RDWR) failed: No space left on device (28)' in Unknown:0 Stack trace: #0 [internal function]: Icinga\Application\ApplicationBootstrap->Icinga\Application\{closure}(2, 'Unknown: open(/...', 'Unknown', 0, Array) #1 {main} thrown in Unknown on line 0
To check what`s going on I tried to find a problem in the disk free space:
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/hostname--vg-root 6.5G 4.8G 1.4G 78% /
tmpfs 201M 412K 200M 1% /run
...
...
/dev/sda1 236M 154M 70M 69% /boot
The problem occurred in the fulfilled inodes:
#df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/hostname--vg-root 441504 441504 0 100% /
...
...
/dev/sda1 62248 318 61930 1% /boot
To check where are the biggest part of the tiny-files I did:
#for i in /*; do echo $i; find $i |wc -l; done
And found that that the point is in /usr and /var/spool/icinga2/perfdata/ directory
/usr
177882
/var/spool/icinga2/perfdata
229326
The question is in the reasons why icinga2 doesn't clean it's performance data and in the way how to clean it manually without damaging. Thanks.
Best Answer
Icinga 2 does not clean rotated performance data files. It expects external daemons/applications to remove these files after being processed. If you're using PNP for example, the NPCD daemon will collect the rotated files, process them and remove them afterwards.
You'd also want to add a "disk" check with inode threshold values to your monitoring in order to find out about such issues in advance.