Munin : notification when everything is ok

munin

I use one Munin-Master to monitore 20+ servers, all run fine, except for one server. Last three Munin mails receveid :

05h25

infra :: backup2.infra :: Disk usage in percent
OKs: /var is 22.55, /run/user/1001 is 0.00, /home is 8.87, /mnt/usb1 is 30.55, /export/oxa is 51.58, /tmp is 0.60, /dev/shm is
0.00, /space2 is 40.39, /run is 8.77, /run/lock is 0.00, /run/user/65534 is 0.00, /space is 76.38, /sys/fs/cgroup is 0.00, / is
18.46.

infra :: backup2.infra :: Inode usage in percent
OKs: /dev/shm is 0.00, /run is 0.05, /space2 is 7.44, /run/user/65534 is 0.00, /run/lock is 0.00, /sys/fs/cgroup is 0.00,
/space is 0.24, / is 8.07, /dev is 0.03, /home is 0.13, /mnt/usb1 is
0.51, /export/oxa is 0.01, /tmp is 0.02, /var is 2.02, /run/user/1001 is 0.00.

07h00

infra :: backup2.infra :: Inode usage in percent
OKs: /home is 0.13, /var is 2.02, /run/user/1001 is 0.00, /dev/shm is 0.00, /run is 0.05, /run/lock is 0.00, /space is 0.24,
/run/user/1003 is 0.00, /tmp is 0.02, / is 8.07, /space2 is 7.44,
/mnt/usb1 is 0.51, /export/oxa is 0.01, /dev is 0.03, /sys/fs/cgroup
is 0.00.

08h50

infra :: backup2.infra :: Inode usage in percent
OKs: /run/user/1001 is 0.00, /tmp is 0.02, /dev is 0.03, /run/user/0 is 0.00, /dev/shm is 0.00, /run is 0.05, /space is 0.24,
/sys/fs/cgroup is 0.00, /mnt/usb1 is 0.51, / is 8.07, /home is 0.13,
/space2 is 7.44, /run/lock is 0.00, /var is 2.02, /export/oxa is 0.01.

infra :: backup2.infra :: Disk usage in percent
OKs: / is 18.46, /mnt/usb1 is 30.62, /sys/fs/cgroup is 0.00, /export/oxa is 51.62, /run/lock is 0.00, /var is 22.29, /space2 is
40.39, /home is 8.87, /tmp is 0.60, /run/user/1001 is 0.00, /space is 76.49, /dev/shm is 0.00, /run is 9.27, /run/user/0 is 0.00.

All is Ok, no error in master logs, but still I received lots of these messages.

Here are the logs on master about this node

munin-update.log:2016/03/25 10:40:24 [WARNING] Service nfs4_client on
backup2.infra/backup2.admin2:4949 returned no data for label fsinfo
munin-update.log:2016/03/25 10:40:21 [WARNING] Service nfs_client on
backup2.infra/backup2.admin2:4949 returned no data for label remove

munin-update.log:2016/03/25 09:55:06 [INFO] starting work in 29082 for
backup2.infra/backup2.admin2:4949. munin-update.log:2016/03/25
09:55:06 [INFO] node backup2.infra advertised itself as backup2
instead. munin-update.log:2016/03/25 09:55:12 [INFO]: Munin-update
finished for node infra;backup2.infra (6.67 sec)
munin-update.log:2016/03/25 09:55:13 [INFO] Reaping
Munin::Master::UpdateWorker. Exit value/signal:
0/0

Configuration for notification

 contact.devs.command mail -s "Munin notification ${var:host}" my@mail.com 
 contact.devs.always_send warning critical

Here is the configuration file for this node (generated, as for all nodes)

[backup2.infra]
     address backup2.admin2
     use_node_name yes
     diskstats_latency.backup2_store_export.avgrdwait.warning :7
     diskstats_latency.backup2_store_export.avgwrwait.warning :7
     diskstats_latency.backup2_store_export.avgrdwait.critical :10
     diskstats_latency.backup2_store_export.avgwrwait.critical :10

Munin Master and node version: 2.0.25-1 (both Debian Jessie)

Where can I watch to undestand and resolve ?

Best Answer

The df plugin in Debian also checks the dynamically mounted filesystems under /run/user/<uid> which appear when a user logs in and which disappear when the user logs out. Even though all levels are OK, this appearance and disappearance is considered a change which triggers an email.

You should be able to avoid this by creating a file called /etc/munin/plugin-conf.d/df with the following content:

[df*]
env.exclude_re /run/user/

To check if your settings work and to list which paths the df plugin considers, use the following command:

munin-run -d df

If you are happy with the outcome, restart the munin-node service (service munin-node restart).

Related Topic