I have Icinga 1.10 installed and I am using NagiosQL to configure it. I am stuck a bit on tweaking alerts and notifications.
Icinga has implied inheritance. Assume we have a production web server called prodweb01. It has two service checks running on it. One to check if Apache is working and one to check if a JVM is working. There are no alarm settings set for either checks and hence it inherits those settings from the host. The host inherits two host templates – Generic-Linux-Host (which has contacts for Sysadmins) and Generic-Web-Host (which has contacts for Developers). Is there anyway to write logic so that if Apache goes down, sysadmins get alerted and if JVM goes down, developers get alerted?
Is this even the best way of doing things? This approach will get tedious for us because we have hosts separated by environment and roles. So we are going to have to create prod/non-prod Generic-Linux-Host templates and prod/non-prod templates for each server role (eg. api, app, web, etc). We are going to have to do this so that the non-prod notifications don't go to SMS while the prod ones do because I haven't seen a way to tell Icinga to alert via SMS for prod and via email for non-prod. The only way to do this is it seems by creating two different contacts for the same person, one with SMS, one with email and applying it to the template or host based on it's environment.
Edit: I realize that someone may point out that why don't I just define the alarm/notification settings in the service itself? Yes, that can be done but that still leaves the question unanswered about how to separate alerts for prod from non-prod. In this scenario, the solution would be to create two versions of the same service check and set SMS alerts in one and email in the other.
Best Answer
Solved my problem as such.
Define contacts.cfg as such
In icinga.cfg, enable this -
In /etc/icinga/, create the two scripts that are being called -
custom-host-notify
custom-service-notify
You can always customize the messages and variables to your liking.
Restart Icinga and you are good to go.
Edit: I decided to forgo implied inheritance and just set the notifications in the service-templates. You can set the notifications in the services itself.