So we run Groundworks (with Nagios) on CentOS to monitor our various servers and processes. I have it setup to automatically send emails and SMS texts when things reach a WARNING or CRITICAL state. Normally this works perfectly. However, twice we've had problems with Postfix on that server where Postfix decides to stop sending email. The most recent time lasted 4 days because none of us noticed.
That leads me to a important question: how am I supposed to monitor my monitoring server?
Best Answer
With a second monitoring server, of course. The second one can be much simpler, since all it needs to do is monitor the first. And it should be monitored by the main monitoring system in turn, of course.
If your group is part of a larger organization with separate IT infrastructures, you may be able to make arrangements for another group's monitoring service to watch yours.
You could also make sure the server sends an "it's okay" message every day, and get in the habit of looking for it. (That's only effective if you're not already overwhelmed with routine messages, of course.)