Zabbix flapping detection

not an answer really, but some pointers:

• definitivly take a look at presentation about nagios @ goldman sachs. they faced problems you mention - redundancy, scalability: thousands of hosts, also automated configuration generation.

• i had redundant nagios setup but at much smaller scale - 80 servers, ~1k services in total. one dedicated master server, one slave server pulling configuration from master at regular intervals few times a day. both servers covered monitoring of the same machines, they had health cross-check between each other. i used nagios mostly as framework for invoking custom product specific checks [ bunch of cron jobs executing scripts doing 'artificial flow controls', results ware logged to sql, nrpe plugins ware checking for successful / failed executions of those in last x minutes ]. all worked very nicely.

• your quorum logic sounds good - a bit similar to my 'artificial flows' - basically go on, ipmplement your self ;-]. and have nrpe just check some kind of flag [ or sql db with timestamp-status ] how things are doing.

• you'll probably want to build some hierarchy to scale - you'll have some nodes that gather overview of other nodes, do look at presentation from first point. default nagios forking for every single check is overkill at higher number of monitored services.

to answer some questions:

• in my case environment monitored was typical master-slave setup [ primary sql or app server + hot standby ], no master-master.
• my setup involved 'human filtering factor' - resolver group who was a 'backup' for sms notification. there was already paid group of technicians who for other reasons had 24/5 shifts, they got 'checking nagios mails' as additional task not putting too much load on them. and they ware in charge of making sure that db-admins / it-ops / app-admins ware actually getting up and fixing problems ;-]
• i've heard lot's of good things about zabbix - for alerting and plotting trends, but never used it. for me munin does the trick, i have hacked simple nagios plugin checking if there is 'any red' [ critical ] color on munin list of servers - just an additional check. you can as well read values from munin rrd-files to decrease number of queries you send to monitored machine.

I think it would be best to concentrate on answering the specific questions you had, taking into account the size of your planned deployment (~10 monitored hosts).

1. What are the general disadvantages of Zabbix?

   • it won't automagically figure out what to monitor, when to alert you and etc - you will have to think about what metrics you are interested in and configure them upfront
   • debugging leaves something to be desired. although with such a small environment help options like forum, irc channel etc should suffice easily

2. Does Zabbix have a small footprint on boxes it is monitoring?

   Yes, definitely. Zabbix can monitor using methods like SNMP, simple network checks (is a port open?), and it also has native agent for many platforms. As the agent is written in C, it has an extremely small footprint (as opposed to bunch of interpreted scripts...). You can easily combine different checks on a single monitored host. Note that you are not limited to monitoring servers, you can also add network devices and other things.

3. Do I really need to setup an entire other server for it? I currently have a server that is under very light load -- can I dual purpose it?

   Depends - if it's running one of the supported operating systems for the server - definitely. For that environment requirements will be really low. Make sure to use default templates only as a guideline, it's suggested to create your own with longer intervals between checks. Basically, Zabbix consists of 3 components - DB, frontend, server. If you desire so, you can reuse existing database server and existing webserver in the company for the first two components, and then run Zabbix server on any supported platform - that's a perfectly valid configuration.

Any specific queries would be very welcome in #zabbix on Freenode.