MySQL Replication Errno1146 Table doesn’t exist

This is how i did it for both masters

log-bin = mysqld-bin
binlog-ignore-db=test
binlog-ignore-db=information_schema
binlog-ignore-db=mysql
log-slave-updates
replicate-ignore-db=test
replicate-ignore-db=information_schema
replicate-ignore-db=mysql
relay-log=mysqld-relay-bin

The big problem with replication is checking

1. that the nodes are all up,
2. all nodes are communicating (not split brain)
3. and processing replication logs
4. and the replication lag

1, 3 and 4 can be captured usingf SHOW MASTER STATUS / SHOW SLAVE STATUS on the relevant nodes, although the replication delay only has 1 second accuracy and only across each hop. The Percona toolkit has scripts for getting more accurate replication lags.

Using multi-master replication (e.g. tungsten, Percona) saves a lot of pain but needs additional effort / software to set up.

If the network between ndoes fails, then the processes can all be running be fine - but will be unable to transfer data - you need monitoring on each node to check it can contact the upstream node.

a MySQL Master database which replicates to a few Slave databases

Best practice would be to designate one of the slaves as a master too - bi directional replication. That way you can easily switch over in the event of an outage, or for maintenance tasks such as rebuilding indexes, backups, schema changes.

Depending on the number of slave nodes, you might also want to designate a fanout node to propogate the changes.

In terms of managing escalations, scheduling scripts to collect data etc, there are lots of tools available which do this - I use nagios, so do lots of other people.