Sorry for being unclear previously.
We have a vmware virtualized server instance that is our main production server. I stores a series of web based applications on close to a hundred unique top level domains. For serving web pages we use a LAMP stack. This server is running our primary and secondary dns servers (on two different ip address than that used for serving web content). And finally we also host our mail (pop and smtp) using exim (i believe).
Recently we've had issues causing our root fs to become read only, preventing apache2 or mysql connections and prevent incoming email. Essentially taking down the web presence and email for many thousands of clients. The nature of the issue (still undetermined by under control) did not affect the bind so dns was still resolving fine.
Since then, we have begun to mirror the production web sites and associated mysql databases onto a secondary server. This server is completely production ready.
My question is, what are recommended methods for a failover in the case that apache on our main production server fails (for what ever reason) to quickly, if not automatically, start forwarding traffic as seamlessly as possible to the secondary.
DNS round robining is undesirable for us since we do not wish to the load over two servers, in fact we only ever want the secondary to receive http requests in the case the main server is non-responsive. This is in part to the fact that our mirroring process is one way and changes to the secondary server would be reflected in the main server and even lost.
Best Answer
DNS round robin is not recommended because:
1- Different servers may not be exposed to the same amount of requests. So, they will be loaded in an unevenly manner.
2- DNS load balancing does not take into account the server availability. The server DNs record will remain and may be used in case of failure.
3- DNS caching will make it even worse. You don't have control over the DNS caches of your clients and any intermediate DNS server in between. If you plan to make your TTL value samller, it may not work as expected. Look at this post. The accepted answer says that
Many DNS server do not honor your TTL.The recommended solution is to install a load balancer like HAProxy along with a high availability solution like heartbeat. This setup should be installed on two machines. If one goes down, the other will take over the VIP (by heartbeat). The running machine will take care of checking backend servers health and distributing the load (by haproxy).
EDIT:
If you want the servers to work in active-passive mode, you don't need a load balancer. You can install heartbeat with pacemaker to monitor the system resources such as apache, mysql, etc. The cluster can be configured to keep only one active server.