Security – Most secure webserver software

• an offsite reverse proxy can serve up a static page for you when your site cannot be reached
• Some 3rd party DNS providers can failover to a second IP address if the first one cannot be reached (this failover can take some time though)
• Setting up an offsite server with a heartbeat/ldirectord could either load balance between your local server and the external one, or just switch to the external one if the internal one is unavailable. This is the method you would use to keep end users from noticing that your site is down. (just make sure they use the same content / database sources)

We use a series of virtual machines, at least one for each live client server, that is kept in sync with the live machines (aside from some of the data being munged so we are not working with personally identifying or otherwise sensitive information).

Any series of changes that are due to go to a production machine are put together in a patch which consists of any files/scripts needed and a script (a shell script usually for a Unix-a-like environment, a batch file or vbs script or powershell script under Windows) that applies those files/scripts appropriately. We then take the relevant VM, update its database(s) from a copy of production if the VM's copy is getting too out-of-date (remembering to re-run the relevant SQL scripts for wiping or randomising sensitive data) and take a snapshot of it in this state (snapshots are supported by virtualbox and most vmware products including the free "vmware server", and probably most other virtualisation solutions too). We then apply the patch as we would to the production server by running the main control script. If there are errors applying the changes, the VM is rolled-back to the snapshot (which takes a matter of at most a few tens of seconds) so the patch can be updated and tried again. This repeats as needed until the patch applies cleanly. Once that is done some testers are asked to give it all a try to make sure the new stuff is working and other old stuff has not been broken (how long you spend on this depends on the scope and severity of the changes and your level of paranoia). If problems are found the roll-back, edit , and re-apply cycle is repeated until all seems well. Once everything seems fine, assuming time allows, one last rollback-patch-and-test cycle is run just-in-case. Once all this is done you hopefully have a patch that can be applied to the production environment by running a single script with appropriate parameters (i.e. relevant passwords, as authentication credentials on the testing VMs should differ from those on the production machines) and you can be pretty confident that it will apply cleanly and have the desired effect with no (or as few as possible) undesired ones.

Always take fresh backups prior to applying anything significant to a production system, no matter how much time you've spent testing the update, just in case, and always plan for at least a little downtime during which you can keep the other users out and give the resulting updated system(s) a further paranoia test (and roll them back to the latest backup if something went badly wrong - if your production environment is virtualised then the snapshot facility can be useful here too).

As a general process this can work on any environment.