Ssl – Change server’s IP with zero downtime

domain-name-systemiptableslive-migrationmigrationssl

I'm changing servers of my website, example.org. The IP of the old server cannot be used for the new one. I want to replace the old server with the new one with zero downtime.

There are many different services running on example.org, it's not just a web server application, it's also a mail server, git server, mumble server, etc.

My idea is:

  1. Get servers' data in sync, so that the new server has data that is an identical copy of the old server
  2. Update the domain name to make example.org point to the new IP
  3. On the old server, set up iptables rules to redirect all traffic for http, https, smtp, etc. to the new server. I'm not sure which IP rules to use for this though, so could use some help with that.

Now, this sounds good and all, but I feel like there will be an issue with SSL.

For example, if a user Alice navigates to example.org in her browser and it resolves to the old IP, since the DNS change hasn't propagated yet for her, and then the old IP redirects her to the new IP, I think her browser would freak out. It would see new IP using SSL cert for example.org, while example.org obviously resolves to a different IP, the old IP, since the DNS cache is not updated for Alice. So the Alice would be greeted with a huge red SSL warning in her browser saying that the new IP is not of example.org.

Similar problem will happen with mail server (SSL smtp) and other services running on example.org. How do I solve this?

An ideal solution would be to somehow use iptables such that the old server would proxy the new server, instead of redirecting the user to the new server. That way users whose DNS cache is not updated yet would communicate like this: [user] <—> [old server] <—> [new server]. They wouldn't know the new server even exists, for them it would looks like their usual communication with the old server. My only issue with this proxy solution is that the new server will see source IPs of the old server, instead of users'. This might break a lot of things, for example Fail2Ban might firewall the old server's IP for 10 minutes because some users entered incorrect mail password a few times, essentially denying access the mail server to all other users who don't have DNS updated and thus use the proxy.

Best Answer

Easiest way to do this doesn't involve proxying through the old server:

  1. Lower TTL to be something like 300 seconds.
  2. Wait for at least old TTL for caches to expire.
  3. Change the IP when there's less traffic.
  4. Modify the TTL back to its original value.

Your maximum downtime is now extremely low.

For your concerns:

  • SSL / TLS don't care about the IP addresses but hostnames alone. It's even possible to have a multi-IP cluster serving the same site with the same certificate. No problem with any browser.
  • Fail2ban has whitelisting: use ignoreip = to make an exception for your old server.
  • You don't need to replicate SQL in real time for this scenario. Move your database in advance and make sites on the old server to use SQL from the new one.
  • Don't proxy SMTP. You can relay all mail through the old server to the new one with your MTA even before the DNS change. Just configure the old server like it was a secondary MX and make the new server to trust it.
Related Topic