How to redirect database traffic to a new master on RDS after promoting a read replica

amazon-rdsamazon-web-services

A lot of RDS's documentation about read replicas contains a magical step along the lines of "direct database traffic to the new master." For instance, their instructions on implementing failure recovery say:

In the event of a failure you would proceed as follows:

  1. Promote the Read Replica.
  2. Direct database traffic to the new master.
  3. Create a replacement Read Replica.

And their instructions on doing a minimal-downtime MySQL version upgrade end with…

  • Make your MySQL 5.6 Read Replica a master DB instance.

…

  • You now have an upgraded version of your MySQL database. At this point, you can direct your applications to the new MySQL 5.6 DB instance

This talk about directing traffic glosses over what is actually a complicated step, though. If I were using EC2 instances to host my database, I could give them elastic IPs, use the public DNS address of the instance to address it (which resolves to its private IP from inside AWS), and then instantly swap my entire stack to the read replica by reassigning the elastic IP (and thus simultaneously reassigning the public DNS). I used this method happily back in the days when RDS was considered straightforwardly inferior to rolling your own database instance on EC2 by many DBAs. RDS instances still cannot have elastic IPs, though, so I cannot use this particular trick to magically redirect all my database traffic to a new instance when using RDS.

What am I supposed to do instead? Do AWS expect me to simply deploy a config change to every single box in my stack that touches the database?

Best Answer

I wouldn't recommend using the RDS DNS domain in my application config ie

*.rds.amazonaws.com

Set up a private DNS zone in Route53 instead. This is only available within your VPC, so you don't need to register it publicly or anything like that.

Then, set up CNAME records for your DB instances, which point to the RDS domain. Set very low TTLs (< 30s).

Then, configure your app to use the CNAME records.

If you then need to re-direct traffic to a new RDS instance, just update the CNAME record.

Also, if budget permits, you may want to consider using Multi-AZ within RDS. You'll pay twice as much, but its very handy in both scheduled and unscheduled failover situations.

Related Topic