Say I have application servers, database servers, and a few dns-round-robin load balancers. All this powered by images created with Packer with deployment managed with Terraform. How do I change the image of the database servers without nuking their data when the instances get destroyed and recreated?
The simplest thing I can think of would be to turn off writes, snapshot the database, and then restore the snapshot to the new servers. But it feels really wrong to rely on manual fiddling like that, and it also feels wrong to take the service down for a simple upgrade. There is a cleaner and better way, right?
Best Answer
There's no simple answer to this question.
Using an architecture designed around images (commonly referred to as "immutable infrastructure") works fantastically for stateless services, like your application servers.
It's most definitely possible to expand that to your stateful services with the right tools, failover systems and upgrade paths, but those are usually overkill for simple systems (as the one you describe).
One thing to keep in mind when using these tools is that you don't have to go "all in". Packer and Terraform are very much designed to work only where you want them. They don't enforce a pattern across all of your systems on purpose.
Practically speaking, the best way to handle this problem is maintain your database servers differently, outside of Packer (building the initial image, yes! But not necessarily upgrading them in the same way as the stateless web servers) or outsource managing the state to someone else. Notable options include Heroku Postgres or AWS RDS.
To round it out – yes, it's possible, but with our current tooling it's probably more trouble then it's worth at smaller scale or with simple architectures.
Packer and Terraform can still be a huge boon in other aspects of the very same infrastructure – Terraform, for example, could provision a Heroku database for use in your DigitalOcean Application servers in a very straight forward way. Packer can handle upgrading and releasing your application server images, and likewise for development.