As part of my job I manage a few tens of CentOS 5 servers, using puppet for the main set up. About half of our servers have a standardised set up for hosting various django sites, while the rest are a mish mash of applications.
I'm gradually sorting out our hosting practices, and I've now got to the point of working out how to manage security updates at the OS level. I'm wary of just having a cron job doing a yum -y update but also don't want to have to go round each server in time and review every package with updates available, as that would take a while.
So I'm wondering if there are any good shortcuts or working practices that would minimise the risks involved and minimise the amount of time I need to spend. Or to put it another way are there any tools or practices that can automate a lot of the work while still giving control.
Steps I've decided on so far:
- disable all third party repositories and set up our own repository so I can control what updates go through there.
- we have staging servers for (most of) our production servers where I could do testing (but how much testing is enough testing?)
Also note that I've looked into the yum security plugin but it does not work on CentOS.
So how do you manage updates for significant numbers of CentOS servers running a heterogeneous array of applications?
Best Answer
In most of my environments, it's usually a kickstart and post-install script to get the main system up and current with updates at that moment. I'll usually have a local repo that syncs with a CentOS mirror daily or weekly. I tend to freeze the kernel package at whatever's current as of the installation time and update packages individually or as necessary. Often times, my servers have peripherals that have drivers closely linked to kernel versions, so that's a consideration.
CentOS 5 has matured to the point where constant updates aren't necessary. But also keep in mind that CentOS 5 is winding down. The rate of updates has slowed somewhat, and the nature of the updates is more inline with bugfixes and less about major functionality changes.
So in this specific case, the first thing you could do is build a local mirror/repo. Use your existing configuration management to control access to third-party repos. Maybe schedule policy to yum update critical or public-facing services (ssh, http, ftp, dovecot, etc.) Everything else will require testing, but I get the feeling that most environments don't run with fully-updated/patched systems.