Linux – How to apply development practices like version control, testing and continuous integration/deployment to system administration

backuplinuxtestingversion controlwindows

Imagine you're going to manage a number of servers with a number of different services that's used by a number of people. Now say you want to reconfigure or replace some software on one of those servers. Obviously you don't want to work on servers that are in production.

If this was a code change, as a developer, I would make the change on my local development machine, test it locally and commit the change to a version control system. The changes could then be deployed in a staging environment, tested further and finally deployed in a production environment. It would also be easy for me to roll back, if necessary.

Generally, or specifically, how do you achieve this in system administration?

(The first thing that comes to mind is to use virtual machines and put virtual machine images in version control, but I'm sure there is a lot of literature and clever solutions I'm not presently aware of.)

Best Answer

Short answer is "OS deployment management", "Configuration Management" and "Software Packaging". Long answer follows.

I'd like to add to Daniel Pittman's reply with a breakdown of what forms a "system" in systems administration.

A system or an environment is going to comprise of:

  • Servers
  • Operating System
  • Configuration
  • Vendor packages; and
  • Local packages

Enveloping these will be processes such as:

  • OS deployment or imaging
  • Configuration management
  • Software package management
  • Auditing/logging
  • Monitoring
  • Backups

And you'd like to have these combine together to assist you with achieving non-functional goals such as:

  • Repeatability
  • Maintainability
  • Measurability
  • Performance
  • Trackability
  • Testability
  • Changeability

This is a quick brain dump. I am sure more could be added to all lists.

Your question touches on a number of these without using the specific words. For example, you want to be able to deploy easily and revert back i.e. want maintainability; you want to do it in a test environment and test till it passes i.e. repeatability, testability and measurability; you're thinking of putting vm images in version control because you'd like repeatability of OS and configuration deployments.

There are plenty of tools to assist you with this, some of them mentioned by Daniel. Some others are:

  • Kickstarts (RedHat based), Preseed (Debian based), WDS (MS Windows) for deploying known OS environments
  • Spacewalk/Satellite (RedHat based), Group Policies (MS Windows) for configuration and package management
  • YUM and APT packaging systems for generating, deploying, upgrading and removing packages (sets of binaries, data and configuration comprising a piece of software)
  • Nagios, OpenNMS and SCOM for monitoring
  • Amanda, Bacula and Windows Backup Server for backups
  • Munin, PCP and Hyperic for performance monitoring
  • CVS, SVN, GIT or Bazaar for Version Control
  • Hudson and Jenkins for build management
  • Selenium and Robot for testing
  • Bugzilla, Request Tracker and Jira for recording, communication and tracking

Again, this isn't a comprehensive list, but something that I keep in my head to guide me and hopefully it'll help you out too.

Related Topic