Linux – the best practice to keep a linux ubuntu server up to date (build packages, dist-upgrade, alt repos…)

linuxstabilityUbuntuubuntu-9.10update

We are running a production server based on Ubuntu 9.10 Karmic Koala, kernel is almost up-to-date (2.6.38.2-grsec-xxxx-grs-ipv6-64) but karmic package repository is now ridiculously outdated, eg. Nginx is 0.7.62 – really buggy – while latest stable is 1.0.x!

In addition, Karmic just reached its end of life.

This question: Best practices for keeping UNIX packages up to date? looks similar but actually only includes some suggestions about package managers; not at all what I need!

So the options that I see are:

  1. Get a new machine, install it from scratch, migrate
  2. Distribution upgrade
  3. Use a different repository (launchpad/ppa / backport / pinning)
  4. Build your own

The disadvantages of #1 are quite obvious.

I do not dare do a dist-upgrade path though, as downtime and possible catastrophic consequences are just impossible to predict for a production server, and currently am mostly re-building my own required packages. But I'm sure I might be missing some.

It is not really clear to me what the risks are (stability/compatibility) of using Ubuntu backports, in addition, nothing is officially provided for 9.10 anymore.
Launchpad are individual-builds, similar question – how much better is this than compiling my own?

Building packages seems fine, but:

  1. Sometimes I have trouble reproducing the correct ./configure options in order to re-use my existing configuration files
  2. I am sure there are tons of packages and dependencies that are now pretty outdated and possible sources of bugs

Finally… what about "old" packages in a recent distribution? I guess there's no other way than re-building them myself? Is a combination of 2 and 4 finally the best path?

Is there any objective consensus on what is the best way to do this, or reasons why some of my options are fine/not fine?

If really there isn't, I will accept that the question gets closed before creating an endless thread!

Best Answer

Maintaining your own distribution is a lot of work. Even if you maintain the backports, you will soon be overwhelmed by security issues to fix, and have to pull low-level libraries to keep updating your software, which might break other things (I maintain servers running 6-year-old distros, it's not fun).

Upgrading is generally a good solution. do-release-upgrade is well made, and you should be able to upgrade without issues (especially if you only used official packages).

My favourite solution though might be the reinstall path. More specifically, your servers should be managed using a configuration management system such as Puppet, Cfengine or Chef. If all your configuration/package needs are specified using such a tool and your data are safe on a separate partition, it's much easier to reinstall quickly. You just install a new distribution without erasing the data partitions, and then run the configuration management tool to reset your packages/configurations. I believe this is the cleanest way to do, especially if you have several servers to manage.

If you are using non-official packages, you might want to identify them before you upgrade/reinstall. maintenance-check can help you identify the packages that are not officially maintained by Ubuntu:

$ bzr branch lp:ubuntu-maintenance-check
$ cd ubuntu-maintenance-check
$ ./maintenance-check -f n

If you want to reinstall, you can also export the list of installed packages:

$ dpkg --get-selections > myinstall.txt

and your debconf database:

$ debconf-get-selections > debconf.txt # from the debconf-utils package

As a note, since you're currently using Karmic, it might not be too violent to upgrade to Lucid, which is an LTS release, still supported until 2015 for the main server packages. This should leave you enough time to setup a viable automated installation for the future.

When you ask about Launchpad packages, I suppose you mean PPAs. There are tons of different PPAs. Some are experimental, some are stable. Some are maintained by official Ubuntu developers, some are maintained by people hardly know how to do a package properly. It's hard to say in general if packages you find on PPAs are good, there's no general rule. The best hint in this case might be too look at the owner of the PPAs to get an idea of the possible quality of their packages.

Related Topic