We release software on RPM-based linux. We receive many packages from different teams, some outside of our business unit, that must upgrade correctly (in the future we want to check dependencies are correct). Recently we were bitten by a bug in upgrading from a package delivered from outside our business unit. The time has come to start attempting to automate testing for this sort of thing.
The sort of thing I had in mind was assuming the install is good (this would generally be caught at functional test time anyway), then check that an upgrade with pre/post scripts modifies files in the same manner as the install. If not then flag for review.
Specifically we want to check upgrades, corner cases like kernel upgrades, chroot installs where the running kernel may not even be available in the chroot, etc.
My question is, does software that does this sort of thing already exist? I haven't had much luck in my searching.
This automated process would also be backed up with packaging guidelines of course, but as some of our RPMs come from outside our organisation, we would still like this fallback to give us assurances in the future.
Best Answer
You have asked rather a general question I think so there may be a variety of possible answers. One option I would recommend from personal experience is Jenkins.
Jenkins is a good continuous integration tool-set. https://wiki.jenkins-ci.org/display/JENKINS/Meet+Jenkins It's quite a flexible tool and you can get a good range of different plugins. The install/setup is quite easy. You can install on own machine linux/windows/... It gives you power and control over your automated jobs.
You can use Jenkins for quite a range of different tasks you wish to automate. * regular builds and test runs (good selection of build error/management and build/test run report plugins) * making regular backups, running regular or triggered tests * checking source code updates and keeping work areas up to date / clean * using ssh plugin to run automated tasks on external machines
In the past I have used it for project-wide jobs and for individual developer tasks.
You would automate your test setup, rpm install and then test as a script or set of scripts. Jenkins would allow you to manage the jobs using web interface. Job status and job/test reports browsable using web interface. Errors or reports sent on in emails or whatever.
Further reading . . .
Continuous Delivery: http://en.wikipedia.org/wiki/Continuous_deployment http://www.martinfowler.com/articles/continuousIntegration.html I have used cruise-control and buildbot in the past but I find they are more limited than Jankins.
This blog entry seems quite relevant to your question: http://www.devops-blog.net/koji/installing-a-rpm-based-build-system Jenkins + a range of other tools