Continuous Integration System – Effective Monitoring

buildscontinuous integration

One of my roles in my team is the build person. I am responsible for maintaining/updating our build scripts and making sure we are building 'smoothly' on the continuous integration server. I usually do not mind this job, though often it feels like I am constantly babysitting the CI server.

This job can be annoying at times because if the build breaks I have to drop the story I am working on and investigate the build failure. Build failures happen daily on our team. Sometimes developers simply do not build locally before committing so tests fail on the CI server. In this situation I like to quickly get to the person who had the 'bad commit' so the build does not stay broken too long. Sometimes (a lot less frequently) a strange condition exists on the CI server that needs to be debugged.

I know that many mature teams use Continuous Integration but there is not a lot of material out there about good practices.

Do my problems point out that our continuous integration is not very mature or is this simply part of the job?

What are some good practices to follow? What are the characteristics of mature continuous integration?

Update

Instead of answering some comments I am going to make an update instead. We have a single, simple command that does exactly what the build server will do when building the app. It will compile, run all unit/integration and some quick UI based tests.

Reading everyone's answers, it feels we might have two major problems.

  1. CI Server not complaining loud enough when a build fails.
  2. Developers do not feel like its everyone's responsibility to make sure their commit goes through successfully.

What makes things harder in my team is that we have a large team (10+ developers) AND we have a couple of off-shore team members committing when we are not even at work. Because the team is large and we established that frequent, small commits are preferred, we sometimes have really a lot of activity in a day.

Best Answer

First and foremost: each person is responsible for the build process. It sounds like members in your team are not mature... No one gets away with writing code and fobbing it off to the CI server hoping that it works. Before committing code, it should be tested on their local machine. You should be sure that the code you're checking in isn't going to break the build. Of course, there are cases when the build break unintentionally (e.g. if a config file has been changed or sloppy commit was inadvertently made).

Most CI servers (I've only used Hudson) will send an automated email detailing the commits made that caused the build to break. The only part of your role is to stand behind them looking tough until the suspect fixes whatever they broke.