Multiple feature branches and continuous integration

Actually DVCS made continuous integration much easier.

With central VCS, every developer has the rights to commit directly in trunk and therefore he can commit buggy code. CI will detect it after the fact. So it's possible to have broken trunk even with CI.

On the other hand, the basic operations in DVCS world are branching and merging. Because merging is explicit and a separate process vs. commit to the trunk, one can always check the result of a merge before it lands on the trunk. I don't have experience with Git, but developers of Bazaar VCS have used this technique successfully for at least 3.5 years with the help of PQM tool.

Basically PQM workflow looks as following: developer publishes his branch so it can be merged, then he sends a special e-mail to the PQM bot with merge instructions. When PQM receives a merge request, it makes a separate integration branch (copy of trunk), then merges the developer's branch and runs tests on the resulting code. If all tests are passed then the integration branch is pushed to trunk, otherwise the developer will receive an e-mail with the log of failing tests.

Running all tests for Bazaar project takes time, but tests are executed on demand on a separate server. Developers won't be blocked by merges and can continue to work on other tasks.

As result of PQM-based merge workflow the bzr trunk is never broken (at least as long as there are enough acceptance and regression tests).

When you talk about scaling CI you're really talking about scaling the use of your CI server to handle all your feature branches along with your mainline. Initially this looks like a good approach as the developers in a branch get all the advantages of the automated testing that the CI jobs include. However, you run into problems managing the CI server jobs (like you have discovered) and more importantly, you aren't really doing CI. Yes, you are using a CI server, but you aren't continuously integrating the code from all of your developers.

Performing real CI means that all of your developers are committing regularly to the mainline. Easy to say, but the hard part is doing it without breaking your application. I highly recommend you look at Continuous Delivery, especially the Keeping Your Application Releasable section in Chapter 13: Managing Components and Dependencies. The main points are:

• Hide new functionality until it's finished (A.K.A Feature Toggles).
• Make all changes incrementally as a series of small changes, each of which is releasable.
• Use branch by abstraction to make large-scale changes to the codebase.
• Use components to decouple parts of your application that change at different rates.

They are pretty self explanatory except branch by abstraction. This is just a fancy term for:

1. Create an abstraction over the part of the system that you need to change.
2. Refactor the rest of the system to use the abstraction layer.
3. Create a new implementation, which is not part of the production code path until complete.
4. Update your abstraction layer to delegate to your new implementation.
5. Remove the old implementation.
6. Remove the abstraction layer if it is no longer appropriate.

The following paragraph from the Branches, Streams, and Continuous Integration section in Chapter 14: Advanced Version Control summarises the impacts.

The incremental approach certainly requires more discipline and care - and indeed more creativity - than creating a branch and diving gung-ho into re-architecting and developing new functionality. But it significantly reduces the risk of your changes breaking the application, and will save your and your team a great deal of time merging, fixing breakages, and getting your application into a deployable state.

It takes quite a mind shift to give up feature branches and you will always get resistance. In my experience this resistance is based on developers not feeling safe committing code the the mainline and this is a reasonable concern. This in turn usually stems from a lack of knowledge, confidence or experience with the techniques listed above and possibly with the lack of confidence with your automated tests. The former can be solved with training and developer support. The latter is a far more difficult problem to deal with, however branching doesn't provide any extra real safety, it just defers the problem until the developers feel confident enough with their code.