I am in need of some help with philosophy and design of a continuous integration setup.
Our current CI setup uses buildbot. When I started out designing it, I inherited (well, not strictly, as I was involved in its design a year earlier) a bespoke CI builder that was tailored to run the entire build at once, overnight. After a while, we decided that this was insufficient, and started exploring different CI frameworks, eventually choosing buildbot. One of my goals in transitioning to buildbot (besides getting to enjoy all the whiz-bang extras) was to overcome some of the inadequacies of our bespoke nightly builder.
Humor me for a moment, and let me explain what I have inherited. The codebase for my company is almost 150 unique c++ Windows applications, each of which has dependencies on one or more of a dozen internal libraries (and many on 3rd party libraries as well). Some of these libraries are interdependent, and have depending applications that (while they have nothing to do with each other) have to be built with the same build of that library. Half of these applications and libraries are considered "legacy" and unportable, and must be built with several distinct configurations of the IBM compiler (for which I have written unique subclasses of Compile), and the other half are built with visual studio. The code for each compiler is stored in two separate Visual SourceSafe repositories (which I am simply handling using a bunch of ShellCommands, as there is no support for VSS).
Our original nightly builder simply took down the source for everything, and built stuff in a certain order. There was no way to build only a single application, or pick a revision, or to group things. It would launched virtual machines to build a number of the applications. It wasn't very robust, it wasn't distributable. It wasn't terribly extensible. I wanted to be able to overcame all of these limitations in buildbot.
The way I did this originally was to create entries for each of the applications we wanted to build (all 150ish of them), then create triggered schedulers that could build various applications as groups, and then subsume those groups under an overall nightly build scheduler. These could run on dedicated slaves (no more virtual machine chicanery), and if I wanted I could simply add new slaves. Now, if we want to do a full build out of schedule, it's one click, but we can also build just one application should we so desire.
There are four weaknesses of this approach, however. One is our source tree's complex web of dependencies. In order to simplify config maintenace, all builders are generated from a large dictionary. The dependencies are retrieved and built in a not-terribly robust fashion (namely, keying off of certain things in my build-target dictionary). The second is that each build has between 15 and 21 build steps, which is hard to browse and look at in the web interface, and since there are around 150 columns, takes forever to load (think from 30 seconds to multiple minutes). Thirdly, we no longer have autodiscovery of build targets (although, as much as one of my coworkers harps on me about this, I don't see what it got us in the first place). Finally, aformentioned coworker likes to constantly bring up the fact that we can no longer perform a full build on our local machine (though I never saw what that got us, either, considering that it took three times as long as the distributed build; I think he is just paranoically phobic of ever breaking the build).
Now, moving to new development, we are starting to use g++ and subversion (not porting the old repository, mind you – just for the new stuff). Also, we are starting to do more unit testing ("more" might give the wrong picture… it's more like any), and integration testing (using python). I'm having a hard time figuring out how to fit these into my existing configuration.
So, where have I gone wrong philosophically here? How can I best proceed forward (with buildbot – it's the only piece of the puzzle I have license to work on) so that my configuration is actually maintainable? How do I address some of my design's weaknesses? What really works in terms of CI strategies for large, (possibly over-)complex codebases?
EDIT:
I thought I explained my problem, but obviously I wasn't clear enough. I am not looking for suggestions for changing CI platforms. It is not going to happen, and answers suggesting that will not get accepted. What I want to know is how other people manage complicated codebases using CI. I have a dozen squared different products, and I have dependencies scattered to the wind, and they're all different. THIS is what I want to know how to deal with.
Best Answer
Although I haven't faced a situation as bad as you describe, I have been maintaining a CI configuration with tens of components, between which there are some 'simple' dependencies. I hope that my approach may give you some hints with which to proceed. The problem is definitely not related only to the choice of CI server, but also to the overall build process and project structure.
I shall break down the problem into 2 parts: Building and CI.
Building
For "Building" I means the process of, changing the source code on hand to the final artifact. Our company mainly uses Java in development, and the build tool we used is Maven. You will probably unable to use that due to nature of your project, but there are some valuable concept in Maven worth noting:
1) In Maven world, each artifact (a lib, the real program etc) needs to be clearly isolated and decoupled. Dependencies between artifact should be clear. I should emphasize, messy dependency, especially circular dependencies between your build artifacts are going to make the build process a mess.
For example, I saw some java projects before that, although after whole build process there are several JARs (you may consider it as lib/dll in Java) built, they are in fact inter-dependent. Like, A.jar is using things in B.jar, and vice versa. Such kind of 'modularization' is totally meaningless. A.jar and B.jar always needs to be deployed and used together. The implication is, later if you want to make them separated into different projects (for reuse in other projects for example), u will not be able to do so, coz you cannot determine which project, A or B, to build first.
Yes, it needs to be catered in your software's design, but I always believe that, instead of paying time to make a complicated building tools/model works in a messy project, I'd rather spend time to reorganize the design to make use a simple building model.
2) Dependencies should be declarative. There are lots of build process I saw before, which they includes all libs it needs locally in the project. It will make build process extremely troublesome if some of the libs are in fact other artifacts that you need to build.
3) "Centralized" storage for the artifacts for getting dependencies, or to deploy artifacts once it is compiled. It doesn't need to be "centralized" for whole office (it will be great if it is), just a local directory will already be fine.
More elaboration on 2 and 3. Just to give an example, I have come across a project that involved 3 independent projects. The each project is built base on the source, plus the libs under the the lib/ directory in the source directory. Project A will build several libs, which is in turn used by project B and C. There are quite some drawbacks: build procedure is complicated and harder to automate; the source control is getting bloated with unnecessary duplicated JARs reused in different project
In Maven's world, what it is done is, project B and C do not really contains A.jar (and other dependencies, like other 3rd party lib) in the project source. It is declarative. For example, in the build configuration of project B, it simply declare that it need: A.lib v1.0, xyz.lib v2.1 etc, and the build script will look up the lib from /A/1.0/A.jar and /xyz/2.1/xyz.lib
For non-Maven world, this artifact dir just need to be one or two directory with agreed directory structure. You may put all 3rd party libs to a share location and let developers sync or copy to their local machines. In my C++ projects I did many years before, what I am doing is setting the lib and header to ${artifact_dir}/lib_name/ver, and with artifact_id declared as environment variable.
When project A is built, it will have a copy of its result in that artifact_dir, so that when we build project B, B can get the result of A automatically, without manual copying.
4) Non mutable release. Once you release A.lib 1.0, that's it, you won't expect A.lib 1.0's content change after 1 month just bcos there are some bug fixes. In such case, it should be A.lib 1.1. The artifact of a changing code base should consider a special "version", for which in Maven we call it snapshot.
Non-mutable release is more an ethic issue. But what it solved is clear: When u have lots of projects, using the same lib, u know which version of that lib you are using, and you will be sure that, same version of lib used in different projects, are indeed the same. I think many of us have go thru the problem of: Why project X and Y is both using lib A but the build result is different? (and turns out that the lib A using by X and Y is in fact different after digging into the content or file size of the lib).
All these ensure that your projects can build independently, without much manual tricks like, build project A first, copy A.lib to project B, and then build project B...
After going through an exercise like this, for example, when you build project A, it will try to get the dependencies from the centralized artifact storage. In case some dependencies (which are other projects of your company, e.g. project B) is not found, what u need to do is, get the source of project B, build it (which will deploy to the centralized storage upon success), and then build project A again.
CI
With a easy build system, with clear dependencies, CI will be much easier. I will expect your CI server serves the following requirement:
1) Monitor the source control, only checkout+build when there is changes in source
2) Able to setup dependencies between projects
With a clear dependency for projects, you simply need to setup projects in CI according to your actual projects, setup CI project dependency according to your projects' dependency. Your CI server should be set to build dependent projects before building any project (and of course, build only happens if the project source really have change).
If everything goes right, you should have a huge, complex, but still manageable CI (and even more important, a manageable project structure)