Development Methodologies – Recommended Workflow for Shared Libraries with Mercurial

Spoike's answer is excellent, but there are a few things I think it would be worth adding which are too large for comments.

Branch organisation

With Mercurial you can happily ignore the whole of your first organisational chart. As Spoke says, each repository has it's own set of tags, branches (named and anonymous) and can be organised according to business need.

If bespokeProjectTwo needs a special version of the charting library, then you would branch charting, add the new facilities and use it in bespokeProjectTwo. The new facilities (and their bugs) would not be used by other projects which would reference the standard charting library. If the main charting library had bugs fixed, you could merge those changes into the branch. If other projects also needed these facilities, you could either get those projects to use the special branch, or merge the branch up into the main-line and close the branch.

Also, there is nothing stopping you having a policy to structure branch names to provide specific facilities like your AUTOMATION branches.

Directory organisation

There is no reason why you can't keep your source directory exactly as it is with Mercurial. The only difference is that whereas with Subversion you have a single monolithic (src) repository, with Mercurial you are better off splitting into repositories which are logically grouped. From your source tree structure, I would probably extract out each of the following as individual repositories:

src-+
      +-(developmentAutomation)
      +-libraries-+
      |           +-(log)
      |           +-(statistics)
      |           +-(charting)
      |           +-(distributedComputing)
      |           +-(widgets)
      +-productLines-+
      |              +-(flagshipProduct)
      |              +-(coolNewProduct)
      +-project-+
                +-bigImportantCustomer-+
                |                      +-(bespokeProjectOne)
                |                      +-(bespokeProjectTwo)
                +-anotherImportantCustomer-+
                                           +-(anotherBespokeProject)

This allows any product or bespoke project to use any combination of libraries, at any revision. Have a look at mercurial sub-repositories for an easy way to manage which libraries are used for any given version of a product or project.

Workflow

An alternative to Spoike's suggested workflow (developer pulls from blessed repo, works locally, issues a pull request and finally the integrator pulls those changes & merges them) would be to use the continuous integration system as an intermediary.

As before, the developer pulls from blessed repo and works locally, but when done, they pull from the blessed repo again and merge themselves before pushing to an unblessed repo. Any changes to the unblessed repo are then reviewed (either manually or automatically) and moved to the blessed repo only if they are approved.

This means that the integrator only has accept or reject a change, not do the merge. ^{In my experience it is almost always better for the developer who wrote the code to perform the merge than for someone else to do it.}

As suggested in the mercurial book, hooks can be used to automate this procedure:

When someone pushes a changeset to the server that everyone pulls from, the server will test the changeset before it accepts it as permanent, and reject it if it fails to pass the test suite. If people only pull changes from this filtering server, it will serve to ensure that all changes that people pull have been automatically vetted.

Other issues

The problem of large test datasets can also be solved by putting that test data into a mercurial sub-repository. This will prevent the code repository getting bloated with test data, while still keeping the test data under revision control.

Fail fast is a good principle - the sooner you know the build is broken, the sooner the offending commit can be identified and the build fixed.

Build on every commit is the right thing to do.

Building every 15 minutes can be pointless if the project has a high volume of commits within such a timeframe - tracking down the bad commit would take longer and may be difficult to determine (one might also be in a situation where multiple commits have different things that break the build). There is also the possiblity that in quiet times (night time?) you end up rebuilding though no changes have been made.

If the build breaks so often that it is a problem, the answer it to re-educate the team on the importance of not breaking the build and in techniques that ensure this does not happen (frequent fetches, checkin dance, compiling and running unit tests locally etc...).