Internal Libraries (Subversion Externals, ‘library’ branch, or just another folder)

svn

Currently working on multiple projects that need to share internal libraries. The internal libraries are updated continually. Currently only 1 project needs to be stable but soon we will need to have both projects stable at any given time.

What is the best way to SVN internal libraries?

Currently we are using the 'just another folder' like so…

trunk\project1
trunk\project2
trunk\libs

It causes a major headache when a shared library is updated for project1 and project2 is now dead until the parts that use the library are updated.

So after doing some research on SVN externals I thought of this…

trunk\project1\libs (external to trunk\libs @ some revision)
trunk\project2\libs (external to trunk\libs @ different revision)
trunk\libs\

I'm a little worried about how externals work with commits and not making library commits so complicated that I am the only one capable of doing it (mostly worried about branches with externals as we use them extensively).

On top of that we have multiple programming languages within each project some of which don't support per-project library directories (at least not easily) so we would need to check out on a per project basis instead of checking out the trunk.

There is also the 'vendor' style branching of libraries but it has the same problem as above where the library would have to be a sub folder of each project and is maybe a little to complicated for how little projects we have.

Any insight would be nice. I've spent quite a bit of time reading the Subversion book and feeling like I'm getting no where.

Best Answer

When dealing with multiple projects, I usually switch common libraries code from directly editable to versionned binaries. This way, I can have releases of this libraries, and it let me choose between updating or not updating the projects that depend on it. You generally don't want to force an update on a project if you can avoid it, and breaking a build is certainly the worst way to force an update. Stability comes to the price of relatively quick edition. But at the end of the day the time you loose managing your library is far less than the time and energy you would loose handling unhappy programmers with their builds broken.

This method implies that the common code is relatively stable, and that's where unit testing and even TDD really pays. It's possible to debate about their Return On Investment in small projects, but in my opinion, they really worth it when you're dealing with code that is used across multiple projects, because you cannot just go around and compile/test every project that use it.

Regarding SVN, If you go the binary way, each library should become a distinct project, and each release implies a label, and optionally a branch if maintenance is preferable to update. Avoid the monster library that rule them all. You would have to update it every time there's a minor change, making it impossible for projects to follow the versions. That's where it takes a bit of architecture, to organize your libraries, if they have dependencies on each others.

As for the binaries, some people upload them in SVN, some people use scripts to download them from a network repository. It really depends on the binaries size, the update frequences, your network architecture, and your personnal preferences.

That's my experience with this specific issue, but I mainly worked in C/C++ and .Net, so there might be other ways to combine editability and stability that I'm not aware of :)

Related Solutions

Version Control – Creating a Version Control Strategy for SVN

If you want a unified build process, then be sure to put branches/tags/trunk at the root, like this:

branches/
tags/
trunk/
  dev/
    ...

If you don't need a unified build process, then you can put branches/tags/trunks within each project if you want. However, it might be difficult to migrate to a unified build after having put them within each project. A unified build has advantages, such as eliminating the need to publish shared components among projects -- they're all part of the build.

Personally, I like a unified build process. Furthermore, I don't think you should have a "dev" project. You should have projects directly under trunk, and then branch trunk into a dev branch. Use tags for releases. For example, I would do it like this:

branches/
  dev/
    Site1/
    Site2/
    WebService/
    SharedCode/
tags/
  release1/
    Site1/
    Site2/
    WebService/
    SharedCode/
trunk/
  Site1/
  Site2/
  WebService/
  SharedCode/

SVN Tagging – Tags or Specify Revision?

It depends on how cleanly all these elements are separated.

In order to put clarification on how exactly i see things - i am trying to put general

As you highlighted you can deal with this in two ways -

A. Single Product Approach
The entire space is one projects and while they are fairly independent - the repo doesn't recognize it.

In this case few things can be observed.

The same release number of any RC is applied to all object products. So if the repo moves from 1.2 to 1.3 via 1.2.1 and 1.2.3 - there is a corresponding revision of each element i.e. project1.1.2.1.so as well as project2.1.2.1.so.
Version number of tags of all projects and application has the same tag.
Between two versions of a particular singular repo, whether the projectA or projectB has any changes or not, the tag keeps advancing as per total progress.
In such case, when you take a branch from any point in trunk, the entire trunk must be mirrored.
Assuming that testing has been good, all projects and application with same version/tag are always perfectly compatible with each other.

B. Independent Product approach
Here each project is an independent product -

They must have different repo (or at least separate trunks for themselves).
The tagging is limited to the projects individually each part of repo will produce end product like - projectA.1.2.1.so OR projectB.1.5.lib
In such cases - when you branch - the branch is always only for the given project and any tags before/after the merge only applies to those projects.
Each project can have independent number of tags depending on the actual number of bugs within it.
In this case each different project advances with different amount - you always need to find which set of tags - for each other works compatibly with other projects. For example - projectA-1.2 works for projectB-1.2 all the way upto projectB-2.3 but breaks after projectB-2.5 or higher. Such nested dependencies is what you need to now manage through explicitly integration testing.

So what are the pros and cons?

In a simple terms - the single product approach is clumsy because all elements are copied when a branch is created. Now if before committing if someone changes something on another person's module it becomes a conflict and that too after some significant time in future when other person decides to commit/merge. This problem essentially grows as the number of people grow in the team.
One of the primary reason why single product approach helps is the simplicity. Ideally if you have a single authority for release management - every full release is not only tested to see that pieces work individually but also that all versions work together. The integration testing is always more painful when independent products grow and you need to keep a tab on multiple versions of each different products that works together and that is the job by itself.
Support cost increases in case of multiple product situation. This is understood in this way - suppose projectB-1.2 and projectB-1.7 depends on different compatible versions of projectA-0.8 and projectA-0.9. Now, if the projectA has got a bug which could have been affecting all its' clients (irrespective of compatibility) it has to be fixed twice - both in projectA-0.8 and projectA-0.9. This doubles the effort on bug support for projectA.
Further suppose if implementations in projectA-0.8 is found to be a bug (say already solved in projectA-0.9) but you realize that solution to this bug under 0.8 will help part of its clients, but break others; in this case projectA-0.8 needs to be further branched out separately to see that now both groups are supported.
Last but most important - when people work in independent systems the communication between them reduces. They establish their work based on their individual tests; later on when the application folk needs to put in all the modules together, they have to put effort to bring pieces together, you realize a lot of issues of integration and further, if the responsibility of the bugs are not appropriately fixed based on root cause, system will tend to accumulate tech debt over time.

In a nutshell the more break-up you do in terms of a projects/modules, more of the nesting dependencies issues arise.

However this complexity is worth it if the size of the repo is too large. In a single repo, everyday people may end-up stuck dealing with conflicts or mergers.

One of the practical solution
In our place this has been a long time debate with all of us.

A simple approach we try to follow is as follows. Initially we all start with a single product approach. As we realize that as APIs become more stable, and certain key objects becomes fairly stable, we remove them slowly to another repo so that it becomes black-box for everyone. This approaches balances the initial need to remain in close communication and integration, and longer term need to keep repo manageable.

Sorry, i wrote too much?

NOTE: From your question - i felt that your current practice and proposed one are both slightly different from how i described; however, i have used the above scenarios since they are well outlined methods so easier to follow.

Best Answer

Related Solutions

Version Control – Creating a Version Control Strategy for SVN

SVN Tagging – Tags or Specify Revision?

Related Topic