Code Organization – How to Organize Larger Projects with Sub-Projects

code-organizationorganization

As a software developer until now, I've mostly worked on projects that were quite "monolithic" with hardly any dependencies on other projects, without building automation (no Make, Ant, Maven, etc.) and kept on a simple version control system (mostly Subversion) with just a few easily managed version branches.

Now together with some friends I'm planning a project that is intended to run on multiple platforms (mostly mobile: Android, iOS, Kindle, Windows, etc.), thus written in several languages and on different development platforms.

This will lead to many dependencies: All projects sharing the same resources (e.g. images) or projects dependent on each other (e.g. a core Java library project used by the Android and other Java based implementations).

So what I need is some basic information on how to answer questions such as: How would the VCS be structured? Would a client-base or a decentralized VCS be better? How to decide building automation system(s) to use?

Since this quite an open question I guess for now it would be great if you could point me to any books or web resources that you can recommend for this topic.

Best Answer

This doesn't answer your actual question, but may be of help.

Consider cross platform tools

There are layers such as PhoneGap, that would allow you develop apps accross multiple platforms in one language.
Also, a cross platform language, such as haXe might be of interest (you can use the C++ backend for native apps, and the JS backend for web and mobile apps (through the aforementioned PhoneGap)).

You can drastically reduce the amount of code you need to write, if you can reuse it on multiple platforms. Having less code in the first place, solves many problems before they even occur.

Focus on a single platform to start with

If you try to write the app for many different platforms at once, the results may become mediocre. Instead, you should rather try to pick a single platform, to focus on. Test the concepts of your app on that platform and then port it to other platforms and adapt it accordingly.

For example, you may realize, that you don't actually want the different ports to share graphics, to better fit with the graphical needs. For example on the iPhone 4, you have a very high resolution, so you might need extra icons for that. Also, the users may have different expectancies.

Porting a working concept to a new platform is relatively straight forward (although sometimes work-intense), in comparison to really developing it in the first place. If you try to do that on multiple platforms at they same time, you may overburden yourself.

So pick a platform, that is sufficiently widely spread and that you feel comfortable with, to get up and running quickly. A user base is a valuable source of feedback, that you'll hardly get anywhere else.

Do not think too big. Start small and grow.

This applies not only to your initial choice of platforms, but also to your projects setup. Adapt your toolchain according to your needs instead of investing time on building up a giant all-purpose environment, that is actually putting a lot of overhead on your development process.

You want to tackle a non-monolithic problem. As a consequence, I think a behemothish, monolithic workflow is maybe not the best thing to do.

Branch organisation

With Mercurial you can happily ignore the whole of your first organisational chart. As Spoke says, each repository has it's own set of tags, branches (named and anonymous) and can be organised according to business need.

If bespokeProjectTwo needs a special version of the charting library, then you would branch charting, add the new facilities and use it in bespokeProjectTwo. The new facilities (and their bugs) would not be used by other projects which would reference the standard charting library. If the main charting library had bugs fixed, you could merge those changes into the branch. If other projects also needed these facilities, you could either get those projects to use the special branch, or merge the branch up into the main-line and close the branch.

Also, there is nothing stopping you having a policy to structure branch names to provide specific facilities like your AUTOMATION branches.

Directory organisation

There is no reason why you can't keep your source directory exactly as it is with Mercurial. The only difference is that whereas with Subversion you have a single monolithic (src) repository, with Mercurial you are better off splitting into repositories which are logically grouped. From your source tree structure, I would probably extract out each of the following as individual repositories:

src-+
      +-(developmentAutomation)
      +-libraries-+
      |           +-(log)
      |           +-(statistics)
      |           +-(charting)
      |           +-(distributedComputing)
      |           +-(widgets)
      +-productLines-+
      |              +-(flagshipProduct)
      |              +-(coolNewProduct)
      +-project-+
                +-bigImportantCustomer-+
                |                      +-(bespokeProjectOne)
                |                      +-(bespokeProjectTwo)
                +-anotherImportantCustomer-+
                                           +-(anotherBespokeProject)

This allows any product or bespoke project to use any combination of libraries, at any revision. Have a look at mercurial sub-repositories for an easy way to manage which libraries are used for any given version of a product or project.

Workflow

An alternative to Spoike's suggested workflow (developer pulls from blessed repo, works locally, issues a pull request and finally the integrator pulls those changes & merges them) would be to use the continuous integration system as an intermediary.

As before, the developer pulls from blessed repo and works locally, but when done, they pull from the blessed repo again and merge themselves before pushing to an unblessed repo. Any changes to the unblessed repo are then reviewed (either manually or automatically) and moved to the blessed repo only if they are approved.

This means that the integrator only has accept or reject a change, not do the merge. ^{In my experience it is almost always better for the developer who wrote the code to perform the merge than for someone else to do it.}

As suggested in the mercurial book, hooks can be used to automate this procedure:

When someone pushes a changeset to the server that everyone pulls from, the server will test the changeset before it accepts it as permanent, and reject it if it fails to pass the test suite. If people only pull changes from this filtering server, it will serve to ensure that all changes that people pull have been automatically vetted.

Other issues

The problem of large test datasets can also be solved by putting that test data into a mercurial sub-repository. This will prevent the code repository getting bloated with test data, while still keeping the test data under revision control.

Git structure for java project

For this kind of work, I suggest you to learn about Maven which deals with dependency compatibility (among a lot of other things).

With Maven, you could for example use one of these two ways:

Either the "standard" way: by creating a Nexus repository which you will feed with a compiled jar of each version of your common package and to fully exploit maven/nexus the version(s) of JDBC you use. Each time you build your project from any commit, the versioned config file will be read and will use the good revision of your library "common".

(I say Nexus, but there are others software doing the same thing, I've also heard of one named Artifactory).

Or, you could make a more complicated script which, when you build your server or client project, go download the source of your library on your "common" repository and build it.

Maven may seems a bit overkill and is (in my opinion) a pain to learn alone, but it's handy, powerful, and is the professional standard in Java and works seamlessly with Eclipse.

As for the Maven usage, the Nexus way is (depending on your scripting skill) a bit heavier to init, but more standard, easy to scale with your project, and can be used for all your other future projects.

If you're working in a professional context, I strongly advise to look for a colleague knowing it to help you mavenize your project.