C++ – How should C++ Unit Test code be organized for maximum Unit Test efficiency

cprojects-and-solutionstddunit testing

  • This question is not about Unit Testing Frameworks.
  • This question is not about writing Unit Tests.
  • This question is about where to put the UT code written and how/when/where to compile and run it.

In Working Effectively with Legacy Code, Michael Feathers asserts that

good unit tests … run fast

and that

A unit test that takes 1/10th of a second to run is a slow unit test.

I think these definitions do make sense. I also think that they imply that you have to keep a set of Unit Tests and a set of Those Code Tests That Take Longer separately, but I guess that's the price you pay for only calling something a Unit Test if it runs (very) fast.

Obviously the problem in C++ is that to "run" your Unit Test(s), you have to:

  1. Edit your code (production or Unit Test, depending on which "cycle" you're in)
  2. Compile
  3. Link
  4. Start Unit Test Executable(s)

Edit (after weird close vote): Before going into the details, I'll try summarize the point here:

How can C++ Unit Test code be effectively organized, so that it's both efficient to edit the (test) code and to run the test code?


The first problem then is to decide where to put the Unit Test code so that:

  • it's "natural" to edit and view it in combination with the associated production code.
  • it's easy/quick to start the compilation cycle for the unit your currently changing

The second, related, problem then is what to compile so that the feedback is instantaneous.

Extreme options:

  • Each Unit-Test-Test-Unit lives in a separate cpp file and this cpp file is compiled+linked separately (together with the source code unit file it tests) to a single executable which then runs this one Unit Test.
    • (+) This minimizes startup (compile+link!) time for the single Test Unit.
    • (+) The test runs super fast, because it only tests one unit.
    • (-) Executing the whole suite will need to start a bazillion of processes. Can be a problem to manage.
    • (-) The overhead of process starts will become visible
  • The other side would be to have — still — one cpp file per test, but all the test cpp files (together with the code they test!) are linked into one executable (per module / per project / pick your choice).
    • (+) Compile time still would be OK, as only changed code will compile.
    • (+) Executing the whole suite is easy, as there's just one exe to run.
    • (-) The suite will take ages to link, as each recompilation of any object will trigger a re-link.
    • (-)(?) The suit will take longer to run, although if all Unit Tests are fast, the time should be OK.

So, how are real world C++ Unit Tests handled? If I only run that stuff nightly/hourly, the second part doesn't really matter, but the first part, namely how to "couple" the UT code to the production code, so that it's "natural" for developers to keep both in focus always matters I think. (And if developers have the UT code in focus, they'll want to run it, which brings us back to part two.)

Real world stories and experience appreciated!

Notes:

  • This question intentionally leaves unspecified platform and make/project system.
  • Questions Tagged UT & C++ is a great place to start, but unfortunately too many questions, and especially answers, are too heavily focused on the details or on specific frameworks.
  • A while ago, I answered a similar question on structure for boost unit tests. I find this structure to be lacking for "real", fast Unit Tests. And I find the other question too narrow, hence this new question.

Best Answer

We have all unit tests (for a module) in one executable. The tests are put into groups. I can execute a single test (or some tests) or a group of tests by specifying a (test/group) name on the command line of the test runner. The build system can run group "Build", the test department can run "All". The developer can put some tests into a group like "BUG1234" with 1234 being the issue tracker number of the case he's working on.