Unit Testing TDD – Tools to Catch Fake Coverage

However, it implies that large parts of the applications code are not covered by tests. Why? Because if you have units (and you need a lot of units to get your Unit Tests right) you need code that wires the units together. This code, IMHO, will get complicated enough that it deserves to be tested on a more granular level that integration tests while it probably falls into "Dirty Hybris":

Your assumption is faulty because you are neglecting a layer of testing - acceptance testing.

Your unit tests cover individual units - the classes and methods that compose them. This enables you to test methods and classes in isolation to ensure that they are behaving as expected. Above this lies your integration tests, which tests the collaboration between classes and ensures that larger modules (packages and even inter-package collaboration) work as expected. Finally, your acceptance tests are used to verify and validate your entire system, as assembled, against the user requirements.

Assuming that you have the appropriate unit and integration tests that correspond to requirements and well-defined acceptance criteria and acceptance test plans, then everything in your system is tested. Other aspects of testing - smoke tests, regression tests, and so forth, are simply an appropriate subsampling of the unit, integration, and acceptance tests.

TDD is a robust way of designing software components (“units”) interactively so that their behaviour is specified through unit tests

That particular quote is also missing something. As I was taught, TDD isn't just about unit tests, but developing all tests first. That includes not only unit tests, but the necessary acceptance and integration tests as well.

You should't enforce code coverage automatically.

This is like enforcing the maximum lines of code per method: agreed, most methods should be less than, say, 20 LOC, but there are valid cases where methods would be longer than that.

In the same way, targeting a given percentage of code coverage per class may lead to unwanted consequences. For instance:

• Boilerplate code classes or classes created by code generators may not be tested at all. Forcing developers to test them won't have any benefit and will have a substantial cost in terms of time spent doing it.

• Simple code handling unimportant parts of the application doesn't necessarily have to be tested.

• In some languages, some code cannot be tested. I had this case in C# with anonymous methods on a library where I really wanted to have 100% code coverage. Those cases may be demoralizing for developers.

More importantly, code coverage should be proportional to two aspects of the code: how critical and how complicated it is:

• A piece of code with a complicated logic which is a part of the major feature of an application would better be tested thoughtfully, because failures or regressions can have important consequences.

• A simple piece of code which handles a feature nobody uses may have basic tests which cover only basic cases.

Of course, you can still use code coverage as a measurement, especially to compare how different teams achieve code coverage: there may be teams which are less disciplined and more reluctant when it comes to testing. In those cases, you may want to combine this metric with others, such as the number of bugs, the time spent resolving bugs or the number of remarks during code reviews.

You may also want to enforce at least some code coverage, say 60%¹, on individual projects where it makes sense (be careful to exclude prototypes, generated code, CRUD, etc.) Making it possible for developers to mark specific classes as excluded from code coverage is also nice². In this case, this may be done under a form of a check which fails a build if code coverage is below the required minimum. I would do it at build stage, not commit stage, since you are not expected to run unit tests during commit.

^{¹ I would consider 60% as a reasonable minimum based on my code base: nearly every project or class which has less than 60% code coverage is really untested. This may vary a lot from a language to another and from a company to another (in some companies, 0% is a standard). Make sure to discuss with your team what is normal and what is too high for them. Maybe they are constantly hitting 95% and can easily target 99%, or maybe they struggle to increase their code coverage from 70% to 75%.}

^{² Given that eventual abuse will be detected during code reviews, you shouldn't be afraid to give this possibility to developers. This is similar to the possibility to exclude some parts of the code from the checks by the linters or style checkers. JSLint, StyleCop and Code Analysis are three examples where exclusion is supported and is actually useful without encouraging abuse.}