Unit Testing – How to Check Code Coverage Automatically

test-coverageunit testing

I am in the process of setting up a Bamboo server for some new projects for a push to TDD in a CI/CD workflow. Sure, unit testing is great, but only as log as it is there.

Now this might be better off in a GIT pre-recieve hook on certain branches (i.e: develop and main release branches) but how should code coverage be enforced, if at all. I am happy to trust committers to ensure code is covered, but how are these things upheld without any slippage apart from diligence and consistency?

In short, I would like to see how others enforce test coverage as an automatic process during either commit or build stages.

Best Answer

You should't enforce code coverage automatically.

This is like enforcing the maximum lines of code per method: agreed, most methods should be less than, say, 20 LOC, but there are valid cases where methods would be longer than that.

In the same way, targeting a given percentage of code coverage per class may lead to unwanted consequences. For instance:

Boilerplate code classes or classes created by code generators may not be tested at all. Forcing developers to test them won't have any benefit and will have a substantial cost in terms of time spent doing it.
Simple code handling unimportant parts of the application doesn't necessarily have to be tested.
In some languages, some code cannot be tested. I had this case in C# with anonymous methods on a library where I really wanted to have 100% code coverage. Those cases may be demoralizing for developers.

More importantly, code coverage should be proportional to two aspects of the code: how critical and how complicated it is:

A piece of code with a complicated logic which is a part of the major feature of an application would better be tested thoughtfully, because failures or regressions can have important consequences.
A simple piece of code which handles a feature nobody uses may have basic tests which cover only basic cases.

Of course, you can still use code coverage as a measurement, especially to compare how different teams achieve code coverage: there may be teams which are less disciplined and more reluctant when it comes to testing. In those cases, you may want to combine this metric with others, such as the number of bugs, the time spent resolving bugs or the number of remarks during code reviews.

You may also want to enforce at least some code coverage, say 60%¹, on individual projects where it makes sense (be careful to exclude prototypes, generated code, CRUD, etc.) Making it possible for developers to mark specific classes as excluded from code coverage is also nice². In this case, this may be done under a form of a check which fails a build if code coverage is below the required minimum. I would do it at build stage, not commit stage, since you are not expected to run unit tests during commit.

^{¹ I would consider 60% as a reasonable minimum based on my code base: nearly every project or class which has less than 60% code coverage is really untested. This may vary a lot from a language to another and from a company to another (in some companies, 0% is a standard). Make sure to discuss with your team what is normal and what is too high for them. Maybe they are constantly hitting 95% and can easily target 99%, or maybe they struggle to increase their code coverage from 70% to 75%.}

^{² Given that eventual abuse will be detected during code reviews, you shouldn't be afraid to give this possibility to developers. This is similar to the possibility to exclude some parts of the code from the checks by the linters or style checkers. JSLint, StyleCop and Code Analysis are three examples where exclusion is supported and is actually useful without encouraging abuse.}

Related Solutions

Unit-testing – How to automatically test a time check

One simple way is to pass in the time from outside the method:

doSomething(long currentTime) {
   //...stuff
   startTime = currentTime;
   //... more stuff
}

Your production code that calls the method can pass in the real value for System.currentTimeMillis(), whereas your unit test(s) can pass in a specific known value.

Of course, this isn't quite the same as your original. Another possibility that is a little more complex but should mimic the behaviour of your of your original more closely:

doSomething(ITimeGetter timeGetter) {
   //...stuff
   startTime = timeGetter.GetCurrentTime;
   //... more stuff
}

Define the interface:

interface ITimeGetter
{
   long GetCurrentTime();
}

Two implementations, one for the production code:

class RealTimeGetter implements ITimeGetter
{
  long GetCurrentTime()
  {
      return System.currentTimeMillis();
  }
}

One for the unit tests:

class TimeGetterForTests implements ITimeGetter
{
  long GetCurrentTime()
  {
     // return a known value.
  }
}

Instantiate an ITimeGetter appropriate to your scenario and pass it in to the method you want to test.

Javascript – How to handle measuring Code Coverage in JavaScript

Read this on code-coverage tools for JavaScript, which has some links, though most are indeed browser extensions.

However, do consider JsTestDriver, which has code coverage support and allows for testing outside of the browser by connecting with another test tool (like HtmlUnit).

Best Answer

Related Solutions

Unit-testing – How to automatically test a time check

Javascript – How to handle measuring Code Coverage in JavaScript

Related Topic