Unexpected Code Coverage Reduction – Causes and Solutions

continuous integrationrefactoringtest-automationtest-coverageunit testing

I am adding unit tests to some existing code which initially was not tested at all.

I have set up my CI system to fail builds where the code coverage percentage is reduced compared to previous build – mainly to set a path of continuing improvement.

I then encountered an unexpected and annoying (although mathematically correct) situation that can be reduced to the following:

  • Before refactor:
    Method 1: 100 lines of code, 10 covered –> 10% coverage
    Method 2: 20 lines of code, 15 covered –> 75% coverage

    Total: 25 / 120 –> ~20% coverage

  • After refactor:
    Method 1: 100 lines of code, 10 covered –> 10% coverage (untouched)
    Method 2: 5 lines of code, 5 covered –> 100% coverage

    Total: 15 / 105 –> ~14% coverage

So even though IMO the situation was improved, my CI system obviously does not agree.

Admittedly, this is a very esoteric problem, and would probably disappear when the bulk of the code will be covered better, but I would appreciate insights and ideas (or tools/configurations) that might allow me to keep enforcing an "improvement" path for coverage with my CI system.

The environment is Java, Jenkins, JaCoCo.

Best Answer

You can mitigate the effect to some degree by allowing the relative code coverage to reduce when the total number of uncovered lines also reduces, or when the total number of lines reduces, since this are pretty clear signs of a refactoring which sets a new base line for your coverage metrics.

In your example, the total number of uncovered lines reduces from 95 lines to 90 lines, and the total number of lines from 120 to 105. That should give you enough confidence that the relative coverage is quite unimportant in this situation. However, if you add new code, your metrics reflects the expectation of not allowing less relative coverage for the new code than the relative coverage you had before.

Side note: be aware, none of these metrics tells you if the tests are covering the most sensible parts of the code base.

Related Topic