Unit Testing – To What Extent Should You Test What Should Not Happen

testingunit testing

In the case of unit testing, I'm testing as many cases as I can that when I do X, Y happens.

I'm also concerned in some cases that when I do X, Z does not happen.

But for the sake of protection, I could also test that A, B, C … do not happen either. And going that direction I could basically test a billion things that should not happen, and, in many cases, won't ever happen.

My question is, at what point do you think you're being too defensive, and when is it still considered useful ?

Let's use this bit of pseudo code as an example 

void MyFunction()
{
    if (X)
    {
        A();
    }
    else
    {
        B();
    }

    C();
}

void AnotherFunction()
{
   //Irrelevant code
}

Here are the tests I can have :

  • A happens when X is true
  • A does not happen when X is false
  • B does not happen when X is true
  • B happens when X is false
  • C happens when X is true
  • C happens when X is false

This is what I would do. But following that reasoning I could also test that AnotherFunction is not called when X is true and false, just like I'm testing that C is called when X is true and false or that A/B are not called when X is false/true.

And going down that road I could literally test a million things that shouldn't happen for every test case I can find. Just to make sure next developers won't break the code.

In my example you might say "this is too obvious to be tested" but I'm sure you can imagine a point where code could be so fragile / complex that testing many different things that shouldn't happen could be done. My question is : should it be tested, and how far ?

Is there some kind of rule ? Am I missing something in the unit testing knowledge ?

On one side I feel that this is safer and offers a good defense against pretty much anything that I can think of.
On the other hand this is far too defensive and abominable to maintain, while time consuming since you get to code 'anything that you could think of'.

Note that I applied this to function calls, but it extends to anything that follows the line of "testing things that should not happen".

Where should one stop testing in that direction?

Best Answer

Tests serve a few purposes.

  1. Make sure requirements are met.
  2. Make sure code behaves as expected.
  3. Describes the expected behavior of the code.

Also, the tests you write need to give you value. Value comes from fulfilling one of those purposes.

For example, writing a test to make sure that 2 + 2 actually yields 4 is something you could do, you know, to make sure the compiler and your CPU work as designed. But that doesn't give you a lot of value and doesn't justify writing the test. (Nor does it really fulfill a purpose of a test.)

It's possible to write tests to make sure that RandomOtherFunction isn't called, but that isn't likely to give you value. It isn't (typically) fulfilling one of those purposes. It's describing what the code isn't. And that would be impossible to write everything the code isn't. It also isn't valuable to you to know what the code isn't and that no solar rays or gramlins have changed the code into that.

Tests that specify too much of what your program isn't also makes it hard to change. What if you, for some reason in the future, have to start using RandomOtherFunction in a place you didn't used to? Now you have to refactor lots of tests on top of doing the work. The tests didn't help you make the change or make it safely, they just made it harder.

You stop writing tests when they don't give you value. How much value justifies writing a test is up to you and your organization. If your company really, really wants to pay you to write a million tests about what the software isn't, fine. Let them pay you for that "value". (You also may want to look for a new job because I can't imagine a company that pays for that kind of value lasting very long.)

Related Topic