Unit Testing – How to Determine What Is Useful?

unit testing

I've been going through phpunit's docs and came accross the following quote:

You can always write more tests. However, you will quickly find that only a fraction of the tests you can imagine are actually useful. What you want is to write tests that fail even though you think they should work, or tests that succeed even though you think they should fail. Another way to think of it is in cost/benefit terms. You want to write tests that will pay you back with information.
–Erich Gamma

It got me wondering. How do you determine what makes a unit test more useful than another, aside from what's stated in that quote about cost/benefit. How do you go about deciding which piece of your code you create unit tests for? I'm asking that because another of those quotes also said:

So if it's not about testing, what's it about?
It's about figuring out what you are trying to do before you run off half-cocked to try to do it. You write a specification that nails down a small aspect of behaviour in a concise, unambiguous, and executable form. It's that simple. Does that mean you write tests? No. It means you write specifications of what your code will have to do. It means you specify the behaviour of your code ahead of time. But not far ahead of time. In fact, just before you write the code is best because that's when you have as much information at hand as you will up to that point. Like well done TDD, you work in tiny increments… specifying one small aspect of behaviour at a time, then implementing it.
When you realize that it's all about specifying behaviour and not writing tests, your point of view shifts. Suddenly the idea of having a Test class for each of your production classes is ridiculously limiting. And the thought of testing each of your methods with its own test method (in a 1-1 relationship) will be laughable.
–Dave Astels

The important section of that is

*And the thought of testing each of your methods with its own test method (in a 1-1 relationship) will be laughable. *

So if creating a test for each method is 'laughable', how/when do you chose what you write tests for?

Best Answer

How many tests per method?

Well the theoretical and highly impractical maximum is the N-Path complexity (assume the tests all cover different ways through the code ;)). The minimum is ONE!. Per public method that is, he don't test implementation details, only external behaviors of a class (return values & calling other objects).

You quote:

*And the thought of testing each of your methods with its own test method (in a 1-1 relationship) will be laughable. *

and then ask:

So if creating a test for each method is 'laughable', how/when do you chose what you write tests for?

But i think you misunderstood the author here:

The idea of having one test method per one method in the class to test is what the author calls "laughable".

(For me at least) It's not about about 'less' it's about 'more'

So let me rephrase like i understood him:

And the thought of testing each of your methods with ONLY ONE METHOD (its own test method in a 1-1 relationship) will be laughable.

To quote your quote again:

When you realize that it's all about specifying behaviour and not writing tests, your point of view shifts.

When you practice TDD you don't think:

I have a method calculateX($a, $b); and it needs a test testCalculcateX that tests EVERYTHING about the method.

What TDD tells you is to think about what your code SHOULD DO like:

I need to calculate the bigger of two values (first test case!) but if $a is smaller than zero then it should produce an error (second test case!) and if $b is smaller than zero it should .... (third test case!) and so on.

You want to test behaviors, not just single methods without context.

That way you get a test suite that is documentation for your code and REALLY explains what it is expected to do, maybe even why :)

How do you go about deciding which piece of your code you create unit tests for?

Well everything that ends up in the repository or anywhere near production needs a test. I don't think the author of your quotes would disagree with that as i tried to state in the above.

If you don't have a test for it it gets way harder (more expensive) to change the code, especially if it's not you making the change.

TDD is a way to ensure that you have tests for EVERYTHING but as long as you WRITE the tests it's fine. Usually writing them on the same day helps since you are not going to do it later, are you? :)

Response to comments:

a decent amount of methods can't be tested within a particular context because they either depend or are dependent upon other methods

Well there are three thing those methods can call:

Public methods of other classes

We can mock out other classes so we have defined state there. We are in control of the context so thats not a problem there.

*Protected or Private methods on the same *

Anything that isn't part of the public API of a class doesn't get tested directly, usually.

You want to test behavior and not implementation and if a class does all it's work in one big public method or in many smaller protected methods that get called is implementation. You want to be able to CHANGE those protected methods WITHOUT touching your tests. Because your tests will break if your code changes change behavior! Thats what your tests are there for, to tell you when you break something :)

Public methods on the same class

That doesn't happen very often does it? And if it does like in the following example there are a few ways of handling this:

$stuff = new Stuff();
$stuff->setBla(12);
$stuff->setFoo(14);
$stuff->execute();

That the setters exist and are not part of the execute method signature is another topic ;)

What we can test here is if executes does blow up when we set the wrong values. That setBla throws an exception when you pass a string can be tested separately but if we want to test that those two allowed values (12 & 14) don't work TOGETHER (for whatever reason) than thats one test case.

If you want a "good" test suite you can, in php, maybe(!) add a @covers Stuff::execute annotation to make sure you only generate code coverage for this method and the other stuff that is just setup needs to be tested separately (again, if you want that).

So the point is: Maybe you need to create some of the surrounding world first but you should be able to write meaningful test cases that usually only span one or maybe two real functions (setters don't count here). The rest can be ether mocked away or be tested first and then relied upon (see @depends)

*Note: The question was migrated from SO and initially was about PHP/PHPUnit, thats why the sample code and references are from the php world, i think this is also applicable to other languages as phpunit doesn't differ that much from other xUnit testing frameworks.

Related Topic