Testing – Deterministic vs Non-Deterministic Approaches

mvctesting

Is it better to have either a

  • Deterministic test suite, that results in the same tests succeeding
  • Non-deterministic test suite, which potentially possibly covers more cases

?

Example:
You write a test suite to test controller functionality in an MVC application. The controller requires application data from a database as input during the test. There are two options to do this:

  • You hardcode which row(s) from the test database are selected as input (e.g. the 10th and 412th row)
  • You use a random number generator to pseudorandomly pick the data from the database (two rows selected by a random number generator)

The first is deterministic: every run of the test for the same revision of code should yield the same result. The second is non-deterministic: every run of the test suite has the possibility to yield a different result. The randomly picked data might however be a better representation of data edge cases. It might simulate a user feeding our controllers with unpredictable data better?

What are reasons to choose one over the other?

Best Answer

When every run of the test suite gives you the possibility to yield a different result, the test is almost completely worthless - when the suite shows you a bug, you have a high chance that you cannot reproduce it, and when you try to fix the bug, you cannot verify wether your fix works (or not).

So when you think you need to use some kind of random number generator for generating of your test data, either make sure you always initialize the generator with the same seed, or persist your random test data in a file before feeding it into your test, so you can re-run the test again with exact the same data from the run before. This way, you can transform any non-deterministic test into a deterministic one.

EDIT: Using a random number generator to pick some test data is IMHO sometimes a sign for being too lazy about picking good test data. Instead of throwing 100,000 randomly choosen test values and hope that this will be enough to discover all serious bugs by chance, better use your brain, pick 10 to 20 "interesting" cases, and use them for the test suite. This will not only result in a better quality of your tests, but also in a much higher performance of the suite.

Related Topic