Unit-testing – Unit testing methods with indeterminate output

randomunit testing

I have a class that is meant to generate a random password of a length that's also random, but limited to be between a defined min and max length.

I'm constructing unit tests, and ran into an interesting little snag with this class. The whole idea behind a unit test is that it should be repeatable. If you run the test a hundred times, it should give the same results a hundred times. If you're depending on some resource that may or may not be there or may or may not be in the initial state you expect then you're meant to mock the resource in question to ensure that your test really is always repeatable.

But what about in cases where the SUT is supposed to generate indeterminate output?

If I fix the min and max length to the same value then I can easily check that the generated password is of the expected length. But if I specify a range of acceptable lengths (say 15 – 20 chars), then you now have the problem that you could run the test a hundred times and get 100 passes but on the 101st run you might get a 9 character string back.

In the case of the password class, which is fairly simple at its core, it shouldn't prove a huge problem. But it got me thinking about the general case. What is the strategy that's usually accepted as the best one to take when dealing with SUTs that are generating indeterminate output by design?

Best Answer

"Non-deterministic" output should have a way of becoming deterministic for the purposes of unit testing. One way to handle randomness is to allow replacement of the random engine. Here is an example (PHP 5.3+):

function DoSomethingRandom($getRandomIntLessThan)
{
    if ($getRandomIntLessThan(2) == 0)
    {
        // Do action 1
    }
    else
    {
        // Do action 2
    }
}

// For testing purposes, always return 1
$alwaysReturnsOne = function($n) { return 1; };
DoSomethingRandom($alwaysReturnsOne);

You could make a specialized test version of the function which returns any sequence of numbers you want to make sure the test is fully repeatable. In the real program, you can have a default implementation which could be the fallback if not overridden.

Related Topic