Unit Testing – Is It Bad Practice to Enforce Execution Order for Unit Tests?

patterns-and-practicestestingunit testing

I am writing tests for a project that consists of multiple submodules. Each test case that I have written runs independent of each other and I clear all data between tests.

Even though the tests run independently, I am considering enforcing an execution order, as some cases require more than one submodule. For example, a submodule is generating data, and another one is running queries on the data. If the submodule generating the data contains an error, the test for the query submodule will also fail, even if the submodule itself works fine.

I can not work with dummy data, as the main functionality I am testing is the connection to a black box remote server, which only gets the data from the first submodule.

In this case, is it OK to enforce an execution order for the tests or is it bad practice? I feel like there is a smell in this setup, but I can not find a better way around.

edit: the question is from How to structure tests where one test is another test's setup? as the "previous" test is not a setup, but tests the code which performs the setup.

Best Answer

I can not work with dummy data, as the main functionality I am testing is the connection to a black box remote server, which only gets the data from the first submodule.

This is the key part for me. You can talk about "unit tests" and them "running independently of each other", but they all sound like they are reliant on this remote server and reliant on the "first sub module". So everything sounds tightly coupled and dependent on external state. As such, you are in fact writing integration tests. Having those tests run in a specific order is quite normal as they are highly dependent on external factors. An ordered test run, with the option of an early quit out of the test run if things goes wrong is perfectly acceptable for integration tests.

But, it would also be worth taking a fresh look at the structure of your app. Being able to mock out the first submodule and external server would then potentially allow you to write true unit tests for all the other submodules.

Related Solutions

Unit Testing – Should Each Unit Test Be Run Independently?

If you went with the first option, each method would be isolated and tested independently, but you would never really know that they can really work together properly.

If your methods are truly independent this shouldn't matter. Your second method should:

a) Work correctly when presented with valid data.

b) Fail sensibly and consistently when presented with invalid data.

Equally your first method should do the same. So as long as you handle the error cases they will work together properly.

If you want to test that the methods work correctly together then that's integration testing, not unit testing.

Unit-testing – Is using unit tests to tell a story a good idea

Yes, it is a good idea to give your tests names of the example scenarios you are testing. And using your unit testing tool for more than just unit tests maybe ok, too, lots of people do this with success (me too).

But no, it is definitely not a good idea to write your tests in a fashion where the order of execution of the tests matters. For example, NUnit allows the user to select interactively which test he/she wants to be executed, so this will not work the way intended any more.

You can avoid this easily here by separating the main testing part of each test (including the "assert") from the parts which set your system in the correct initial state. Using your example above: write methods for creating an account, logging on and post a comment - without any assert. Then reuse those methods in different tests. You will also have to add some code to the [Setup] method of your test fixtures to make sure the system is in a properly defined initial state (for example, no accounts so far in the database, noone connected so far etc.).

EDIT: Of course, this seems to be against the "story" nature of your tests, but if you give your helper methods meaningful names, you find your stories within each test.

So, it should look like this:

[TestFixture]
public class Authentication_Bill
{
    [Setup]
    public void Init()
    {  // bring the system in a predefined state, with noone logged in so far
    }

    [Test]
    public void Test_if_Bill_can_create_account()
    {
         CreateAccountForBill();
         // assert that the account was created properly 
    }

    [Test]
    public void Test_if_Bill_can_post_comment_after_login()
    { 
         // here is the "story" now
         CreateAccountForBill();
         LoginWithBillsAccount();
         AddCommentForBill();
        //  assert that the right things happened
    }

    private void CreateAccountForBill()
    {
        // ...
    }
    // ...
}

Related Topic