Python Unit Testing – How Much to Break Up Unit Tests

pythonunit testing

I recently learned about unit tests and TDD and have started going back to add unit tests to my code (and thank god, it's proven that some things I've written are much more broken than I thought).

That being said, I'm having a hard time figuring out how much (or little) should go into each test. For example, consider a class I wrote with several methods. Right now I do something like

class test_myclass(unittest.TestCase):

    def setUp(self): 
        ...
    def test_func1(self):
        ...
    def test_func2(self):
        ...
    etc
    def tearDown(self):
        ...

But what if each (or at least some) method can fail in a lot of different ways, or has a lot of edge cases? Then I'm stuck doing something like

def test_func1(self):
    self.assertEqual(func1(working_input), expected_value)
    self.assertRaises(Some_error, func1, args)
    self.assertRaises(Some_other_error, func1, args)
    etc

And it could get pretty hairy, and rather unclear where things are failing. So I thought I might break it up more, and devote a class to each function (that has enough test cases to be worth it) but that seems like a lot of work, and more time than I should probably be spending on testing.

I also considered writing multiple tests for a given method, such as

def test_func1(self):
    self.assertEqual(func1(working_input), expected_val)
def test_func1_bad_input_1(self):
    self.assertRaises(error, func1, bad_args)
def test_func1_bad_input_2(self):
    self.assertRaises(error2, func1, bad_args_2)
...

And this seems cleaner than a unit testing class per method, but still seems like it might be to little in any given test.

How much testing is too much?

While my example was given in python, I'm open to language agnostic answers

Best Answer

In an ideal world, a single unit test has only one thing that can cause it to fail, and that one thing can only cause that test to fail.

I'm not sure I've seen that ever be true though... A more attainable approach is that a single unit test tests a single path though a function. The concept of code coverage is helpful here, but the gist of it is that any decision Your code makes represents (at least) a different path. Some paths may be too difficult (or actually impossible) to unit test, but that is the level of granularity to work towards.

The benefit is that when a test fails, you spend less time and effort figuring out why.

And yes, it seems like a bit more work, but once you're used to it, the tests are quick and easy to write - and they'll save you tons of time.

Related Solutions

Unit-testing – Is using unit tests to tell a story a good idea

Yes, it is a good idea to give your tests names of the example scenarios you are testing. And using your unit testing tool for more than just unit tests maybe ok, too, lots of people do this with success (me too).

But no, it is definitely not a good idea to write your tests in a fashion where the order of execution of the tests matters. For example, NUnit allows the user to select interactively which test he/she wants to be executed, so this will not work the way intended any more.

You can avoid this easily here by separating the main testing part of each test (including the "assert") from the parts which set your system in the correct initial state. Using your example above: write methods for creating an account, logging on and post a comment - without any assert. Then reuse those methods in different tests. You will also have to add some code to the [Setup] method of your test fixtures to make sure the system is in a properly defined initial state (for example, no accounts so far in the database, noone connected so far etc.).

EDIT: Of course, this seems to be against the "story" nature of your tests, but if you give your helper methods meaningful names, you find your stories within each test.

So, it should look like this:

[TestFixture]
public class Authentication_Bill
{
    [Setup]
    public void Init()
    {  // bring the system in a predefined state, with noone logged in so far
    }

    [Test]
    public void Test_if_Bill_can_create_account()
    {
         CreateAccountForBill();
         // assert that the account was created properly 
    }

    [Test]
    public void Test_if_Bill_can_post_comment_after_login()
    { 
         // here is the "story" now
         CreateAccountForBill();
         LoginWithBillsAccount();
         AddCommentForBill();
        //  assert that the right things happened
    }

    private void CreateAccountForBill()
    {
        // ...
    }
    // ...
}

Should Data Be Hard Coded Across All Unit Tests?

Ultimately, you want to write as little code as possible to get as much outcome as possible. Having a lot of the same code in multiple tests a) tends to result in copy-paste coding and b) means that if a method signature changes you can end up having to fix a lot of broken tests.

I use the approach of having standard TestHelper classes that provide me with a lot of the data types that I routinely use, so I can create sets of standard entity or DTO classes for my tests to query and know exactly what I will get each time. So I can call TestHelper.GetFooRange( 0, 100 ) to get a range of 100 Foo objects with all their dependent classes/fields set.

Particularly where there are complex relationships configured in an ORM type system which need to be present for things to run correctly, but aren't necessarily significant for this test that can save a lot of time.

In situations where I'm testing close to the data level, I sometimes create a test version of my repository class that can be queried in a similar way (again this is in an ORM type environment, and it wouldn't be relevant against a real database), because mocking out the exact responses to queries is a lot of work and often only provides minor benefits.

There are some things to be careful of, though in unit tests:

Make sure your mocks are mocks. The classes that perform operations around the class being tested must be mock objects if you are doing unit testing. Your DTO/entity type classes can be the real thing, but if classes are performing operations you need to be mocking them- otherwise when the supporting code changes and your tests start failing, you have to search for a lot longer to figure out which change actually caused the problem.
Make sure you are testing your classes. Sometimes if one looks through a suite of unit tests it becomes apparent that half of the tests are actually testing the mocking framework more than the actual code that they are supposed to be testing.
Don't reuse mock/supporting objects This is a biggie- when one starts trying to get clever with code supporting unit tests it is really easy to inadvertently create objects that persist between tests, which can have unpredictable effects. For example, yesterday I had a test that passed when run on its own, passed when all tests in the class were run, but failed when the entire test suite was run. It turned out there was a sneaky static object way off in a test helper that, when I created it, would definitely never have caused a problem. Just remember: At the start of the test, everything is created, at the end of the test everything is destroyed.

Best Answer

Related Solutions

Unit-testing – Is using unit tests to tell a story a good idea

Should Data Be Hard Coded Across All Unit Tests?

Related Topic