Unit Test Framework – Is It Really Necessary?

language-agnosticunit testing

Currently at my job, we have a large suite of unit tests for our C++ application. However we don't use a unit test framework. They simply utilize a C macro that basically wraps an assert and a cout. Something like:

VERIFY(cond) if (!(cond)) {std::cout << "unit test failed at " << __FILE__ << "," << __LINE__; asserst(false)}

Then we simply create functions for each of our tests like

void CheckBehaviorYWhenXHappens()
{
    // a bunch of code to run the test
    //
    VERIFY(blah != blah2);
    // more VERIFY's as needed
}

Our CI server picks up "unit test failed" and fails the build, emailing the message to the developers.

And if we have duplicate setup code, we simply refactor it like we would any other duplicated code we'd have in production. We wrap it behind helper functions, make some test classes wrap setting up frequently used scenarios.

I know there's frameworks out there like CppUnit and boost unit test. I'm wondering what value do these add? Am I missing what these bring to the table? Is there something useful I could gain from them? I'm hesitant to add a dependency unless it adds real value especially since it seems what we have is dead simple and works well.

Best Answer

As others have already said, you already have your own, simple, home made framework.

It seems to be trivial to make one. However, there are some other features of a unit-test framework that are not so easy to implement, because they take some advanced knowledge of the language. The features I generally require from a test framework and are not so easy to homebrew are:

Automatic collection of test cases. I.e. defining new test method should be enough to get it executed. JUnit automatically collects all methods whose names start with test, NUnit has the [Test] annotation, Boost.Test uses the BOOST_AUTO_TEST_CASE and BOOST_FIXTURE_TEST_CASE macros.

It's mostly convenience, but every bit of convenience you can get improves the chance developers will actually write the tests they should and that they will connect them in correctly. If you have long instructions, somebody will miss part of them now and than and perhaps some tests won't be running and nobody will notice.
Ability to run selected test cases, without tweaking code and recompiling. Any decent unit-test framework allows you to specify which tests you want to run on command-line. If you want to debug on unit tests (it's most important point in them for many developers), you need to be able to select just some to run, without tweaking code all over the place.

Say you just received bug report #4211 and it can be reproduced with unit test. So you write one, but than you need to tell the runner to run just that test, so you can debug what's actually wrong there.
Ability to mark tests expected failures, per test case, without modifying the checks themselves. We actually switched frameworks at work to get this one.

Any decently sized test suite will have tests, that are failing because the features they test were not implemented yet, were not finished yet, nobody had time to fix them yet or something. Without ability to mark tests as expected failures, you won't notice another failure when there are some regularly, so the tests stop serving their main purpose.

Related Solutions

Testing – Need for Unit Tests with Integration Tests

You've laid out good arguments for and against unit testing. So you have to ask yourself, "Do I see value in the positive arguments that outweigh the costs in the negative ones?" I certainly do:

Small-and-fast is a nice aspect of unit testing, although by no means the most important.
Locating-bug[s]-easier is extremely valuable. Many studies of professional software development have shown that the cost of a bug rises steeply as it ages and moves down the software-delivery pipeline.
Finding-masked-bugs is valuable. When you know that a particular component has all of its behaviors verified, you can use it in ways that it was not previously used, with confidence. If the only verification is via integration testing, you only know that its current uses behave correctly.
Mocking is costly in real-world cases, and maintaining mocks is doubly so. In fact, when mocking "interesting" objects or interfaces, you might even need tests that verify that your mock objects correctly model your real objects!

In my book, the pros outweigh the cons.

Unit-testing – How to not test implementation when method returns void

Dependency Injection and mocking is your friend. Whenever you find something hard to test, it usually means your class is having too many responsibilities. I'm assuming you are sending your emails using some library. That library most likely takes your message info (destination address, subject, body). What you want to do, is decouple your business logic that's in charge of collecting all email addresses, email subject and body, from that 3rd party library. How? By abstracting that library using an interface, and then injecting it into your Email class.

IEmailLibrary
{
   void SendEmail(EmailInfo emailInfo);
}

class Email
{
   private IEmailLibrary _emailLibrary;

   public Email(IEmailLibrary emailLibrary)
   {
      _emailLibrary = emailLibrary;
   }

   public void SendEmailNotifications()
   {
      EmailInfo emailInfo = ComposeEmail(); 
      _emailLibrary.SendEmail(emailInfo);
   }

   private EmailInfo ComposeEmail()
   {
      //your business logic
   }
}

Now in your production code, you would create a real EmailLibrary like this:

public RealEmailLibrary : IEmailLibrary
{
   ...

   public SendEmail(EmailInfo emailInfo)
   {
      _3rdPartyEmailComponent.SendEmail(emailInfo....);
   }
}

and then inject it into your Email class, like this:

RealEmailLibrary realEmailLibrary = new RealEmailLibrary();
Email email = new email(realEmailLibrary);

And if you wanted to test your business logic, all you need to do in your test code, is inject either a mock, or your own stub that implements an IEmailLibrary interface. For example:

public FakeEmailLibrary : IEmailLibrary
{
   public EmailInfo _iWasCalledWithThisEmailInfo;

   public void SendEmail(EmailInfo emailInfo)
   {
      _iWasCalledWithThisEmailInfo = emailInfo;
   }
}

You inject it in your test like this:

FakeEmailLibrary fakeEmailLibrary = new FakeEmailLibrary();
Email email = new email(fakeEmailLibrary);
email.SendEmailNotifications();
AssertStuff(fakeEmailLibrary.iWasCalledWithThisEmailInfo, expectedEmailInfo);

I recommend using mocking frameworks, as opposed to my stub example (like Moq for C#), which make it really easy to verify your injected dependencies were called with the right arguments.

Best Answer

Related Solutions

Testing – Need for Unit Tests with Integration Tests

Unit-testing – How to not test implementation when method returns void

Related Topic