I'm currently designing an automated testing framework for doing embedded systems testing on an embedded Linux system. There exists an old system, but it is particularly hard to use, mainly because it wasn't designed as a framework and was originally designed to test one product only. What this means is that the framework changes with each new product to test, which would take a lot of time spent simply understanding where to modify the system and how.
Now, I've been given the opportunity to create a new system from the ground up, with the goals of making it much more easy to use and a lot more intuitive. One of the things that I noticed in both the old system and other test systems I used was the idea of setting up all the tests and verification conditions, and then running the test so that the set up and test running were two different paths that occurred sequentially. One of the downsides to this is that any print statements would occur outside the flow path of the actual tester, so printing out failed variables needs to be added into the tester somehow.
Now that it falls on me to redesign the system. I'm wondering what added benefits are presented by going with a test-runner style architecture over simply setting things up, with checking variables and reporting back all at the same time.
Best Answer
I suggest taking a glance at the book XUnit Test Patterns. It covers a lot of different ways to define a test framework, and it discusses some of the pros and cons of many different approaches.
I think the best benefit of using a test runner vs. the approach that you mentioned, is that it becomes easier to run a subset of the tests (which could be just one of them).