When writing tests for one piece of software, say a library, do you prefer to compile all unit tests into one, or separate them into several executables?
The reason I'm asking is because I am currently using CUnit to test a library I'm working on. The tests are split up into separate suites that are compiled into one executable complete with printed output for failures. Now, the build system for that library is CMake (which, despite its name, has little to do with CUnit), which comes with its own testing framework, CTest. CTest allows me to register a list of executables that serve as tests.
I'm pondering whether to use CTest for automated testing runs. However, this would require me to split up the tests I've written so far into separate compile targets. Otherwise, I can't really utilize some of CTests advanced features, such as selectively running tests.
I realize this is more a question of what tools to use and their handling and conventions, but apart from that, are there any other reasons to prefer a single test executable over separate ones? Or vice versa?
Best Answer
I like to have my automated tests in individual binaries, or at least clumped per "belongs-together" group, and then call them from a simple shell script (where a non-zero exit code signals failure, and output on stderr may be captured to record an explanation). This way, I retain full flexibility on the testing - I can run individual tests directly from the command line, I can make all sorts of fancy scripts if I want to, I can reorder them as I see fit without recompiling anything, etc.
But more importantly, it also allows me to include tests written in different languages or using different toolchains in the same run. For example, the unit tests I write are most likely in the project's main language, and running them is a matter of building and invoking the binaries; but I also want to test my database, and I might want to feed SQL scripts directly to the database for this; I might want to run some static code analysis tool on my code (even if it's just some sort of linter). I may want to run my static HTML through a validity checker. I could run a
grepcommand over the codebase to check for suspicious constructs, coding style violations, or "red-flag" keywords. The possibilities are endless - if it can be run from the command line and adheres to "zero exit status means OK", I can use it.