I have often heard unit tests help programmers build confidence in their software. But is it enough for verifying that software requirements are met? I am losing confidence that software is working just because the unit tests pass.
We have experienced some failures in production deployment due to an untested\unverified execution path. These failures are sometimes quite large, impact business operations and often requires an immediate fix.
The failure is very rarely traced back to a failing unit test. We have large unit test bodies that have reasonable line coverage but almost all of these focus on individual classes and not on their interactions.
Manual testing seems to be ineffective because the software being worked on is typically large with many execution paths and many integration points with other software. It is very painful to manually test all of the functionality and it never seems to flush out all the bugs.
Are we doing unit testing wrong when it seems we still are failing to verify the software correctly before deployment? Or do most shops have another layer of automated testing in addition to unit tests?
Best Answer
In general no. For toy programs (say solutions to Project Euler problems), sure.
This is before getting into a religious discussion of what are unit tests (if it uses a file system object, is it a unit test? Let the inquisition begin!)
For N-Tier applications, where N > 1, integration tests are needed, and even they may not actually be suitable for requirements verification, as most integration tests are between two tiers, and not normally end-to end testing. Most end-to-end testing (a minimal test for most requirements) is usually done as / in conjunction with system level test.
On a large project I worked on, we had CUT (Code and Unit Test) done by the developer. The build code then went into Integration and Test (I&T), where some minor requirements that could be verified (warning alerts will be displayed in 'FUNKY_RED' in the admin console). When it passed that, it was released to System Level Test (SLT). The 'Requirements Testing' was done in SLT, where they had a mirror of the operational site. Finally, it was deployed and went to Acceptance Test.
Some of this would change, depending on the scale of the project. But in my experience, anything with more than one module/jar/library will not be sufficient with just unit tests.
Add in testing a GUI, and it gets really unlikely that a you can have 100% confidence without live interaction.
EDIT: Looking back, I think that take away is that for complex systems, simple testing (like unit testing) is not going to be sufficient for requirements validation. Complex systems will require complex (sophisticated?) testing to verify.