The organisation I work for has recently employed a testing officer to run manual tests, but when I asked about being given time as a developer to write unit tests the response was that the manual testing would provide a bigger bang for the buck. That is something that feels wrong to me and I'm looking for a means to rate manual and automated testing against each other that is more scientific than a gut-feeling. I'm not saying there is no place for manual testing – but automated testing would seem, to me at least, to remove some of the more repetitive and boring tasks. We have a build server that runs some unit tests and some selenium tests – so its not like the idea of automated testing is a no-go, but just seen as a lower return on investment.
I can understand that having someone perform a full end to end test of a system tests the final product and ultimately that's all the user cares about, but it is slow going and very repetitive. Manual regression testing means repeating all the previous tests and confirming nothing has changed and if there's 4 paths through a process then that's 4 manual tests that could well take 5 minutes each.
So are there any verifiable facts and figures I can use to make the case for budgeting time for automated testing? For that matter what are the disadvantages to automated testing beyond those in the link?
Best Answer
I'd be careful at all when thinking about "manual" testing vs "automated" testing in a return-on-investment (ROI) context. It's a form of fallacious thinking when discussing the value of automation (see the "Time Machine" section of that link).
The Time Machine fallacy summed up is this: the only way you can truly determine the ROI of a testing approach is by having a time machine where you can visit the future and see what happened. Since you (most likely) can't do this, ROI estimations should be seen as a heuristic at best, and not some hard rule. Use current information to make an informed decision about a given testing effort is useful to your team or not.
Instead, when it comes to testing, think about which areas of your product need the most attention, and where the largest risks may be. Perhaps it's been identified that end-to-end integration testing might find the most issues with your app. Even if unit testing brings benefits, those benefits may not be worth the associated costs (getting developer buy in, writing and running tests, etc). Likewise, try thinking about benefits/costs instead of advantages/disadvantages for testing.