As I am looking more and more into Continuous Delivery for our organisation, I struggle to understand how we can effectively incorporate manual QA testing before a release is pushed out into production.
I have found multiple approaches, such as the branch-per-feature one explained by Adam Dymitruk or git-flow and the likes, which all seem to include a lot of manual work to get things setup each sprint or perform rollback operations.
I'll try to explain my issue by given an example.
Most CD documents outline the "happy" path flow as follows:
develop -> CI build -> deploy to QA -> test on QA -> promote to Production
This is easy enough to understand and works fine if there are no issues. But let's assume the following workflow which requires manual testing before a deploy to production is authorized: developer 1 works on feature A, developer 2 works on feature B. I use the name features, but I don't necessarily mean this in the broad range of a feature which could be contained by using feature toggles. Just some work.
developer 1 finishes his work, merges everything into master – or the equivalent release branch -, triggers the build which is deployed into QA. QA starts testing. During QA tests, developer 2 finishes his work, merges into master, triggers the build. QA then rejects feature A. Thus, the release from developer 1 cannot continue. At this point the entire process halts. As long as feature A is not ready, feature B cannot be tested because it also contains the rejected code from feature A.
And this is the part which I don't really understand how to fix in an easy, automated way. How can we effectively rollback feature A without complex manual interventions? How do people do this? Surely this must be an issue everyone faces, but none of the books or schematics I've read so far seem to really address this issue.
Taking it one step further. GitHub flow seems to suggest to deploy the feature branch into production, and rollback if a production issue occurs. That might work in a single developer single tester scenario, but what happens if multiple features are signed of by multiple testers at the same time?
developer 1 starts his feature, branches of master, starts working, pushes into master, QA 1 tests. At the same time, developer 2 went through the same process and QA 2 tests his feature build. Given the schematic, feature A does not contain feature B and feature B does not contain feature A. QA 1 signs off, feature A goes into production. QA 2 signs off, feature B goes into production. But feature A is lost. It's in master, yes, after the deploy it had been merged, but it won't get back into production until feature C gets deployed.
How would GitHub flow effectively work in larger teams? Has anyone some practical examples on this?
Best Answer
When a code change A is rejected, don't think necessarily in terms of "rollback feature A by undoing the merge using SCCS". Think in terms of "adding a new change to the code which fixes the defects found by QA".
Depending on the issue, this can mean:
fixing a bug immediately, but leaving the code of A in master
disabling the functionality added by A (by deactivating a feature entry point, maybe by utilizing a "feature toggle"), until the issues can be fixed, but still leaving the related code in master
adding changes to the code which effectively remove the code changes of A from the master - but in a manner this does not interfer with later changes like B
The third bullet point can often, but not necessarily, mean to let the developer (and not the QA team by some magic automatism) use the SCCS and revert the code changes of A. If this rollback shows no collision with B, fine, if there is a collision, there might be some manual work involved (comparable to the manual work when A and B were integrated one after another).
And yes, picking the right strategy is a manual decision, and it will involve some time and manual work - this is not a fully "automated way", but it does also not need "complex manual intervention" every time it happens. In reality, the range of an issue fix or rollback goes from "trivial" to "complex". When aiming for continous delivery, however, one should strive to have the trivial rollbacks happen much more frequently than the complex ones.
The best approach for this is probably to pick features for parallel development in a way it makes it unlikely they need to touch the same part of the code base. Furthermore, there should be as many automated tests as possible. So the devs can run these tests during feature development before their code reaches "master". That should prevent frequent show-stoppers during QA.
Now, let us apply these recommendations to your scenario:
But why not? When A and B were eligible for parallel development, they should be mostly independent from each other and so eligible for independent testing, which means B can in most cases be tested even if the issues with A are not fixed. Lets say QA reports a serious bug from A which was missed by the automated tests. The fact the automated tests did not fail should mean at least the application should not be completely unusable, so they can start with the tests of B. They know master cannot be delivered to production as long as the issue with A is not fixed, but during the test of B, there should be enough time to fix or rollback A in the one of the 3 ways I mentioned above.
Of course, one has to make a decision what the quickest way is to remove the issue with A and make the code "production-ready" again. So when the test of B is done, a fix for the issue with A should be already available (or at least soon), so QA can approve that the issue is gone now (either with the functionality from A now or not).
Looking at the timeline, after each test cycle your "master" can switch between two main states: either it has "known defects", or there are "no known defects any more". Whenever it reaches the state "no known defects", you can deploy to production. To make continous deployment work, one has to plan the feature slices in a way the state "no known defects any more" is reached as frequent as possible. The key here is to
make the individual feature slices like "A" and "B" from above as small as possible
pick feature slices for parallel development as independent as possible
plan for reverting or disabling a feature slice in a very quick and smooth way in case it contains a bug which prevents the delivery to production
In a bigger team (maybe a dozen or more developers), when you notice you reach the state "no known defects" too seldom because of the constant checkins to master, you can also consider to use an additional "pre-production" (or "staging") branch where only features are integrated which were approved by QA on the master branch. However, this does not come for free, you need another person doing the integrations on "pre-production", another QA step on "pre-production" to check the integrations itself did not introduce a bug, and additional administrative overhead for managing the pre-production environment.