A bit of a background here – we are a small team (of 5) of RAD developers responsible for internal software development in a big non-software company. "Internal software" varies from a desktop .NET application using MSSQL server as a backend to Python scripts running on the background to MS Word documents and templates – a zoo of technologies.
The whole team consist of all-arounders able to get the requirements from users, code it up, test it and deploy into production. Once the software in the production it is being looked after by another team but it usually easy for us to intervene if something goes wrong.
All sounds good so far, but there is a problem – being a RAD team we have to release often, and there is no day going by without us releasing new versions of one or two applications(or it could be a script, updated word document, C++ console app, etc) into the production. We do a development testing and also do involve end-users by letting them run the software in UAT environment …
… but the bugs are creeping in to production anyway. Users do understand that these bugs and the occasional instability is the price they are paying for getting what they want really quickly, but at the same time it got us thinking – perhaps we could improve our development or a release practices to improve the stability of the software and reduce the number of bugs we introduce when adding a new functionality.
The good thing – we don't really have much of the processes in the first place, so it should be easy to start improving, the bad thing – being a small RAD team we don't really have the much time and resources to implement something big, but we have been thinking about the following initiatives and would welcome any feedback, tips, hints and suggestions.
-
Currently some of the applications are being released into the production straight after the developer testing, bypassing the user acceptance testing. That practice should be discontinued and even a small change has to be tested by a end-user. Each application will have a dedicated beta-tester selected from the end-users. Only after a beta-tester has ok-ed the new release it is promoted from test to production environment.
-
We don't conduct code reviews – but we'll start doing code reviews before one of us checkin the changeset. I was also thinking about a "rollout review" – basically one of the developers has to sit next with the other watch him/her doing the software rollout (copy binaries, update configs, add new table to database, etc) – it usually only takes a 5-10 minutes so it won't take much of a "rollout review" time.
-
How to miminise the rollback time when a new release is proven to be buggy enough to be pullout from production and to be replaced by a good previous version. We do store a history of all releases (as a binaries) to make it easy to go one version back – and though it is quick "overwrite a newly released binaries with a previous versions binaries" it is still a manual process which is error prone and demanding at times "what if the rollback will fail and will render the system unusable instead of buggy".
This is where we ran out of our ideas and we'd like to get your feedback on these and if you could share some simple release / dev process improvement advices – that would be awesome.
Best Answer
+1 for touching upon a great subject. When we do "Release early release often" line of development, things pick up real pace and as the momentum builds many such issues arise (as you described) which we are otherwise not very prepared to cope up with. Worst fear is when people see speed as an enemy of good work.
I have seen very limited literature on this however, this is what we practice that definitely helps:
1. Effective Bug tracking
Make bug tracking more effective - what we do is not only keep a list of bugs and tick mark, but when closed, we must define certain things like "was the problems reproducible?", "is this a permanent solution or work fix?", "what is the root cause" of trouble? This allows knowledge of what happened, when this bug was visible last time. This is key to ensure that bugs do not repeat often.
2. Define key fall back points
We all know that bugs will arrive, so we will need to provide effective fall-back which works most often. Time and again we finalize (with a ratio of about 1 of every 10 in our case) a most common release that works everywhere in most reliable manner. The total number of releases can be many but if anything goes wrong, the fall backs are select few and you don't have to fall back any further. One of the simplest way to know the best fall-back is to see which earliest release which has been running longest in production without much issues.
3. Distinguish risky and stable or small bug fix releases
When we know we have a major algorithm changes, more likely that bugs might creep in on scenarios that are not all foreseen. Where as there are times when issues are very small (or well understood) as well as little risk. Do NOT club such functionality and simple bugs in same releases. Always have a smaller bug fixed first, which must go wherever required. Make dedicated releases for special feature sets at best you can deprecate that feature but but all other important bugs are still available fixed in prior release.
4. Branch for significant feature development
Anything which associate changes which has design affect must be done separately on a branch. Larger development doesn't get completed quickly as compared to smaller bugs. If we introduce intermediate commits where 'partial' work related to feature which is still not in use - is a potential bug introduction region; the bugs which wouldn't have arise if full work for the feature would have completed atomically - hence these are bugs which we would n;t have to solve if there were branches.
5. Always plan release which are theme based
Many a times many different bugs arrive of different releases -but it is best to organize bugs (and features) which affect similar modules eases the repeat work and minimize the number of bugs originated from that work. Always prepare release road-map well in advance; bugs keep pouring in - and that falls into different target releases to optimally have a good group of bugs to be shot together in a good release. When similar bugs are combined together, it always gives better insight about contradicting scenarios.
6. Extend any new release first to a few customers
In our case, we see test it in couple of sites first and all other sites are applied a release only when there is a demand for it. Many a times some (or most) users would jump only from stable release to another stable releases only.
7. Regression Testing
Along the lines of bugs being collected - build regression suit. Also if possible mark critical bugs and test to be most important that become minimum qualifying criteria to be tested before a release candidate becomes a release indeed.
8. Pause and reflect
When many things go in full speed, there should be time to put some breaks - take a pause and have releases that are functionally no better. In fact have holiday of releases for some time. (the duration is inversely proportional to frequency). For example, many a times we have these so called "clean-up" releases which achieves nothing new from functionality point of view - but that helps great in keeping code maintainable. Most such releases are great fall back points that you almost never recall the history prior to that.
9. Perhaps the most strange
I find this one difficult to implement often but is a sure shot good trick. Swap the owner of certain modules. When people are asked code-reviews to be done, not much comes out of this practice. But when you have to seriously deal with that new code, when you swap authors, potential "bad" ailments gets noticed quickly much before they start polluting the code. Of course, this reduces the speed - but if you do this often, chances are that people master various parts of the code and learn about whole product which is other wise very difficult to teach.
10. Last but not the least
Learn to go back to white board often. The more you re-think as if this feature would have been part of our most initial design, how would we have thought of the design at that time? Sometimes, the biggest challenge with incremental work is just that we are too constrained by order of functionality we built first and quite often can't go back to basics. The trick is to keep seeing how would we generalize rather than accommodate this new feature or scenario. This requires that design remains current, and that happens only if we go back go drawing board often. Also, as new generation programmers join in, they become part of the thinking tank rather than just putting patches around.
EDIT
11. Keep track of work-around and design gaps.
Quite often we are under pressure of timelines to fix the bug and release in production. However, when the bug is at design level quite a few things needs to change but often people will fix by some short-cuts to meet the deadline. This is OK. However, as multiple such work around solutions increases, the code becomes fragile. Keep a special track on how many design gaps are already gone in. Typically, when you negotiate the timelines with project manager it is best to make him/her commit that we shall deliver this in short-cut to save production but we shall also have timeline and resources to get permanent solution.