I have been using an agile methodology (SCRUM) for about three years now and I see certain advantages to it, especially in the short-term feedback at many levels (from customers having early access to implemented features, from testers that can test features as soon as they are implemented, from other developers that can provide very early feedback on new code through review, etc).
On the other hand, I have two open problems, the first of which I will try
to explain in this question.
Problem : Difficulty to get a good design
I try to do refactoring as soon as the code gets messy, I write unit tests as much as I can (which does help to prevent bugs in general and when refactoring in particular). On the other hand, developing some complex feature in small increments, with daily commits and continuously rethinking the code when it gets unstructured is not allowing me to produce really good design.
The only well-designed module I could produce recently I obtained by taking a different approach: I analyzed the problem for a few days (I actually had had the problem going through my mind for a couple of months before I started to work on it seriously), sketched a rather detailed design of all the involved classes and their relationships for a couple more days, and then locked myself up in my office and wrote down the whole code by working without interruption for about three weeks. The result was the best thing I have produced in a while, with very few bugs that were rather easy to locate and to fix, and with a very clear design, which has not required any relevant changes since then.
So up to now I found it much more effective to get an overall picture of what I want to do beforehand than starting to write code in small increments in the hope that the big picture magically emerges in the process. With my best effort, the small-increment development approach has always led me to worse design.
Question: Has anyone had a similar experience? Am I applying SCRUM the wrong way or what should I pay attention to if I want to develop in small increments and still end up with a well-designed piece of software? Or should I schedule a design user story before starting the actual coding? Is this considered a good practice, at least for features that are more complex than average?
EDIT – NOTE
I am aware of the fact that good design is not something absolute and
does not have a value on its own but it depends on the context,
and that one should aim at a design that is good enough for the problem
at hand.
For example, I do not care (too much) about good design if I have to implement a simple component that (1) must be ready as soon as possible, (2) is going to be used only once, (3) is not used by other parts of the system (YAGNI).
I do care about good design when a component (1) will be used several times and in several different releases of a product, (2) needs to be maintained and extended over time, (3) has a lot of other components depending on it.
Best Answer
Then don't do that.
Not everything fits into the nice agile box. Often times a product will have a few complex things that can't be farmed out to individuals and cannot be completed in any sane manner within a sprint. Forcing them into that box will only cause trouble. But these should be few and far between. If you find that many of your components are like this, your product owner needs to work at breaking things up better (with your team). I am fine doing like what you did: take the story, design it, then hammer it out ASAP with the expectation that it will take a few weeks.
In the more general case, there are 3 things that I've seen done for getting good design in Agile:
Actually do design - A lot of places I've seen start their sprint and just start cowboying out code. You'll get bad design this way. Agile doesn't change the development process of plan - code - test - release, it just shortens it to more granular pieces. At the beginning of the sprint, sit down as needed and actually design your solution.
Have an architect/lead - Once your project is big enough, you'll end up with multiple scrum teams working on different pieces of the application. It is very useful to have someone (or multiple people depending on the size of the application) whose primary job is to know what all of the teams are doing from a design standpoint. They can answer questions and guide the teams towards a more harmonious over-arching design. Having each scrum team have a lead that knows most of what other teams are doing I've also seen and was very effective.
Be a pragmatist - I covered this one above. Agile is a tool, and like any tool; it doesn't apply to all problems. If it doesn't make sense to apply to some component, then don't apply it there. Your goal is to ship good software; don't forget it.