You're missing the point of agile a little bit. What you are calling a user story, I see as at least six: one bare-bones search, and one for each of your bullet points. By all means, make enough plans to avoid painting yourself into a corner that's going to be expensive to get out of, but the whole idea is that you provide the minimum needed to deliver some value, and do it quickly enough to get rapid feedback.
When you divide a story up like that, it not only makes it easier to estimate, it allows the product owner to prioritize in a more fine-grained manner. Certainly they like the ability to sort the search results, but maybe it's not as important as another item on the backlog that's completely unrelated to search.
Also, on the idea of programmers needing everything specified, try to look at it from the customer's point of view. A lot of times, it's like you're going in to buy a car, and the salesman is asking what color you want for the windshield wiper knob. A lot of details we might find important are completely irrelevant from a customer point of view. I've worked where requirements are highly over-specified, and trust me, it is not very fun. The kind of latitude you're complaining about, a lot of programmers would love to have.
I see some wrong assumptions in this question:
- code with design patterns, though applied correctly, needs more time to be implemented than code without those patterns.
Design patterns are no end in itself, they should serve you, not vice versa. If a design pattern does not make the code easier to implement, or at least better evolvable (that means: easier to be adapted to changing requirements), then the pattern misses its purpose. Don't apply patterns when they don't make "life" easier for the team. If the new Null object pattern was serving your friend for the time he used it, then everything was fine. If it was to be eliminated later, then this could be also ok. If the Null object pattern slowed the (correct) implementation down, then its usage was wrong. Note, from this part of the story one cannot conclude any cause of "spaghetti code" so far.
- the customer is to blame because he has no technical background and does not care about cohesion or the coherence of the product
That is neither his job nor his fault! Your job is to care about cohesion and coherence. When the requirements change twice a day, your solution should not be to sacrifice the code quality. Just tell the customer how long it takes, and if you think you need more time for getting the design "right", then add a big enough safety margin to any estimation. Especially when you have a customer trying to pressure you, use the "Scotty Principle". And when arguing with a non-technical customer about the effort, avoid terms like "refactoring", "unit-tests", "design patterns" or "code documentation" - that are things he does not understand and probably regards to be "unnecessary nonsense" because he sees no value in it. Always talk about things which are visible or at least understandable to the customer (features, sub-features, behaviour changes, user docs, error fixes, performance optimization, and so on).
- the solution for rapid changing requirements is to rapidly change the code
Honestly, if "underlying interfaces change twice per day for three months", then the solution should not be to react by changing the code twice a day. The real solution is to ask why the requirements change so often and if it is possible to make a change at that part of the process. Maybe some more upfront-analysis will help. Maybe the interface is too broad because the boundary between components is chosen wrong. Sometimes it helps to ask for more information regarding which part of the requirements are stable, and which are still under discussion (and actually postpone the implementation for things under discussion). And sometimes some people just have to be "kicked in their asses" for not changing their minds twice a day.
Best Answer
There are two terms for "cleaning up" depending on how drastic the changes are:
As part of normal practice, refactoring should be encouraged. Targeted rewrites will need a bit more planning to get incorporated.