Agile – Disadvantages of vertical user stories

You're missing the point of agile a little bit. What you are calling a user story, I see as at least six: one bare-bones search, and one for each of your bullet points. By all means, make enough plans to avoid painting yourself into a corner that's going to be expensive to get out of, but the whole idea is that you provide the minimum needed to deliver some value, and do it quickly enough to get rapid feedback.

When you divide a story up like that, it not only makes it easier to estimate, it allows the product owner to prioritize in a more fine-grained manner. Certainly they like the ability to sort the search results, but maybe it's not as important as another item on the backlog that's completely unrelated to search.

Also, on the idea of programmers needing everything specified, try to look at it from the customer's point of view. A lot of times, it's like you're going in to buy a car, and the salesman is asking what color you want for the windshield wiper knob. A lot of details we might find important are completely irrelevant from a customer point of view. I've worked where requirements are highly over-specified, and trust me, it is not very fun. The kind of latitude you're complaining about, a lot of programmers would love to have.

This seems to me to be the unspoken elephant in the room with Agile projects: how do you prevent them from evolving into chaos?

Let's look at the Agile Manifesto for a moment. Agile desires:

1. Early and continuous delivery
2. Embracing changing requirements
3. Delivering working software frequently
4. Developers and business stakeholders working together daily
5. Building projects around motivated individuals, giving them the environment and support they need, and trusting them to get the job done
6. Face to face conversation as the primary mode of communication
7. Working software as the primary measure of progress
8. Sustainable development
9. Continuous attention to technical excellence and good design
10. Simplicity - Maximizing the work you don't have to do
11. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly

I've highlighted the last four. Why? Because they're the ones that will prevent the project from collapsing under its own weight.

Sustainable development means that you (hopefully) have someone that oversees the overall development process, someone who can steer the ship beyond growing the software a little bit at a time, someone who has an overarching vision of the entire project, someone with a keen knowledge of software architecture, system design, business logic principles and UI ergonomics. An architect, in other words.

The architect is the one that says "I know you asked for this, but if we build this other thing, we can avoid these other three features that are just confusing, and focus on the core requirement." He's the one that gives the team time to reduce technical debt. He's the one that brings an overarching structure and organization to the project. He's the one that calls meetings where the team gets together and asks "How can we do things better?"

And he's the one that maintains the master requirements document.

In the book Mastering the Requirements Process, the authors maintain that there are three kinds of customers, three kinds of software projects: Rabbits, Horses and Elephants. Rabbits are the small software projects that only need informal requirements; you work directly with the customer in small sprints, the scope of the project is reasonably constrained, and the software gets done within a relatively short time frame. Elephants are those mammoth projects that require a lot of up-front planning, a tremendous amount of documentation, and a long time horizon. They are arguably not agile, although you can break them up into smaller projects that can be built using agile.

It's the Horse projects that are somewhat confounding from an agile perspective. They can still be built iteratively, you can still use short sprints, but without some discipline in the requirements gathering and planning processes, they can easily run off the rails. These are the projects that can benefit from a disciplined requirements gathering process, and then careful adaptation and modification of those requirements as the project grows, while still maintaining an overarching vision for the entire project.

From a personal perspective, I work with a small team of about a dozen developers. At any given moment, we might be working on three software projects at the same time, ranging anywhere from a few thousand lines of code to over 100,000. Our customer thinks they are a rabbit, but they're really a horse. The customer is engaged, but not on a daily basis.

By far our weakest area is gathering specific, testable, measurable requirements and managing the customer's expectations relative to project scope. But we're working on that.