ISO 12207 has a rather non-standard (and very fuzzy!) distinction between verification and validation. In most circles, verification answers the question "did we build it right?" (does our product meet the requirements?), while validation means "was this the right thing to build" (do we have the right set of requirements?). ISO 12207 mentions requirements verification. How do you verify requirements, particularly the top level requirements? You don't. You validate them.
The product on which I spend the bulk of my time used to distinguish verification from validation. Problems with this distinction popped up on an annoyingly frequent basis. The distinction was clear to me, but my interpretation wasn't universally agreed upon. Is this a validation test or a verification test? Who cares? Why are we having this stupid fight? A rose by any other name smells just as sweet. The ultimate goal is to answer the question "Is it right?"
Our resolution to this ongoing imbroglio was to get rid of that distinction between verification and validation. We think of verification & validation as one word which together answer the question "Is it right?" We don't split V&V into arbitrary parts. What we split into parts is how we go about answering that singular question. Whether you're looking at our top level document or a detailed model document such as how we model gravity, chapter 5 is titled "Inspections, Tests, and Metrics". The old title, "Verification and Validation": It's gone. We don't split verification from validation. We instead split things into inspections, tests, and metrics. The chapter has typically has four sections, inspections, tests, traceability (typically automated), and metrics (also typically automated). Aside: Traceability analysis is just a specialized (but very important) kind of inspection. It's elevated as a separate section because of it's importance. We wrangled a tiny bit as to whether to call the chapter "Inspections, Tests, Traceability, and Metrics". The answer was no. No more religious wars over names. Traceability is a kind of inspection; the title still fits.
Things that answer the question "is it right" without actually using the product goes under the heading of "inspections". Requirements analysis, traceability analyses, analyses that demonstrate that the simplifying assumptions made by the model are acceptable for the model's intended use: They're all inspections because we didn't use the product. Traceability, while an inspection, has a home of its own. Using the product to answer that question go under the rubric of "tests". Everything from unit tests to tests against some artificial setup with a known analytic solution to integrated tests where we're comparing the results from the product to real data collected from some previously flown vehicle: They're all tests. Along the way we collect lots of metrics, many of them automated. Different levels of documents have different kinds of boring but important numbers. SLOC count, complexity, code coverage, number of waivers granted, number of issues in the CM system. Those boring but important numbers go under the rubric of "metrics".
Our developers and testers love this change because now it's crystal clear where things go. Project management loves this change because its consistent and because it removes all that silly wrangling over nomenclature. Our CMMI overlords love it because we're still answering the right question ("Is it right") and because we have a well defined process for getting there. Bean counters love it because we're giving them lots of beans that they are very easy to find and very easy to count.
Given the lack of this document how are we supposed to prove that the
system does or does not do what it's supposed to to an auditor?
Why is that a given? There is no rule of scrum that the only thing created is software. Make your acceptance criteria include "done when we have a detailed ISO 9001 compliant functional specification". Or, make that a separate story altogether: "as a developer on an ISO 9001 team, I need a functional spec so that our software is compliant". Create the spec, and then from that you can come up with the stories necessary to implement that spec.
In the comments to this question, @Pete expressed concern that if the spec were a story, how can the product be "potentially shippable" if all that is delivered after an iteration is a spec?
Look at it this way: your end product is composed of many pieces: A, B, C, D, etc. At the end of each iteration you should have one of those pieces finished and potentially shippable, right? When A is done it should be potentially shippable. When B is done it should be potentially shippable, and so on.
Where does it say that A must be a software product? A can be potentially shippable even if it is simply a functional spec, or a test case, or a DB schema, etc.
Don't look at your functional spec like it is somehow outside of your product. Instead, see it as an integral part of the product you are delivering, on equal footing with software, DB schemas, test cases, user guides, etc.
Best Answer
The scenario should be set up in the
Givenstep. This involves mocking parts of the system. However, a mock might be used in theWhenstep when that's part of the action.When reading your two scenarios, they both look reasonable as natural language:
However, the second variant does not conform to the Given-When-Then structure.
Both of your scenarios obfuscate the actions by using passive voice: “the button is pressed”. In my experience, it is better to name an actor who interacts with the system (third-person perspective) or to write the scenario from a first-person perspective: “When the driver presses the button” or “When I press then button”.
From this, it is clear that “the engine fails to start” is not an action – it is an outcome. But asserting that as an outcome makes no sense for the scenario that you're trying to test. That the engine will fail to start is an essential part of the context of your scenario, so it should be specified in the
Givenstep.I would write your scenario as:
Thanks to Greg Burghardt for pointing out in the comments that the dashboard – not the entire car – might be the system under test, so that the engine might be the actor. This would lead to the following, equally valid scenario: