Defect containment metrics, such as total defect containment effectiveness (TDCE) and phase containment effectiveness (PCE), can be used to give a good indicator of the quality of the process. TDCE captures the defects that are captured at some point between requirements and the release of a product into the field, indicating the overall effectiveness of the entire process to find and remove defects. PCE provides more detail at each phase of the software development life cycle and how the defect detection and removal techniques are working.
Applying these metrics makes sense at a level where you have a well-defined process and methodology for product development, often a project. However, some organizations provide a process framework that is tailored at the project level. This process framework would include the necessary guidance for meeting certifications (ISO9001, CMMI), practices for incorporating known good techniques (agile methods, Lean, Six Sigma), and requirements for legal or regulatory reasons. However, the specific details of how to gather requirements, design the system, produce the software, conduct test, and release are left to the product development teams.
Is there any effective way to apply defect containment metrics at an organizational level when only a process framework exists at the organizational level? If not, what might be some ideas for metrics that can be distilled from each project (each using a tailored process that fits into the organizational process framework) that captures defect containment metrics to discuss the ability of the process to find and remove defects?
The end goal of such a metric would be to consolidate the defect containment practices of a large number of ongoing projects and report to management. The target audience would be people in roles such as the chief software engineer and the chief engineer (of all engineering disciplines) for the organization. Although project specific data would be available, the idea is to produce something that quantifies the general effectiveness of all tailored processes across all ongoing projects. I would suspect that this data would also be presented as part of CMMI, ISO, or similar audits to demonstrate process quality.
Best Answer
Consolidation of defect management sounds like a good idea unless the projects are so orthogonal as to require separate defect managing systems. For instance, a solo developer doing his own testing could get by with bullet list, while a distributed organization with customers reporting bugs to a support team requires a much more heavy-weight process. This is obvious, but it is important to consider how your consolidation will effect all the developers involved and whether they will be motivated to buy into it.
The problem with bug metrics is if developers feel that bug counts are a way of measuring their performance, then they may get fudge numbers (grouping multiple bugs into one, underestimating bug impact, saying they caught the bug further upstream). This is especially a risk if you present the reports to management. Even you get support from all the developers there can be large variances simply from how they individually report bugs.
Defect containment metics, in particular seem to hold the assumption of an exponentially increasing cost curve downstream. While this may be the case, agilists make a convincing arguments for changing the cost-curve rather than stopping defects as early as possible. I would be skeptical of any metric that assumes a fixed cost-curve, since each project and type of bug may have a dramatically different cost-curves.
If it is dramatically more expensive to catch a bug downstream, then lowering the cost curve by adopting better tools, adopting continuous integration, automated testing, or other agile practices may be a better solution. Of course, in some projects there may be large unavoidable downstream costs (e.g. space shuttle code) in which case defect containment metrics make more sense.
Otherwise, useful 'metrics' in my opinion are categorizing bugs by type to help analyze any systemic flaws. Tracking where the bug was caught (development, integration, or in the field) is also important for analysis, but as I said before I would be hesitant in plugging them into a fixed cost curve to get a single number. Instead cost to fix bug can be estimated by tracking the hours a developer spends on it using a tool such as Mylyn, although this requires some discipline on the developer's part.