What Is the Cost of a Requirement Error?

Many studies have shown that requirements errors are very costly. By one estimate (in an article by Donald Firesmith for the Software Engineering Institute), requirements errors cost US businesses more than $30 billion per year and often result in failed or abandoned projects and damaged careers. The common wisdom is to find and fix requirements errors early in the lifecycle of a project, but that is easier said than done. Furthermore, the actual cost of a requirement error has been hard to quantify in the past. This resulted in a "business-as-usual" approach rather than proactively creating programs to find these errors early.

If you are a business executive, IT manager, project manager, or business analyst, you need a way to calculate the cost of requirements errors. Once you know that cost, you can make controlling faulty requirements an appropriate priority.

You can calculate your cost of a requirement error based on the simple formula presented in this article. The formula is based on extensive practical experience, as well as data from a wide variety of industries and software development environments.

Background
The cost to fix a software defect varies according to how far along you are in the cycle, according to authors Roger S. Pressman and Robert B. Grady. These costs are presented in a relative manner, as shown in figure 1.

The relative costs shown in the chart above will be important in a later calculation, as you move from relative costs to a realistic approximation of the cost to find and fix each defect.

A. Estimate the Number of Screens and Reports in Your Final Product
If you don't know the number of screens and reports you expect to have in your final software deliverable, then use the visual representation of use cases to help you count them. The best way is via responsibility diagrams. You can most easily accomplish this by using a software product that automatically produces the diagrams from the textual use cases, but you can also do it manually by just listing the functions or activities that each actor performs in each use case. Some of these activities will involve displaying screens or running reports for users, and this makes the number of screens and reports easier to count.

B. Multiply by Four Person-Weeks
Development Effort = 4 pw × (# screens + # reports)

For now, you can take this estimating technique on faith (or use your own estimating techniques if you prefer). This rule of thumb has been proven in projects ranging from $100,000 to $20 million, and it has worked across development environments from the latest object-oriented techniques through 4GLs, Visual Basic, COBOL, and assembler. How can it work across different software environments? Well, for example, COBOL is a less-efficient environment in which to program than C#, but a COBOL screen is typically a lot less complex than a C# screen.

This estimate is expressed in terms of typical US-based development teams with a cost of $2,500 per person-week of effort (salary plus benefits). In other words, it costs $10,000 per screen or report to develop software.

Step 2-Estimate the Number of Defects to Expect at System Test
Your defect rate will vary depending on your level of quality control in the development process. The author's team that won the ITAA Total Quality Award for Software averaged less than 0.1 defects per person-week. However, the industry average is 1.0 defect per person-week, so that makes it simple:

# Defects = # pw

Step 3-Estimate Number of Requirements Defects
Various industry surveys indicate that half of all defects are related to requirements. In the absence of your own specific measurement, use the 50 percent estimate:

# Requirements Defects = 0.5 x Total # Defects

Step 4-Estimate Rework Cost
If you don't already measure your percent of rework on development projects through an accurate time accounting system, then estimate where you stand in the industry spread of 20 to 40 percent of the development effort that is rework. For example, assume that your project will come in at a 30 percent rework rate:

Rework Cost = 0.3 × Project Dev Cost

Step 5-Estimate Amount of Rework Cost due to Requirements Errors
The Chaos Report and Dean Leffingwell's Managing Software Requirements indicate that about half of all software defects are due to missing or bad requirements, but the cost of finding and fixing requirement defects is higher than that for other kinds of defects. In fact, they indicate that 70 to 85 percent of the rework cost is due to requirements defects. For our example, you could use 75 percent (or use your own measurement):

ReCost of Requirements Defects = 0.75 × Rework Cost

Step 6 - Divide Cost of Requirements Rework by Number of Requirements Defects

Cost per Requirement Defect = Cost of Requirements Rework / # Requirements Defects

Example
You develop the initial requirements for an applications software development project. You estimate that the combined total of screens plus reports will be twenty-five. You multiply by four person-weeks to get one hundred person-weeks as your development effort.

You now expect to have one hundred defects (one defect per person-week) at your system test or user acceptance test.

You estimate that half of those, fifty, will be due to requirements problems.

Now you estimate your rework cost at 30 percent of the initial development cost. You average $2,500 per person-week ($100,000 per year of salary and benefits, with a utilization factor that yields forty person-weeks of real development work per year per developer). So your development cost will be 100 pw x $2,500 = $250,000. Your rework cost will add 30 percent to that:

0.3 × $250,000 = $75,000 rework cost

Total development + rework cost = $325,000

Next you estimate the amount of rework that will be due to the requirements errors:

0.75 × $75,000 = $56,250

Finally, you divide this rework cost by the number of requirements defects to determine the cost per requirement defect:

$1,125 = $56,250 / 50

Back to the $30 Billion
Recall the statistic that said that the US wastes $30 billion annually in rework due to requirements errors. That works out to 300,000 person-years of effort every year ($30 billion /$100,000). Our statistics indicate that these 300,000 person-years implies that there are 1.33 million software engineers and programmers in the US as you can see from this equation:

300,000 person-years = 0.75 × 0.30 × (total person-years)

In the above equation, the 75 percent is the proportion of rework cost due to requirements errors, and the 30 percent is the proportion of development effort that goes to rework.

This yields the total number of person-years expended per year (i.e., the number of people) engaged in software development in the US:

Total U.S. software developers = 300,000 / (0.75 × 0.30) = 1.33 million

This relates very well to similar information from the Department of Labor's Bureau of Labor Statistics.

Now What?
If you run the example project on a business-as-usual basis, you will incur $75,000 of rework. Of that number, $56,250 will be due to requirements problems. You will pay $1,125 to fix every requirement problem that you find in system test-and even more if defects pass through the system test.

Now recall the chart showing the relative cost to fix a defect at various stages. There is a 15:1 advantage to finding and fixing the requirements errors prior to coding. If you could detect those requirements problems early, you could pay the discounted price of $75 = $1,125 ÷ 15.

Reducing Requirements Errors
One approach is to take advantage of the high energy typically present during the requirements elicitation stage. This is where people believe their inputs actually will have an impact on the requirements. Note that as soon as you produce a formal specification document, the stakeholder participation energy reduces dramatically. How many people actually read those weighty documents once they're published?

• Write simple draft use cases while doing the elicitation-don't worry about fancy "pre-conditions," "post-conditions," or "\"alternate flows."
• Create diagrams at the push of a button.
• Automatically detect all logical errors (called "specification errors"), and fix them on the spot with help from the system.
• Collaborate with the stakeholders and subject matter experts during elicitation to inspect the requirements visually by walking through the diagrams and making changes on the fly. This can be done in a joint application development session.
• Elaborate your draft use cases on the fly during the elicitation sessions by typing more detail, adding subprocesses that are automatically linked, and updating the diagrams at the push of a button.
• Perform a final visual inspection using a presentation mode to walk through the activity diagram step by step with the validators.
• Request comments from other stakeholders. Your requirements automatically are stored on a collaboration server, and now you can gather comments from other stakeholders who can see the requirements and diagrams in a browser-based interface with the server.
• Track the rate of change of requirements through a browser-based report (e.g., a "requirements governance report") to reduce project risk and to know when to start design and coding for this iteration's baseline.
• Produce the final requirements specification document, and then publish the final document to a specification library on a collaboration server.

Conclusion
You now have a simple, practical way to estimate the cost of finding and fixing a requirement error at system test. You also can estimate the size of an upcoming project and the number of defects to expect. If you choose to implement a modern technical solution, you may well be able to achieve a steep reduction in the cost of finding and fixing requirements errors and to achieve better project outcomes.