The fact that you test an application extensively does not itself render the application more stable. However, NOT testing an application increases the risk that the software may not comply with the requirements and won't necessarily provide the expected business value. Here is a method for approaching risk-based testing.
In the ten years that I have worked in the software industry, I have come to realize that there is a widespread, but essentially wrong, belief amongst most software companies that software quality is a function of testing. The fact that you test an application extensively does not render it more stable, nor does it provide added value to the customer. How many times have you heard, "This software is not stable enough, we should have had more QA."? Unfortunately, testing has nothing to do with stability. You test to ensure that the software functions as it was initially requested by those who dictate the requirements. Therefore, not testing software increases the risk that the software may not comply with the requirements and won't necessarily provide the expected added business value.
This leads us to risk-based testing, which I would describe as Steve Wakeland defines IT risk, as 'the likelihood that a program fault will result in an impact on the business'. Further, I would specify a NEGATIVE impact on the business. Let me explain.
I classify bugs with severity ranking, as do most people, and define severity by measuring the negative impact a bug has on the business. The severity ranking is usually high, medium or low. High meaning the customer cannot work with the software and wastes time and money, medium meaning the user has to use some alternative workaround to achieve the aimed goal and low meaning the customer can STILL work with the software (feel free to specify additional levels for a more granular ranking). Thus, if I classify bugs found by QA in terms of severity by measuring (or guessing) the negative impact it has on the business, then I should focus my testing using the same mindset, i.e. negative impact.
Thus, testing and risk (as defined by Steve Wakeland) are related. How? Is it a straight line as shown in Figure 1? In that case, the only strategy is: Test anything; it's sure to lower the risk.
Figure 1
Ideal Relationship
Pareto was probably right when guessing that 20% of the total feature set allows a user to do 80% of his/her work. What if the relationship between the testing effort and risk was more like Figure 2?
Figure 2
We can define a better testing strategy that optimizes the QA effort to minimize risk. It is important to note that no matter how much testing you invest, there is always a risk. Therefore, releasing the software is directly related to the level of risk you are willing to accept.
However, if you see testing as an investment, the strategy becomes: What is the minimum testing effort that I should invest in order to maximize risk reduction? In other words, what should I test first to be in Zone A of Figure 2?
I control my testing effort with a simple methodology based on usage. Although users have their own way of using software, you can usually predict the top-ten features list that allows them to do most of their work.
Methodology
Step 1: Identify the 'vital' functionalities that could prevent the user from using the software if a bug was found. (This bug would be a high severity.) A good example would be a login page for a Web application that does not work. An efficient way to gather this list of functionalities is to survey the end-user community, ask a domain expert, or assemble statistics from the log of a previous version of the application. Since the risk increases with the frequency of use, you should look at the most used feature by a user to identify the riskiest ones.
Step 2: Design and then assign test cases to each functionality listed in Step 1.
Step 3: Size (in hour or minutes) the QA effort required to run the test cases identified in Step 2.
Step 4: Sort test cases in ascending order of effort so you have the test case with the minimum effort first.
Step 5: Start running test cases in the order established in Step 4 until you run out of time.
A Word about Automation
Ideally, you always want to lower the risk in the shortest period of time in order to release versions more aggressively. One way to shorten your QA cycle and still having the same confidence level in the software is to automate with QA tools for both functional and unit testing the minimum required QA effort to reach Zone A in Figure 2.
Let's say I want to implement two new features in my software. I have the choice of implementing the two features in the same version or implementing each feature in two successive versions. From a QA standpoint, having two successive versions with the same confidence level has a huge impact on the workload unless you automate the test cases listed in step 4 of the above methodology.
Conclusion
Looking at Figure 3, where the relationship of Figure 1 and Figure 2 have been superimposed, clearly shows that lowering the risk to 50 percent can be achieved in a shorter period of time if testing prioritization has been done with risk in mind.
Figure 3
Further Reading
Steve Wakeland: Testing as a component of an organizational IT risk management strategy. Cutter IT Journal, August 2002
