I see many test strategy documents in my work with clients. Almost without exception, they are big—fifty or more pages—and stuffed with soporific boilerplate copied from one project to the next. Defect management process, test suspension and resumption criteria, entry and exit criteria, meaningless generic risks ... yawn ... again. One frequently used model even parrots textbook test definitions, from unit to systems integration to load.
I have several problems with test documents like these. In many organizational cultures, mandatory documents become confused with the things they are supposed to represent. A test strategy document is not a test strategy; it’s a document. It might or might not serve to communicate a strategy. The bigger and heavier a document is and the more replicated content it contains, the lower the likelihood it will convey anything important that people can take meaning from. In reality, the central ideas in most genuine test strategies can be expressed effectively in a couple of diagrams or a page or two of clear prose.
There’s a more basic problem. A big test strategy document rarely contains an actual strategy. That leads me to ask if, as testers, we really understand what “strategy” means.
I’ll come back to the documentation issues a little later. For now, let’s acknowledge that you need to communicate your test strategy to project stakeholders in some way and negotiate their agreement.
First principles—What Is a Test Strategy?
A strategy is the overarching direction or design of a campaign, whether that’s a marketing campaign, a football season, or a campaign of war.
A test strategy is the set of big-picture ideas embodying the overarching direction or design of a test effort. It’s the significant values that will inspire, influence and ultimately drive your testing, and the overall decisions you have made about ways and means of delivering on those values.
Note that was “overall” decisions. A strategy is not a detailed plan. It’s the design behind the plan that you, as a skilled tester, have thought about and developed.
Overarching Direction
Learning about the product and project, you develop ideas about how to test. What approaches are you going to use, and what potential test design do you already have in mind?
It’s unlikely you’ll be able to test everything in every way and to the same level of rigor. An essential strategy decision, therefore, is how to draw the boundaries for coverage and on what basis. (This is probably the most important element for stakeholders to understand.)
For example, will you prioritize coverage and types of testing based on risk? What do you mean by risk, and how will you identify and assess risks that could be mitigated by testing?
Who are your stakeholders? What does product quality mean to them in practical terms? How do they characterize the value they expect to get from the product? Not all stakeholders are equal—in most organizations, the concerns of some are more important than those of others. Organizations and individuals differ in their appetite for risk and their perception of what constitutes risk. These are critical considerations in determining what your test will cover and how.
Overarching facts, principles, and beliefs may drive test priorities or severely constrain the test. For example:
- This product will be used by emergency services. People could die or suffer serious injury as a result of product bugs. Ease of use is critical for some features; others must produce fast and accurate results with no tolerance level for error.
- The product is strategic to the business. Threats to the publicly advertised launch date will be catastrophic to the company’s reputation and share price.
- The product will operate within a strictly regulated environment. Certain types of bugs could result in prosecution of the company’s executives.
- The product is a website that will be hit on a specific date by thousands of users in an event-based marketing campaign. If it crashes or crawls, your company’s name will be mud and it will lose its biggest, most prestigious supplier.
Value descriptions like these provide the “why” for test strategy decisions. They may sound obvious, but stating project values up front gives stakeholders an opportunity to say whether you’ve based your strategy on the considerations that matter most to them.
Other strategy ideas will occur to you according to the risks, constraints, and opportunities you discover in exploring the project context. For example, would it make sense to use one or more high-level models, such as “financial year” or “customer experience lifecycle”? Will you use exploratory testing, pre-designed tests, or a mix? How might you sequence major test activities for optimum test value?
Communicating the Test Strategy
On most projects, negotiating agreement to your key decisions is essential for managing stakeholder expectations. Stakeholders need to know about barriers that could impede or prevent important testing, because they will have either to accept or remove them. Suppose you see no way to do year-end period testing on a financial system, or you find that company security rules prevent you running SQL queries on the central product database. If project timelines mean you can do only a superficial once-over feature confirmation on a complex application, say so up front and give stakeholders an opportunity to respond.
Rather than using a strategy document template to inform stakeholders, try to find a lightweight medium that will convey the central ideas clearly without the distraction of masses of detail. It could be the tool you’ve been using to work through your thinking. I have used one or some of: a mind map, sticky notes, a drawing on a board, a colorfully highlighted diagram, an outline document, or presentation slides. The choice depended on the organization I was working with and the nature of the test. Whatever medium you choose, it should work as a tool you can walk around with to talk to stakeholders, or use in meetings to promote discussion.
This isn’t to say that your existing template is useless, though it probably demands things you don’t need and ignores things you do need. You may be required to fill in its blanks for the final secure-it-in-the-vault artifact. But don’t start there. The template for a big document is a poor tool for promoting conversation and securing agreement to a strategy. I find it better not to constrain my thinking by starting with a template.
It’s About Thinking
The size and risk of what you have to deal with and how much is already known and acknowledged by the project will drive how much you do to outline your test strategy. (Don’t be surprised if you find yourself questioning some of what “everyone else” on the project believes.)
Whether it takes two days or forty, at the end of this process you will know your stakeholders and which values and principles are essential to your test. You will have defined the test’s high-level boundaries and approaches, and you will be in a position to draw stakeholders’ attention to any risks, issues or constraints that will impede testing within those boundaries, as well as any assumptions you feel able to make.
But don’t fall too heavily in love with your strategy, or create an expectation in your stakeholders that it’s fixed in stone. A test strategy can change substantially as the project changes and as you discover more about the product.
The key word here is “thinking.” Approach each test assignment as a new problem to solve, and recognize that you will be learning and solving problems to the end of the project. Even if you are testing a well-known product on a path you’ve traveled before, consider standing back and taking a fresh strategic look. It can stimulate new ideas that could improve your testing and save your project time and money.
Acknowledgements: Fiona thanks colleagues Anne-Marie Charrett, James Christie, Karen N. Johnson, James Lyndsay, John Owen Thomas, and Peter Welan for stimulating conversations and reviews that helped to inform this article.
