Many options are available for test teams to help them document how a system should work. A test strategy, test plan, test charter, test cases, test scenarios, and automation scripts are examples. This article has a matrix comparing the types of test documents you might choose and can help you pick which is right for you based on project characteristics.
The test team has an obligation to collaborate, delegate, and communicate what it is we do. One form of communication is through documentation.
I have to admit that the word documentation makes me wince. It has been the source of so much overhead and waste, making test teams seem slow and cumbersome. So, what’s the right documentation strategy for the tester or test team?
Many options are available for test teams to help them document how a system should work or how it meets the expectations of the client. A test strategy, test plan, test charter, test cases, test scenarios, and automation scripts are examples of several. There are systems to manage this information, ranging from open source to high-dollar, that integrate with every conceivable step of the software development process.
Resistant to drawing a line in the sand about specific document use, I worked with a few other great minds—Michael Corum, Vicky Roberts, and Matt Heusser—to devise a decision matrix for test documentation. The teams I support are diverse. I needed an enterprise solution and a tool for communicating that vision that enables the teams to make autonomous decisions about the documentation they create.
About the Matrix
The matrix compares the types of test documents you might choose based on project characteristics. It serves as a reference tool to set expectations and prompt discussion, but it is not a directive.
Let’s start by defining what I mean by each of the documentation types listed along the x-axis of the matrix. They may not be perfect, but they get the job done.
Document Type | Definition |
Test Strategy | Describes the approach of how testing will be organized, tool usage, priority handling, team structure and resource needs, environment needs, schedule, metric needs, etc. |
Test Plan | Documents the methods, coverage, and who will perform what in a testing effort |
Test Charter | Also known as a session charter. A "feature x risk" list for a single story that defines a tester's mission |
Test Scenarios | Scenario-based executable activities. They can vary in granularity, such as business requirements, acceptance criteria, and user scenarios. |
Test Cases | A formal or informal unit of measure that is used by a tester to determine if a feature accomplishes what it is intended to do. Test scenarios, acceptance tests, and business requirements are all varying degrees of measurement. Formal test cases often contain steps. |
Automatic | Many types of tests can be documented solely in an automation script such as Selenium or JMeter, an output log such as URL lists, a behavior-driven script such as Cucumber or FitNesse, or reporting from a speed-monitoring site such as SpeedCurve. These are tests that do not need manually written supporting documentation. |
Ticket Description | Within a development tool, such as Jira, VersionOne, or Rally, this is testing information added to the ticket. |
Memory | Your noggin |
The y-axis lists common characteristics you may experience on a project, such as the characteristics of the work, type of teams, type of testing, and schedule. Test documentation on a project should reflect the value it adds to the project. Rolling up projects into project characteristics ranging from “no project” to highly intensive projects helped cut down on the complexity behind the visual representation. A project can benefit from one or many documentation types.
The Test Documentation Matrix
With this matrix as a basis for strategy, the tester or test team has the freedom to choose not only the type of document that has the most value for the project but also the medium that will work for them. The teams that deal with small projects and are collocated could use a whiteboard effectively, while a geographically diverse team handling a yearlong project likely needs a different solution. Characteristics of shareability, reusability, security, and traceability affect the document method.
A Year Later
I created this matrix for a discussion with my team more than a year ago. The conversation it provoked was engaging, with a lot of agreement, some disagreement, and plenty of moving boxes around the graph. The end result is not a directive; it’s guidance that empowers people to create the documentation that feels right.
In practice, we never looked at the matrix again. The matrix was valuable as a one-time-use document to convey a vision, and this vision is still a part of the culture we have today. As President Dwight D. Eisenhower and many others have said so well, planning is valuable, even if, when the action starts, the plans themselves are useless.
Our goal was to move beyond documents written from heavyweight methods (or no method at all) to something that is flexible and valuable. I think we succeeded.
How do you use documentation? Does it work for you?
User Comments
Nice article! An interesting overview of large, corporate project testing. I think some of these strategies can also apply to smaller, web-based application projects - giving the customer a good feeling about the professionalism of the developer's capabilities.
Well explained the Matrix of test documentation. It must have required lot of research and energy to present this in article. Keep Posting Tara!