Test Notes and Coverage Maps--Aids for Rapid Testing

In this article, we explain some of the tools and techniques that we have developed and used that have helped provide test support for faster delivery cycles for a range of highly configurable products. Some of the ideas and tools contained in this article come from training we received from James Bach as part of his Rapid Software Testing course, which has influenced many of the recent changes in our company's testing process.

We always have implemented exploratory testing at our company, but after meeting James Bach our approach techniques changed. We now view exploratory testing as a key part of our test strategy for most releases. It is a planned, structured, and managed activity that supports, and in some cases replaces, other test approaches.

Two key concepts we adopted from James are:

• Session-based testing and regular session reviews
• Concurrent test execution and test design as a core competency for testers

Use of Test Notes to Support Session-based Testing and Concurrent Test Design
One of the issues we faced with exploratory testing was how to track and report our test coverage.

Now we promote that our testers should consider the notes they keep during an exploratory test session a key test deliverable. We encouraged this in a number of ways, but most effectively by regular review of test sessions and the test notes produced during these sessions.

Of course, we are interested not only in recording what we tested, but also in helping our testers make good decisions about what to test and what not to test. One way in which we have tried to meet the need to capture test outcomes and promote good design practices during exploratory testing is through the use of a test notes template. The test note template prompts the tester to consider risks and associated tasks, capture new test missions, and identify test ideas that he wishes to explore during the current test session. The test notes template is not and should not be applied universally to each test session, but it always should be considered for application, as the template establishes common expectations about the kind of information that should be delivered. It also is a useful guide for good test design.
However, one of the first decisions a tester may make is not to use the template and to capture his notes in another form.

If the concept of concurrent test execution and test design is taken on board, then test managers need to be open to receiving and reviewing test notes that are captured in a variety of different forms. They then need to be able to ensure that the various customers of testing are able to use the information provided. This isn't so hard to do if you've already built a relationship of trust with your development team. Exploratory testing as a whole works much better if the developers are plugged in to your process. This is best achieved by having regular discussions with the developers and getting them to explain the code structure, while highlighting any areas that they are concerned about. Doing this establishes a good rapport with the developer.

Figure one shows a fairly typical example of a set of test notes for a relatively tightly constrained piece of testing.

Figure1. Typical Test Notes at Tait Electronics

Use of Coverage Maps to Support Concurrent Test Design
A "coverage map" representing various test scenarios also can be used to help capture test notes and encourage good test design.

In our vocabulary, coverage maps are diagrams that can capture a lot of information and generate many test ideas. Most of the coverage maps we have used to date capture test scenarios associated with transitioning between two known states of a single product feature or characteristic. The simplest cases of a state transition diagram we have developed have an appearance similar to a ladder-hence the name "ladder diagram."

Each leg of the ladder represents a particular state. The "rungs" represent input/output actions or, more preciously, an event. An arrow at the end of the line represents the flow of activity at any point of time. Any text above or below these action lines represents either the configuration state or the expected outcome, depending on its position in the diagram. The text in the ladder step boxes represents the parameter of an event, configuration state, or both.

Figure 2 is an example of a ladder diagram. It represents some of the test scenarios associated with the login page of a Web site. Steps can be added to the diagram as new scenarios are discovered. The process of dynamic test design can be captured easily. In the notes box of the diagram specific test scenarios are highlighted to help the tester to further explore the feature for better test coverage. By marking a tick against each scenario and adding comments during test execution can enhance this diagram as a quick test reporting tool.

Figure 2. Ladder diagram Web page login

Test notes and coverage maps have served as aids to our rapid test activity. They have increased our overall test efficiency and have allowed us to capture additional data about our test coverage. They also serve as a resource to capture and reuse good test ideas.

The test notes template was designed to support better test reporting and better concurrent test design. A coverage map can be produced for any test activity, and should complement the test notes template. Additional instructions, such as notes and comments, can enhance coverage maps further.

Acknowledgment
The authors gratefully acknowledge numerous discussions with their team members that helped to develop the materials in this article. They also thank Mr. Heiko Müller-Cajar, the test manager at Tait, for organizing James Bach's training. Special thanks to James Bach for his motivational presentation on Rapid Testing.

References

• Rapid SW Testing, by James Bach
• A Practitioner's Guide To Software Test Design, by Lee Copeland
• StickyMinds.com