STAREAST 2008: Understanding Test Coverage

Test coverage of application functionality is often poorly understood and always hard to measure. If they do it at all, many testers express coverage in terms of numbers, as a percentage or proportion-but a percentage of what? When we test, we develop two parallel stories. The "product story" is what we know and can infer about the product-important information about how it works and how it might fail. The "testing story" is how we modeled the testing space, the oracles that we used, and the extent to which we configured, operated, observed, and evaluated the product. To understand test coverage, we must know what we did not test and that what we did test was good enough. Michael Bolton proposes alternatives for obtaining and describing test coverage-diagrams, strategy models, check lists, spreadsheets and matrices, and dashboards--and suggests how we can use these tools to build a clearer understanding of coverage to illuminate both the product story and the testing story.

• Beware of expressing coverage numerically as a simple percentage
• Why code coverage is not test coverage
• Novel approaches to model test coverage