This column asks the all-important question, "What isn't there that should be?" The same idea for spotting black holes also applies to spotting "holes in designs and requirements." For example, there are often connections between the quantity of bugs filed against an area and whether the area is thoroughly tested. There can also be holes in what KIND of bugs have been reported. Hendrickson lays out the territory for the search and goes on to suggest how to "look for where there's a lot of nothing."
"Dad, how do they find black holes?" I asked this question many years ago while walking with my father on a clear winter night, admiring the stars. Dad looked down at me, hands stuffed into his pockets, our breath visible in the cold air, and smiled. "Well, kiddo, since they can't see what's not there, they look for where there's a lot of nothing."
The idea stuck: find something by watching for nothing. It's how I look for holes in designs and requirements. If nothing in the design says anything about security, no one is thinking about it yet. If there are no requirements from a given stakeholder, chances are that stakeholder has not been represented in the requirements process.
It's also one way that I monitor the testing effort. If an area has no bugs filed against it, it's an indication that it hasn't been tested. Of course, just because a lot of bugs have been filed about a given area doesn't mean that it is well tested. So I also look at the kinds of bugs that have been filed. Do they get to the heart of the functionality or are they all superficial? Do any bugs involve bad input data, buffer overruns, or long path names? I'm looking for an indication that there's a hole in our testing—kinds of bugs that haven't been filed yet.
Looking for the absence of information isn't easy. You have to know what you expect to see before you can notice that it is missing. That means that you need a mental list of bug categories you might expect to find in the software under test: simultaneous user problems, data corruption bugs, timing issues, etc. Your list depends on what your software does.
You also need to know, in detail, what has already been seen. If you survey the bugs that have been found to date—reading them rather than counting—you can look for patterns in the testing that led to finding the bugs. Just counting isn't enough. What's missing doesn't always fit neatly into a drop down list in the bug tracking system.
As the project progresses, looking for patterns of missing bugs becomes more difficult. The more bugs that have been filed, the more difficult it is to spot the areas with a dearth of bugs. And yet this is the time when it is critical. You're almost done. Soon, someone along Executive Row will start asking why you haven't shipped yet. The next thing you know, the software is released and/or posted to the Web. That's not the time to find out that no one tried searching the catalog in two different Web browsers simultaneously.
One way to look for testing holes is to tabulate bugs on a matrix as they're filed in the bug tracking system. Along one side, divide the software under test into areas. Then list categories of tests that apply to all areas along the top. For example, if you are testing an editing program, the areas along the side might include Draw Tool, Text Tool, Insert Picture, Printing, etc.
Categories along the top might include Undo, Drag and Drop, International Characters, Low Memory, etc. You could have as few as ten items per side, or as many as thirty. (Too few cells in your matrix and you won't get the information you need; too many and your matrix will become unwieldy.) Every time someone files a bug, put a tick mark in the appropriate box in your matrix. You're looking for boxes with no tick marks.
This matrix is a lightweight tool for your own use in spotting black holes. Don't try to put too much information in each cell. Even trying to squeeze in the bug number might be too much. If you try to track any more information than a single tick mark, keeping the matrix up to date will become onerous and you probably won't do it. If you just use tick marks to count bugs in cells, five or ten minutes a day of reading through bugs and marking off cells will be enough.
Months into the project, you'll have a map that shows you bug counts by areas of the software and categories of tests. Now you can find the holes. An added bonus: When the project is over, the matrix can also provide information about weak areas on the software so those areas can be improved in future projects. Did every feature have problems with boundary values? Perhaps the software needs more error checking in general.
Finding something by watching for nothing is difficult and time consuming—but worth the effort. The areas where we left unfilled holes are the ones that will cause us to run around with our pants on fire, desperately trying to ship a patch to an irate customer whose system blew up on upgrade, or frantically posting a new Web site that works with different browsers. So look carefully—what isn't there that should be?
