The mood in the meeting was grim. All eyes were trained on Doris, the head of customer support. Doris surveyed the room as she spoke..."We have a problem in the field with the new release. Sixty-three users reported unrecoverable errors this week—a record high. An additional 152 people reported crashes, but the software recovered after reboot. This morning, I talked to an irate user who said he'd uninstalled our software after it crashed on him five times in a row. He wanted us to give him a full refund plus expenses. In short, the users are really angry. What do I tell them? When will we have a fix?"
Silence.
"What the heck is going on out there?" the CEO snapped.
The VP of Engineering clenched his jaw. "I'll get right on it," he replied.
This scene has been replayed in software development organizations since the dawn of the information age. How does bug-infested software escape into the real world, causing users and technical support representatives such grief?
A common response is to blame the Test group: "How did you guys not find this bug?" This blaming stance puts testers on the defensive: "Maybe it's because the programmers gave us rotten code, management dictated the ship date, and no one listened to what we told them about the quality of the software!"
Let's suspend the attacking and defending for a moment and look at how bad software escapes. If there's a critical bug in released software, one of two things happened:
- No one found the bug before release.
- Someone found the bug but no one fixed it before release.
None of us likes it when bugs in the first category bite us, but it's the bugs in the second category that hurt the most.
Consider the history behind Doris's report. The testers reported that the software under test occasionally crashed for no apparent reason. The problem occurred very rarely, once or twice a week during peak testing times. The testers could not determine the circumstances under which the crash would occur. The programmers looked at the problem and couldn't figure it out either. The project team reluctantly deferred the issue to make the ship date.
Within a month after release, Doris was completely fed up with handling complaints from irate users and took her concerns to the executive staff.
Predictably, Doris's report spurred a great deal of finger pointing. The testers, who reported the problem, blamed the programmers for not investigating it thoroughly enough. The programmers blamed the testers for not being able to reproduce the problem in the lab. Everyone blamed Management for absurd schedules. None of the blaming led to better software. In fact, the blaming led to worse communication and schedule slips as team members scrambled to cover their backsides.
It doesn't have to be that way.
Everyone on the project can play a role in preventing bad software from escaping.
So what advice would I have for the testers at Doris's company for their next release?
- Stop the blame: Blame gets in the way of learning. Don't accept blame and don't dole it out.
- Find extreme cases: Find ways to turn minor glitches and occasional anomalies into major problems or frequent errors—even if that means doing things "no user would ever do." Users have a funny way of doing what no one expects. Although the testers at Doris's company were never able to make the software crash reproducibly, they later learned from field reports that the crashes seemed to occur more often when the system was under load or the user did things at unexpected times. The testers then used this information to make the software crash consistently in twenty minutes instead of once a week.
- Explain the impact of bugs: Make sure the bug report describes the effect of the bug on the end user. A bug titled "uninst.isu file missing" may be correct, but a bug titled "user cannot uninstall" is much more compelling.
- Champion serious bugs: If you believe that a bug will cause serious problems in the field, fight for its correction. Don't accept excuses like "too risky to fix," "no user will do that," and "there's a workaround" at face value. Ask questions. "But what's the risk if we don't fix it?" "How do you know no user will do this?" "What if the workaround isn't acceptable to users?"
As a tester, I get a sinking feeling in the pit of my stomach when I hear that there's a problem in the field with the software I tested. I feel even worse when it's a problem we knew about but let go because it seemed mostly harmless.
Now I look for fangs and stingers on all bugs, no matter how innocuous they seem at first. "Oh, that's cute. You can make the program do something it shouldn't," becomes "And I wonder what side effects that might have? Let me just edit this thing here and delete that thing there. Oh look! Corrupt data!"
Whenever I'm able to expose a harmless-looking, gnat-sized bug as a blood-sucking monster, I cackle. I can't help it. I'm just elated to have thwarted the bug. "That's one more nasty defect that won't make it to the real world to hurt users. Mwa-ha-ha-ha-ha-ha." Blaming is counter-productive, but quiet cackling in the privacy of your workspace can be downright therapeutic.
Just don't let the programmers hear you.
