When do your testers join the software development process? If you're like most teams, testing doesn't start until late in the development cycle. Once testing is engaged, you work around the clock finding the important bugs, fixing those you can fix, and trying to get the code ready to ship. In all, your testers and developers will put in several extremely late nights and probably a few weekends to get the code out the door.
Last-Minute Heroes
Testing that starts late in the software development cycle gives many bugs that were originally small and simple a chance to become deeply entangled with other bugs in the system. These entrenched bugs are often difficult to find and nearly impossible to fix well.
On the bright side, everyone who survives the dreaded days of debugging will likely be proclaimed a hero. But I wonder if heroism in these situations is as necessary as we are often led to believe.
Let's step back for a moment and look at similar heroics in a totally different context. Smokejumpers are firefighters who parachute into forest fires armed with shovels, picks, and little else. This matches up pretty well with my idea of heroism. Their work is important, exhausting, and dangerous. Fatalities occur on a regular basis.
But an ounce of prevention may really be worth a pound of cure--or hundreds of acres of forest. Some communities have begun using goats to deal with wildfires. Instead of waiting until a fire starts, these communities hire goats to munch away on the underbrush. This starves a potential fire of the fuel that would allow the flames to spread. This is not glamorous work, but the goats effectively clear many acres in a short time.
 |
| Mike McMillan/Spotfire Images |
Goats vs. Smokejumpers
- Goats work cheap. 650 goats probably cost as much as one firefighter.
- Goats are not nearly as exciting as smokejumpers. Watching goats munch on underbrush is only marginally more interesting than watching the underbrush grow.
- Using goats requires planning. Goats are of little use once the fire has started. (Unless you plan to feed them to the firefighters.) To be effective, they must be used before the emergency hits.
- Goats are not a complete solution. Fires will still happen and you will still need firefighters-hopefully not as many or for as long.
- Measuring the success of goats is tricky. You can't talk about fires extinguished; you have to talk about fires prevented. Fortunately, if you have some idea of how many fires would usually flare up, you can get an estimate of how helpful the goats have been.
Now, let's return to your team's software development efforts. On your last project, did you get pulled into firefighting mode because your testing found serious bugs only late in the development cycle? Did you repeatedly send in your developers to contain and debug the problems? I'll bet your developers worked mainly in reactive mode, moving from one hot spot to the next since they had no time to create long-lasting solutions. In other words, your people were heroes heading straight into burnout.
But what if your team had taken a more goat-like approach, stopping small, isolated bugs before they could become large, tangled bugs? What if your developers and testers used cheap, simple methods during the development phase to lessen the presence of bugs later in the process?
 | |
| Reused with permission: San Diego Union-Tribune | |
The truth is we have had such tools and methods around for a long time. We just haven't always used them as well or as often as we could have. For instance, to name just a few:
- Code reviews and inspections. Code review is probably not on anyone's list of exciting ways to spend a few hours, but its reported 70-80% find rate certainly deserves attention.
- Automated Inspections. For the lazier among us, there are tools that will automatically inspect code for suspicious constructs.
- Unit testing. With the emergence of extreme programming, unit testing is now in fashion.
Early Detection vs. Late Night Heroism
- Early detection tools cost much less than finding showstopper bugs at the last minute.
- Early detection methods find bugs in a simple and boring way, which is good. Software development currently has too many nail-biting moments and death marches.
- Early detection tools require planning. Once you are in reactive mode, it's too late.
- These tools do not find all bugs. You still need testers and developers to fix bugs-hopefully not as many since the number of bugs will be smaller.
- Measuring the impact of these tools can be tricky since their real value is in preventing worse bugs down the line.
Keeping It All Under Control
Being a hero is great, but only if your heroism doesn't involve a situation you could have avoided in the first place. Good engineering doesn't consist of random acts of heroism. Good engineering should celebrate a subtler form of accomplishment.
Further Reading
- For quick background on the effectiveness of code reviews, read "Inspections - Evolution and History: 1972-2001" by Michael Fagan
- JUnit and www.NUnit.org provide good starting points for finding out about the latest craze in unit testing.
- For astounding photos of smokejumpers in action, visit Mike McMillan's website, www.spotfireimages.com.
- For un-astounding photos of goats in action, visit www.signonsandiego.com/uniontrib/20040416/news_1n16goats.html
- And, finally, "Nobody Ever Gets Credit for Fixing Problems that Never Happened," at web.mit.edu/nelsonr/www/Repenning=Sterman_CMR_su01_.pdf is a good read about making sure that prevention techniques get recognized, rewarded, and sustained.
User Comments
Americans as a people are problem solvers, not problem preventers. We happily rebuild homes destroyed by hurricanes, brushfires, and mudslides. In their original locations! And our banks, insurance companies, police, firefighters, and relief agencies happily subsidize this insanity! (One definition of "insanity" is: "The belief that if you keep doing what you've always done you will somehow stop getting what you've always gotten.")<br/><br/>This archetype has been around for so long that it's got a name: The Loose Cannon Syndrome. Named after an infamous event on a British warship of the ropes-and-canvas era. As a storm approached, a young officer failed to double-check the knots he had tied when lashing a cannon to the deck. The pounding of the storm quickly undid the knots and the cannon began bouncing around the deck, smashing up the superstructure and creating the terrifying possibility of bashing through some really important timbers that would allow the ship to take on water.<br/><br/>The officer gallantly dashed out onto the heaving deck and, risking his life at every turn, managed to single-handedly immobilize the cannon and secure it properly.<br/><br/>When the storm had passed the captain assembled the entire crew on the deck for a ceremony to give this performance its due. He pinned a medal on the chest of the young officer for being a hero... and then shot him dead for being a careless idiot.<br/><br/>The Brits, bless their stodgy old hearts, got it right. They are a nation of problem avoiders. In America we give out the medals... and then send the careless idiots back to their duty stations.
Americans as a people are problem solvers, not problem preventers. We happily rebuild homes destroyed by hurricanes, brushfires, and mudslides. In their original locations! And our banks, insurance companies, police, firefighters, and relief agencies happily subsidize this insanity! (One definition of "insanity" is: "The belief that if you keep doing what you've always done you will somehow stop getting what you've always gotten.")<br/><br/>This archetype has been around for so long that it's got a name: The Loose Cannon Syndrome. Named after an infamous event on a British warship of the ropes-and-canvas era. As a storm approached, a young officer failed to double-check the knots he had tied when lashing a cannon to the deck. The pounding of the storm quickly undid the knots and the cannon began bouncing around the deck, smashing up the superstructure and creating the terrifying possibility of bashing through some really important timbers that would allow the ship to take on water.<br/><br/>The officer gallantly dashed out onto the heaving deck and, risking his life at every turn, managed to single-handedly immobilize the cannon and secure it properly.<br/><br/>When the storm had passed the captain assembled the entire crew on the deck for a ceremony to give this performance its due. He pinned a medal on the chest of the young officer for being a hero... and then shot him dead for being a careless idiot.<br/><br/>The Brits, bless their stodgy old hearts, got it right. They are a nation of problem avoiders. In America we give out the medals... and then send the careless idiots back to their duty stations.
Americans as a people are problem solvers, not problem preventers. We happily rebuild homes destroyed by hurricanes, brushfires, and mudslides. In their original locations! And our banks, insurance companies, police, firefighters, and relief agencies happily subsidize this insanity! (One definition of "insanity" is: "The belief that if you keep doing what you've always done you will somehow stop getting what you've always gotten.")<br/><br/>This archetype has been around for so long that it's got a name: The Loose Cannon Syndrome. Named after an infamous event on a British warship of the ropes-and-canvas era. As a storm approached, a young officer failed to double-check the knots he had tied when lashing a cannon to the deck. The pounding of the storm quickly undid the knots and the cannon began bouncing around the deck, smashing up the superstructure and creating the terrifying possibility of bashing through some really important timbers that would allow the ship to take on water.<br/><br/>The officer gallantly dashed out onto the heaving deck and, risking his life at every turn, managed to single-handedly immobilize the cannon and secure it properly.<br/><br/>When the storm had passed the captain assembled the entire crew on the deck for a ceremony to give this performance its due. He pinned a medal on the chest of the young officer for being a hero... and then shot him dead for being a careless idiot.<br/><br/>The Brits, bless their stodgy old hearts, got it right. They are a nation of problem avoiders. In America we give out the medals... and then send the careless idiots back to their duty stations.
Americans as a people are problem solvers, not problem preventers. We happily rebuild homes destroyed by hurricanes, brushfires, and mudslides. In their original locations! And our banks, insurance companies, police, firefighters, and relief agencies happily subsidize this insanity! (One definition of "insanity" is: "The belief that if you keep doing what you've always done you will somehow stop getting what you've always gotten.")<br/><br/>This archetype has been around for so long that it's got a name: The Loose Cannon Syndrome. Named after an infamous event on a British warship of the ropes-and-canvas era. As a storm approached, a young officer failed to double-check the knots he had tied when lashing a cannon to the deck. The pounding of the storm quickly undid the knots and the cannon began bouncing around the deck, smashing up the superstructure and creating the terrifying possibility of bashing through some really important timbers that would allow the ship to take on water.<br/><br/>The officer gallantly dashed out onto the heaving deck and, risking his life at every turn, managed to single-handedly immobilize the cannon and secure it properly.<br/><br/>When the storm had passed the captain assembled the entire crew on the deck for a ceremony to give this performance its due. He pinned a medal on the chest of the young officer for being a hero... and then shot him dead for being a careless idiot.<br/><br/>The Brits, bless their stodgy old hearts, got it right. They are a nation of problem avoiders. In America we give out the medals... and then send the careless idiots back to their duty stations.
This reminds me of the story of a chinese family of healers. One of them becomes famous for his skills and is called infront of the great emporer. The emporer asks him who is the greatest healer in his family. He says that he heals the sick, sometimes with dramatic results and is hailed a hero throughout the country. His elder brother heals those whose illness has just started to take root, and he is known only in the local village. His eldest brother prevents sickness before it starts, and he is unknown outside of his own home.
This reminds me of the story of a chinese family of healers. One of them becomes famous for his skills and is called infront of the great emporer. The emporer asks him who is the greatest healer in his family. He says that he heals the sick, sometimes with dramatic results and is hailed a hero throughout the country. His elder brother heals those whose illness has just started to take root, and he is known only in the local village. His eldest brother prevents sickness before it starts, and he is unknown outside of his own home.
This reminds me of the story of a chinese family of healers. One of them becomes famous for his skills and is called infront of the great emporer. The emporer asks him who is the greatest healer in his family. He says that he heals the sick, sometimes with dramatic results and is hailed a hero throughout the country. His elder brother heals those whose illness has just started to take root, and he is known only in the local village. His eldest brother prevents sickness before it starts, and he is unknown outside of his own home.
This reminds me of the story of a chinese family of healers. One of them becomes famous for his skills and is called infront of the great emporer. The emporer asks him who is the greatest healer in his family. He says that he heals the sick, sometimes with dramatic results and is hailed a hero throughout the country. His elder brother heals those whose illness has just started to take root, and he is known only in the local village. His eldest brother prevents sickness before it starts, and he is unknown outside of his own home.
Very funny. (I mean it.) But goats don't prevent bridges from collapsing or airplanes falling from the skies. Goats' big problem is that they require the entire ecology to be in place before they can get to work. Developers find that it takes more effort to create "testable" units for testers early in the process than to simply check their work themselves. Inexperienced and lazy developers don't even do it themselves (beyond the minimally useful "unit tests.") "Early detection" tools identify the kinds of simplistic errors that only beginners create. Software will remain only as good as its developers until the field matures and becomes as repeatable as bridge-building. That didn't happen by hiring goats - no matter how cheap.
Very funny. (I mean it.) But goats don't prevent bridges from collapsing or airplanes falling from the skies. Goats' big problem is that they require the entire ecology to be in place before they can get to work. Developers find that it takes more effort to create "testable" units for testers early in the process than to simply check their work themselves. Inexperienced and lazy developers don't even do it themselves (beyond the minimally useful "unit tests.") "Early detection" tools identify the kinds of simplistic errors that only beginners create. Software will remain only as good as its developers until the field matures and becomes as repeatable as bridge-building. That didn't happen by hiring goats - no matter how cheap.
Pages