This is part 2 of a three-part list of nine "commandments" that can be used as a guide to improving your life as a test engineer. As a response to numerous inquiries about these tenets listed on James' Web site, this column provides his explanations behind the "Thou Shalts." These are intended to be useful suggestions as well as for fun. Read on and see what you think.
Here is the full list of commandments so you can see them in one list as they originally appeared on my Web site:
- Thou shalt pummel thy app with multitudes of input
- Thou shalt covet thy neighbor's apps
- Thou shalt seek thee out the wise oracle
- Thou shalt not worship nonreproducible failures
- Thou shalt honor thy model and automation
- Thou shalt hold thy developers sins against them
- Thou shalt revel in app murder (celebrate the BSOD!)
- Thou shalt keep holy the Sabbath (release)
- Thou shalt covet thy developer's source code
You can click at the end of this column to read part 1, commandments 1 and 2, from last month. And now, here are my interpretations of numbers 3-6.
3. Thou Shalt Seek Thee Out the Wise Oracle
We all know that there are at least two parts to testing. First we apply, then we check. When we apply inputs we are testing whether the software did what it was supposed to do with those inputs. Without the ability to verify that this is indeed fact, testing is much less effective.
Testers call this the "oracle problem" in reference to the wise oracle that knows all the answers. Of course, the answer we are interested in is "did the app do what it was supposed to do when I applied some test?" This requires our oracle to intimately understand what the application is supposed to do given any specific combination of inputs and environmental conditions. Automating the oracle is very hard, but a worthwhile pursuit, not only as a valuable testing tool but also as an intellectual pursuit. Forcing yourself to think like such an oracle can often be more productive than anything else that you might choose to do, whether or not you ultimately succeed in automating it.
4. Thou Shalt Not Worship Irreproducible Failures
We've all been here, haven't we? You see a bug, usually a good bug, then it won't reproduce. The better the bug, the worse you feel about it. I have seen many a good tester waste hours and even days trying to reproduce a bug that they saw only once.
The effort to reproduce such a bug is often valiant but without the proper tools, the effort can be a waste of time. But the problem I see is that the time is wasted anyhow, without the tester even realizing it. I had a tester spend an entire day trying to remember the reproduction steps of a crashing bug, with no success. I would have preferred that the particular tester spend his time in better ways than that. I understand the frustration as well as any tester but the pursuit of such a bug is often time not well spent.
The moral of this commandment is twofold. First, try your best to be ever alert and remember (or record) the sequences of actions you are taking against the software. Remember also the application's response. Second, consider using debugger-class tools that can track your actions and the state of the software. This takes much guesswork out of reproducing bugs and prevents otherwise good testers form breaking this commandment.
5. Thou Shalt Honor Thy Model and Automation
Commandment one was about the importance of random testing-emphasis on random. This commandment is about intelligent random testing-emphasis on intelligent. When intelligence meets automation, the result is called model-based testing. Get used to the term because it is the automation technology of the future.
Software models such as objects, black boxes, or structure diagrams help us to understand software. Testing models help us understand testing. A testing model is a blend of intelligence about what an application does (the model) and how it does it (the automation). Good models can make your automation smart enough to respond to errors and cover code that is out of reach of dumb automation. Modeling is an exercise that at the very least will make you more prepared to test, even if you don't bother to automate it.
6. Thou Shalt Hold Thy Developers Sins Against Them
Development work is hard, very hard. Developers over the past few decades have had to solve the same problems over and over again and in doing so, often make the same mistakes over and over again. We testers must remember those mistakes and design tests that ensure that lessons are being learned.
If one developer makes a mistake coding some module, then we should assume that other developers might make the same mistake on similar modules. If a particular developer is prone to coding infinite loops, then we need to make sure we test for such errors in every module that the developer writes. This is "learning from experience" and we are here to make sure that is what our developers do: understand their patterns of mistakes so those mistakes can be eradicated.
Let me know what you think of these commandments. Next month part 3 will explain numbers 7-9. I hope you enjoy reading and thinking about them as I have over the years.
References
- Random Testing and 'App Compat': The Ten Commandments of Software Testing, Part 1
- Test-Physicians, Source Code, and BSODs: The Ten Commandments of Software Testing, Part 3
- Massive scale random testing is a sub discipline of model-based testing. Learn more about both at www.model-based-testing.org.
- Test Oracles are elusive, but there is help out there. My very favorite article on the subject is from long time testing guru Elaine Weyuker. The paper is called "On Testing Non-testable Programs," The Computer Journal, Vol. 25, No. 4 (1982), pp. 465- 470. Sure it's an old paper, but I consider it a classic.
