This is part 3 of a three-part list of nine "commandments" that can be used as a guide to improving your life as a test engineer. These are intended to be useful suggestions as well as for fun. Read on and see what you think.
As a response to numerous inquiries about the commandments listed on my Web site . This column is part 3 of the explanations. I mean these to be both fun and useful. You can also click below to read parts 1 and 2. Here is the full list of commandments so you can see them in one list.
- Thou shalt pummel thy app with multitudes of input
- Thou shalt covet thy neighbor's apps
- Thou shalt seek thee out the wise oracle
- Thou shalt not worship nonreproducible failures
- Thou shalt honor thy model and automation
- Thou shalt hold thy developers sins against them
- Thou shalt revel in app murder (celebrate the BSOD!)
- Thou shalt keep holy the Sabbath (release)
- Thou shalt covet thy developer's source code
And now, here are my interpretations of numbers 7-9 (yes, there's really only nine).
7. Thou shalt revel in app murder (celebrate the BSOD)
I often make an analogy between testers and physicians. Physicians, I say in the story, treat their patients gingerly. They say "Does it hurt when I touch you here?" And then they promptly stop touching that spot when you say "Yes!" If testers were physicians, the story would be somewhat different.
Test-Physicians would also inquire "Does it hurt when I touch it here?" But when the pain is confirmed, a test-physician would then poke, prod, and probe, until the pain became unbearable. Why? Well, it isn't sadism, it's our job. No bad deed should go unpunished.
You see, every bug is a proud moment for a tester, but no bug should go without further investigation. So you found a bug that causes ill-formatted data to be displayed on the screen. Great, but can you go further and make that same bug corrupt data that the application is storing internally? If so, you have a better bug. And, can you then make that corrupt data be used by the application in some internal computation? If so, you have now turned a simple little formatting bug into a severity-one bug that causes data to be corrupted and the application to crash.
And, of course, the Holy Grail would be to crash not only your application, but to cause the entire operating system to hang. Ahh, the Blue Screen Of Death. I remember my first like it was yesterday, I anticipate my next every time I apply a test.
The moral of this commandment is that behind every good bug, there may be a better bug. Never stop exploring until you've discovered just how deep the bug goes and just how damaging it can be.
8. Thou shalt keep holy the Sabbath (Release)
Oh so many times I hear testers whine about release dates. Testers most often want to extend release dates and, more often than not, their reasoning for wanting to do so is right on the mark. But their reasoning sometimes doesn't matter.
The fact is that there are more factors than just quality that go into determining when to release an application to users. Quality is important but market pressures, competition, strength of user demand, staffing and personnel issues and many more nontesting issues must determine a suitable release date. As testers, we must simply get the most work done that we can in the amount of time allotted to us.
We should not complain about release dates. We should, instead, warn about consequences. That is the extent of our responsibility and it should be the extent of our concern.
9. Thou shalt covet thy developer's source code
I am not much of a believer in white box testing. I think that it is something developers should learn to do well so we testers can concentrate on more important and complex behavioral tests. That said, however, don't look a gift horse in the mouth. If you have access to the source code, use it.
But, use it as a tester should, not as a developer would. My interest in source code is many fold and too involved to discuss all the issues here. But I think there is much to be learned from reading the source. Top of my list is looking for error-handling code and the dialog boxes that will indicate to us that the error code is executing. Error handlers are the hardest code to see or get to from the user interface. Understanding what error handlers have been written and what inputs it takes to trigger them is time well spent.
Indeed, there are many such clues we can glean from the source that give us insight into tests that need to be performed. We should not be shy about asking for and using the source code.
So that's the commandments. By the way, there is a reason there are nine instead of ten. We might assume that just because they are "commandments," there have to be ten of them. Since we know the assumption to be true (because that's the nature of an assumption) then we convince ourselves that there is no need to ever bother checking whether the assumption may become false.
Assumptions are a very bad thing for software testers. Assumptions can reduce productivity and undermine an otherwise good project. Assumptions can even undermine a career. Good testers can never assume anything. In fact, the reason we are called testers is that we test assumptions for a living. No assumption is true until we test and verify that it is true. No assumption is false until we test that it is false.
Any tester who assumes anything about anything should consider taking up development for a career. After all, what tester hasn't heard a developer say "Well, we assumed the user would never do that!" Assumptions must always be tested. I once heard a test consultant give the advice: "Expect the unexpected." With this I disagree; instead, expect nothing, only then will you find what you seek.
I hope you have enjoyed this series as much as I have. Bon Appetite!
References
- Massive scale random testing is a sub discipline of model-based testing. Learn more about both at www.model-based-testing.org.
- Test Oracles are elusive, but there is help out there. My very favorite article on the subject is from long time testing guru Elaine Weyuker. The paper is called "On Testing Non-testable Programs", The Computer Journal, Vol. 25, No. 4 (1982), pp. 465- 470. Sure it's an old paper, but I consider it a classic.
- Random Testing and 'App Compat': The Ten Commandments of Software Testing, Part 1
- Oracles, Failures, Models, and Sins: The Ten Commandments of Software Testing, Part 2
