A tester's job is to provide information about elements of the system that might make a user unhappy. But Jon Hagar finds that many testers implement limited tours, even when they have robust programs. He writes that when looking for bugs, testers need to look beyond the software to the system and the user scenarios, too.
Jan Lundeen, a tester friend of mine, reported a situation to me:
“I got my Subaru Forester. It's a great car, but with keyless entry, I'm getting error messages that the ‘key is not found’ when I have the key in the car. Not sure what is happening yet, but it's a good example of how software and hardware interact. I'm just hoping that I don't get stranded.”
As a new user of hardware and software, Jan sounded a little worried. Later, she wrote:
“As for the ‘key not found’ error, the people at the Subaru dealership recommended that I keep the fob (used for keyless entry) in the driver's seat. The fob for keyless entry can be really sensitive. I had it in the passenger seat (of my car) in my fanny pack, so that might have been too far away. Also, I had my cell phone in my fanny pack, so that may have interfered with the signal. I read in the manual that I shouldn't keep it (my fob) by a computer, but nothing was mentioned about the cell phone.
“Since most women keep their keys in their purse and don't have good pockets (one thing I don't like about women's clothes), I’m surprised that this scenario was not considered (in testing). It sounds like the QA and engineers didn’t include real life scenarios (like having it in your purse on the passenger seat). And most women I know keep their cell phones in their purse with their keys. This illustrates why it's important to consider all potential users when designing a product. However, they could have been limited by the technology as well (since it's fairly new).”
Jan is thinking the way testers need to think. As testers, our job is to provide information about elements of the system that might make a user unhappy. As well as different scenarios, we also must consider the many different users. I find that many testers implement limited scenarios or tours, even when they have robust programs.
Consider how mobile cars are tested in the field.
In the photo above, some unknown car company is testing a new and unannounced vehicle (hence the drapes) in the real world. This is pretty robust testing. The testing field they are using is pretty extreme, considering this was taken in Colorado’s high country and on back roads. The test conditions are rough—steep inclines, snow, dirt, a high altitude, etc.
However, on both occasions I saw their testers, they were men, and they ran the same route. This made me wonder what scenarios and users are not being considered with their testing. For example:
- A female driver (Men and women have been found to drive differently)
- Other roads and conditions
- An older driver
- Handbags containing key fobs being stored in the back seat
- Hackers trying to attack the security of the car
- A small child (Will or should the car start for them?)
- A teenaged driver
- A dog that gets the fob and is jumping around in and out of the car, like my dogs do
These testing conditions might have exposed the error message Jan was getting, might have found safety or security bugs, and perhaps have made the car “smarter.”
Further, can we as testers build on the issues Jan noticed? For example, consider this scenario sequence:
- Start car with radio-frequency identification key fob in drivers seat
- Move the fob to a side seat
- Move the fob to a back seat
- Put the fob in a bag with a smartphone
- Success is the car keeps running
- Then test more actions, such as pushing the start button, turning the car off and then on, etc.
These are different uses and scenarios that testers may wish to accommodate in their testing—hopefully prior to any user experiencing a grave security error.
As testers, one of our jobs is to observe. Another is to think. My friend Jan was observing and following up on what she noticed. She and I then did some thinking exercises. “Is this a feature or a bug?” “Is there an improvement bug here?” Finally, I extended the testing to “How can I make the scenarios better to perhaps expose more useful information?” In your day-to-day testing, you might want to have similar patterns of testing: Observe, think critically, do some extended thinking, and take good notes.
Mobile, embedded software systems are getting smarter, but is our software and system testing getting better too? General Motors has had to recall more than twenty million cars and trucks this year due to various issues affecting customers’ vehicles. During development, how much testing was done? Were test resources limited? Was management informed and just chose to ignore the information provided by the testers? I don’t know what GM’s procedure was or is, but this is an example of how much can go wrong, even with extensive testing.
There are no perfect answers when dealing with limited time and resources, but here are some things to consider.
Kill as many birds with one stone as you can. Design test scenarios that can each expose as much information as possible. This usually means going beyond just checking requirements.
Provide information early, often, and as completely as possible. In reporting potential errors, go beyond reporting “The key fob is broken” and give a more complete examination of the issue, as Jan did.
Consider the weakness of current testing. Look at your tests and ask yourself if they are suffering the same problems as the car under test above. Running the same test over and over may be a waste of time. Change your tests and look for new users and scenarios to improve tests within existing budgets and schedules.
Run scared. There are bugs out there, so test, observe, think, and then extend within your given test resources. A tester is not paranoid if the bugs are out there. Keep in mind that the industry produces very little bug-free software.
Find a new job. There are some places and companies that will not listen to test information until something bad has already happened. In some cases, it is best just to move on.
I would like to thank Jan Lundeen for sharing her story, and I want to ask you to share your bugs and test scenarios too. I learn from my bugs, and I am still learning to be a tester. What about you?
User Comments
Excellent example to make people understand about the most common problem by testers " key not found " Just liked it Jon ! Really fascinated with the co-relation you expressed in the article
Thanks John, I try in my testing to see the "big picture" and "co-relationships". It make testing harder, but for me more fun. We need to think hardware, software, users, and risks of what can go wrong.