If you think that test automation is mostly about executing tests, then you're missing out on a big opportunity. Or rather, you're missing a lot of small opportunities adding up to a big one. Consider this: stop thinking about test automation as merely executing automated tests, stop thinking about test automation as something you need expensive tools for, and start discovering automation you can implement in a couple of days and usually with extremely inexpensive tools or tools you already have available. In this week's column, Danny Faught and James Bach suggest taking a more Agile approach to test automation.
You don't need anything special to make test automation more Agile. Just adopt a very broad view of test automation and start exploring the Internet for tools that can help you. The rest takes care of itself. But in most cases it will work better if you establish a toolsmith role in your team. A good toolsmith should know how to program in a high level language, like Java, Perl, or Python, to whip up solutions quickly. A good toolsmith loves tools and learns about the free or inexpensive tools that are helpful. And a good testing toolsmith will know something about testing.
The method we use for Agile automation is straightforward: The toolsmith watches a tester work and determines how tools can help with what the tester is doing. The toolsmith's attitude is to help the tester on the tester's terms, rather than to push some private agenda.
If you have no toolsmith, then each individual tester should be on the lookout for automation opportunities. Keep in mind, the more the tester knows about tools and programming, the more effective he will be at finding useful tools.
Examples of Our Agile Approach to Test Automation
A tester had previously done a manual spot check on two comma-separated value (CSV) files and found no differences. With assistance from Danny, the tester used a tool to compare the two CSV files. The tool found an entire column of data that was wrong. After about an hour of further research, they found another free tool that did a better job of finding non-matching data.
James helped a test team query an auction system for a full report of auction status at any moment. The tool enabled them to automatically confirm that they had tested the scenarios and conditions they wanted to test, instead of simply hoping they had made no mistakes in executing their test cases. This team had tested for nearly two years without such a report, yet writing the tool took only three hours from start to delivery.
Danny has used the Perl WWW::Mechanize module to write a quick hack that loaded tens of thousands of records into a Web-based database front-end overnight. The next day, he quickly identified a performance problem with the application that was related to running with a large database.
We have both found situations where installation test tools could help the tester see exactly what an installation procedure did to the filesystem and registry.
In all of these examples, we produced real value with only a few hours of effort. We helped testers improve their testing with the help of tools. To us, that is the very definition of test automation: tool supported testing. This extends the scope of test automation far beyond the idea of happy robots that test while you sleep. Why not do automated test design? We've done that. Why not use automated test probes that alert the tester to certain kinds of problems? We've done that. It's all test automation, but it's not your father's test automation.
Deliver Something Useful in Less than Forty Hours, or Else...
Or else you picked the wrong task.
Incremental delivery is a key part of Agile automation. We find that there is a quantum difference between solutions delivered in less than a work week and those that take longer. Longer projects are riskier and absorb resources that might be used for other quicker tasks. Long tasks can be broken up into discrete deliverables of just a few days each-each one delivered and working for a tester before embarking on the next.
If your automation project wasn't going well, wouldn't you rather know about it a week into the project rather than months later? What if you're over-engineering your solution? With an Agile approach, after a few incremental deliveries, the tester might decide that the automation he has is good enough, and the toolsmith will move on to other automation priorities.
With short tasks, it is more feasible for the toolsmith and tester (especially if they are the same person) to self-approve their projects, eliminating a lot of management oversight. Automation tasks that exceed a week in duration are likely to require approval from higher levels, which further complicates and slows the automation process.
If you predict that some task will take more than forty hours, or if you approach forty hours of effort on something you thought wouldn't take so long, it's time to step back. Have you really broken it down into small enough chunks? Is your approach really viable?
No Automation Cathedral
So many "classic" automation efforts we've seen seem to withdraw into the proverbial cathedral or ivory tower. In one case, James witnessed a grand automation strategy that was so remote from its clients that the testers he interviewed didn't even know it existed--despite nine solid months of development work. Danny has seen a test team rebel against a centralized automation group that didn't take the team's needs into account.
What is the opposite of the cathedral? Automation driven by the day-to-day test process, with as many different tools as there are booths in a bazaar; and test automation serving specific immediate needs, rather than future needs. Certainly there is a place for long-range automation projects that establish an infrastructure for automating difficult tests. However, our experience is that the big strategy can blind people to the many small solutions that can have a big impact immediately.
The essential ingredients for an Agile test automation are this: Think broadly about where tools can assist the full range of testing tasks that you perform. Download, create, or buy tool elements to automate a task for the first time or to improve existing automation, and complete each piece in less than a week. Try to establish a toolsmith role in your organization, and don't allow an automation cathedral to slow your progress. There is so much potential for tools to help you. Now get to it!
Authors' Note: Thanks to Neill McCarthy for his early feedback on this article.
Further Reading
- Agile Test Automation whitepaper by James Bach
- "The Cathedral and the Bazaar" by Eric Raymond
- "Manifesto for Agile Software Development"
