A Word with the Wise: Automation Analyzed with Linda Hayes

Over the past two decades, testing automation has moved through four distinct phases. The first three—capture/replay, capture/script/replay, and keyword or action word frameworks—eventually failed, due in part to the difficulty of maintenance.

"I believe the next wave is a buy-versus-build model," Linda Hayes says. "Few companies develop their own accounting systems anymore, for example, because it is cheaper to buy one. I think companies will start trying to reduce or eliminate the amount of custom code they have to write for test automation for the same reason."

Automation testing products are often considered tools, but Hayes believes they will take on more elements of applications in the future. She says test cases will be "stored and managed as data instead of as code. This empowers domain experts to directly participate in automation and allows automation engineers to support ten or twenty applications each, instead of only one or two."

During her career, Hayes has witnessed two distinct types of automation abusers. Some companies dive in and try to automate everything, without accounting for how the automation fits in with their test objectives. Others take the opposite but equally disastrous approach, "building a massive automation infrastructure that diverts resources from actual testing and expends it instead on yet another development project." Either way, Hayes says, the companies are unable to measure the effort or results of their automation implementation.

"To me it's pretty straightforward: The time you spend automating should be exceeded by the amount of test execution time and coverage you get back," she says. "That means you have to take into account not only the initial implementation and development time but also the maintenance to keep the tests current."

Companies with "clear, realistic goals and a logical project plan that is properly staffed and scheduled" are the most effective users of automation. Rather than over-automating, they only automate areas that are highly repeatable or time-consuming. Keeping automation plans orderly reduces confusion and inefficiency and, in the end, can save a company the trouble of having to find or decipher script code.

"Without adequate planning, design, training, and time for implementation, introducing an automation tool can cost more than it saves," Hayes says. "But tools can also be used to enforce process, because you can't automate something that is undefined."

As businesses introduce more customer-facing systems, software quality becomes more crucial to global business practices. Or, as Hayes puts it, "No one wants to be on CNN with a public software failure." Once the focus of testers alone, software testing has now entered other areas of the business world.

"Business analysts must write testable requirements and provide comprehensive use cases and business process flows, as well as provide test data values for acceptance testing. Developers must perform unit, integration, and build testing to assure a working product.

Independent QA and testing must perform black box tests to verify functionality. Production support must assure a repeatable and stable test environment for both software and data refreshes or updates, and contribute to load-testing scenarios to assure performance," Hayes says. "Each of these roles and types of testing are essential. Any organization that believes only testers test is doomed to fail."