Tales of Test Automation Failure (and How to Find Success)

Rakesh, the senior QA engineer of his testing team, had just returned from an enlightening conference on software testing. Now he had to present his key takeaways from what he’d learned at the conference to everyone else in his department.

Everyone gathered in the meeting room with their laptops and coffee mugs, expecting a boring presentation. Instead, Rakesh displayed a single slide, reading: “We need to automate, but we have failed in our previous attempts. Should we try again?”

You could have heard a pin drop. No one, least of all management, had dared to admit that they had failed to implement automation in their testing plan, yet they all knew the truth. Rakesh had just displayed what everyone was thinking.

Adopting test automation in your test plan is more challenging than it may initially appear. Understanding these challenges can help QA teams avoid some common mistakes and set a better, more realistic plan for their next attempt at automation.

Let’s look at three of Rakesh’s team’s previous failed attempts to automate.

Attempt 1: Automate because Everyone Else Is

The first time Rakesh’s team tried to automate their tests, the reason behind the attempt was simply so they would not be left behind. Their application was not stable enough, there was no proof of concept to determine feasibility, and the testers were confused about whether to test new code or automate what was already working. Should they focus on finding new bugs or meeting the weekly target of automating fifteen test cases?

This confusion continued until the release date was near and everyone abandoned automation to focus on the essential testing. No one talked about automation for a while after that.

Attempt 2: Automate 100 Percent

Hoping to be better prepared this time, the managers at Rakesh’s company attended a conference on test automation. They came back with anecdotes about how Company ABC pushes code to production in only x seconds due to automation. They don’t have a single tester; everything is automated. Let’s just do what they do.

Rakesh and the other testers had a tough time convincing management about the difference between testing and checking. Thoughts on how testing cannot be automated fell on deaf ears. The managers were convinced that 100 percent automation was the key to the other companies’ success.

The test team couldn’t change the managers’ minds, so they ended up working toward 100 percent automation. However, as they didn’t have separate test tools focused on all the different aspects of the system and application, there were some gaps. The tools they did have could only check variables, not figure out complex, end-to-end scenarios, so usability suffered. Some customers lost faith in the product’s quality and defected to competitors, and 100 percent automation was finally abandoned.

Attempt 3: Automate Only Key Scenarios

Learning from the last failure, managers agreed that the team should automate only certain types of scenarios. This attempt started with identifying key scenarios for automation. They had their test artifacts for manual testing, and they hired two engineers who would help them with automation.

The first complaint was that the automation engineers couldn’t easily understand the test artifacts. The team spent a considerable amount of time writing scenarios in a way the automation engineers would understand. Everything looked smooth until the final suite was automated.

Automation scripts were running successfully, but it took more than a day for the full suite to run. As the team was practicing agile, they were getting daily builds, but the automation results for a build wouldn’t come until the next day. By then, the team was testing the next build. The failures also had to be validated by the so-called manual testers, as there were many false positives.

No one was ready to accept this attempt as a failure because the team had spent a lot of time and effort in this exercise. It can be difficult to accept one’s mistakes, can’t it?

Success: Automate to Complement Manual Testing

Finally, some good sense prevailed, and the testers were courageous enough to voice their opinion on where automation could help them in their testing. They did not pick the tool first and then look for scenarios the tool could automate; they picked the scenarios to be automated and then looked for the tool that could help them.

The team understood the reason for automation: to complement the testing they were already doing. They learned to use tools to come up with test data quickly and to run smoke tests on every build, to use scripts to confirm that things are working fine after minor changes in the code, and to build suites to check across configurations quickly.

Test coverage increased, testers could spend more time performing exploratory and dynamic testing, and, most importantly, product quality improved. Rakesh and his team finally found test automation success.

Lessons Learned

There are many test teams like Rakesh’s that periodically try to jump on the automation bandwagon and try to see if they can succeed this time around. Some invest in their skills, some spend on costly tools, and some hire engineers who are good at automation. But without a sound test plan and a realistic goal, they’re setting themselves up for another failure.

Don’t build your career around a particular test automation tool. Tools should only be thought of as ways to complement your testing. And realize that you need both manual and automated testing. You can't simply add automated tests to an existing test plan. Instead, you need to rethink your entire approach and determine which tests would work better automated and which should remain manual.

Think deeply and critically before you embark on your journey of test automation.

Which attempt resembles your story? Share your experiences in the comments section below so we can learn from each other.