Start Trusting Your Test Automation Again

Development teams these days often rely on feedback from automated tests to determine the quality of the application they are working on. Running automated tests every night or after every commit, depending on the type of test and the adopted development process, gives teams information about the impact that changes made to the code have had on its overall workings.

However, with great power comes great responsibility. The more you rely on the feedback from your automated tests to decide whether to let a build pass to the next step in your build pipeline (and this next step might just be a deploy into production!), the more you need to be able to rely on the quality and defect-detection power of these very automated tests.

In other words, trust in the quality of an application is required, so if the application is tested in an automated fashion at any point, you must also trust in the quality of these automated tests.

Unfortunately, this is where test automation all too often fails to live up to its promise. Instead of being the stable and reliable guardians of application quality they should be, automated tests regularly are a source of deceit, frustration, and confusion. They end up undermining the very trust they are meant to provide in the first place.

How can we start trusting our automated tests again? Let's take a look at two ways automated tests can harm trust instead of creating it, then explore what you can do to repair the situation—or, even better, prevent it from happening in the first place.

False Positives

Tests that fail for reasons other than a defect in your application under test—or at least a mismatch between an expected and an actual test result—are known as false positives.

This type of harmful automated test occurs most often with user interface-driven tests, simply because these tests provide the largest amount of potential ways things can go wrong (synchronization and timeout issues, insufficient exception handling in the automated test solution, etc.). If your timeout and exception handling aren't implemented properly, false positives can be very time-consuming in terms of root cause analysis, as well as preventing them from happening in the future. In cases of intermittent occurrence of these false positives (also known as “flaky tests”), it can be even harder to find out what is causing the trouble.

False positives are especially frustrating when your team or organization is adopting a continuous integration or continuous delivery approach to software development. When your build pipeline contains tests that intermittently cause false positives, your build might break every now and then simply because your tests aren't robust enough to handle exceptions and timeouts in a proper manner.

This might eventually cause you to just remove the tests from your pipeline altogether. Although this could solve your broken builds problem at that moment, it is not a strategy that is sustainable in the long term. There is a reason you're putting effort into creating these tests (right?), so they should be part of the automated testing and delivery process. Therefore, you should put effort into investigating the root cause of these false positives as soon as they occur and repair them right away.

Even better, you should put effort into creating robust and stable tests, including proper exception handling and synchronization strategies, to prevent these false positives from occurring in the first place. This will undoubtedly take time, effort, and craftsmanship up front, but a solid foundation will pay off in the long term by resulting in the eventual absence of these pesky false positives.

False Negatives

While false positives can be very frustrating, at least they make their presence known by means of error messages or broken CI builds. The real risk of decaying trust in test automation lies in false negatives: tests that pass when they should not.

The more you're relying on the result of your automated test runs when making procedural decisions, such as a deployment into production, the more important it is that you can rely on your test results being a good representation of the actual quality of your application under test, rather than an empty shell of tests that pass yet do not perform the checks they are supposed to.

False negatives are especially tricky to detect and deal with for two distinct reasons:

• As said above, they do not actively show their presence, such as by throwing an error message, like true positives and false positives do
• While some tests might be false negatives right from the start, a significant part of false negatives are introduced over time and after several changes in the application under test

A vital part of your test automation strategy, next to creating new tests and updating outdated ones, should therefore be to regularly check up on the defect detection power of your existing automated tests. This applies especially to tests that have been running smoothly and passing since their creation. Are you sure they really still execute the right assertions? Or would they also pass when the application under test behaved erroneously?

I like to call this regular checkup on your existing test suite “keeping your tests fresh.” Here’s a solid strategy for periodically making sure your tests are still worthy of the trust you put in them:

1. Test your tests when you create them. Depending on the type of test and assertion, this can be as simple as negating the assertion made at the end of the test and seeing whether the test fails. When you're adopting test-driven development, you're automatically doing something similar, because your tests do not pass until you implement working production code.
2. Periodically, go through your tests and make sure they a) are not made redundant by the changes that have been made to your application since the test was created (a passing test that's irrelevant might be nice for the statistics, but imposes a maintenance burden you likely can do without), and b) still possess their original defect detection power.

For unit tests, I recommend at least looking into mutation testing as a tool for creating and maintaining a high-quality, powerful test suite. Mutation testing creates mutant versions of your code (for example, by flipping a relational operator such as a “greater than” symbol) and subsequently seeing if the test fails. If the test still passes despite the change made by the mutation testing algorithm, it is incapable of detecting these changes and should be dealt with accordingly.

Even though doing a mutation testing run can be a time-consuming process, especially for large applications, it might give you some useful insight into the quality and defect-detection power of your unit test suite.

Unfortunately, I am not aware of any similar tools for other types of tests, such as integration and end-to-end tests. For these tests, I recommend regularly going through your suite manually to keep stock of the quality of your tests and keep your tests fresh.

Create the Right Tests in the First Place

Building trustworthy automated tests requires development skills, but it may be even more important to have the skill required to decide whether to automate a test at all in the first place.

It also takes skill to determine the most efficient way to automate a given test. I like to do as my namesake Edsger Dijkstra recommended and look for the most elegant solution to a test automation problem—the simple, clear strategy. If the path I'm taking isn't leading to an elegant solution, then this solution probably is not the most efficient one, either.

This translates to trustworthiness, too: If your solution isn't elegant, there are likely too many ways in which the test results can deceive you. Simplicity and clarity will help make you secure in trusting your test automation.