A Primer on Continuous Testing

About nine years back, I was involved in doing QA for a Fortune 500 retailer, leading a small team. The application we were working on was available on web and mobile, with new features getting added constantly. Despite good coverage from QA, bugs were found in production, and customer support teams were flooded with complaints citing wrong products delivered, shipping issues, and product availability challenges, which were passed on to engineering, particularly QA.

Continuous integration was becoming popular around that time, and our engineering team established a process for developers to integrate their work frequently. We also implemented processes for working closely with developers and business representatives, writing behavior-driven tests to remove ambiguity in user stories, and considering nonfunctional testing aspects of the application, such as performance and security, early in the project.

A deployment pipeline was established where a set of initial tests would run to validate the build, and if those build verification tests passed, further automated acceptance tests would run to test key business functionality on the release candidate. These automated acceptance tests were behavior-driven, designed to work closely with business requirements, and based on the user journey of visitors who use the application.

The automated acceptance tests started as a small suite, with about 10 tests end-to-end cutting across modules such as merchandising, inventory, and shipping. This was the "holy grail" for engineering since it was executed on each release candidate and provided quick feedback on how key business functionalities were behaving. As new features kept evolving, the test suite size grew, as did the time it took to run the automated acceptance test suite. We decided to split the suite into two parts.

We discussed the user stories for new functionality with business representatives, and the tests for new functionality were automated in parallel with development. These stories were added to the automated acceptance test suite, and earlier tests that were in that suite were moved to a smoke regression test suite, which was run as often as the application got updated.

The term “continuous testing” was not common during those days, but teams insisted on automating as much as possible, and this tactic went on to become one of the recommended practices of continuous testing. I continued to work on many projects after the retail assignment that followed the continuous delivery discipline, and I saw teams taking up continuous testing seriously. I was able to connect the dots back to the retail application that introduced me to this process. Though I cannot claim that it subscribed to all aspects of continuous testing fully, it focused on all the key principles and practices of continuous testing that got established over the years. 

Based on my experience working on projects of all sizes since then, I have some proven practices from organizations that have used continuous testing effectively to realize tangible benefits.

Create a Continuous Testing Strategy

It is important to have goals that you want to achieve from your continuous testing approach, and that demands a testing strategy.

Chad Wathington, the chief capability officer of ThoughtWorks, listed four essential questions he believes are central to creating a testing strategy, and I have modified these questions in the context of continuous testing:

• Can we create tests that can be executed early and often enough to mitigate business risks?
• Can the tests be designed to exhibit actual user behavior?
• Can testers collaborate well with all stakeholders in the software development process to ensure requirements are fully understood?
• Are other components in place to ensure effective continuous testing?

Getting answers to these questions can be a good starting point for setting up continuous testing.

Use Automated Acceptance Tests

Automated acceptance tests are an essential component of continuous testing. These are the tests that help engineering teams understand the system’s business behavior. Running these tests early and often to identify business risks is at the core of continuous testing. These tests are not designed to check every line of code, but rather focus on the behavior of the application under test.

In general, sources on continuous testing recommend that developers practice test-driven development (TDD) when writing unit tests and complement them with automated acceptance tests. Please note that they don’t recommend using a dedicated QA team to write and manage automated acceptance tests.

However, most organizations cannot follow this approach in reality. For example, the retail assignment I worked on nearly a decade ago had both dev and QA, but we never treated them differently. We were all “engineering.” The expectation was that every member of the engineering team should broadly understand what every other person is doing, and if required, should be able to get into details. Developers, testers, and business analysts worked together to discuss stories and then build and test them.

On the other hand, I have also been part of projects where test automation is in place but not continuous testing. Automation engineers build UI automation tests using tools such as Selenium, but they are not behavior-driven. Bugs found by users in production can never be traced back to these automated UI tests. The automated UI tests provide their share of benefits, but they are not the acceptance tests that can be used to get quick feedback on business risks.

Realize Continuous Testing Doesn’t Work in Isolation

The deployment pipeline is a key element of continuous delivery. Similarly, continuous integration, configuration management, and automated deployment are all also key elements. Continuous testing doesn’t exist without them.

I have seen organizations with no continuous integration setup claiming that they do effective continuous testing. That’s not continuous testing.

Prioritize Collaboration

Finally, a word on collaboration in general on continuous delivery projects. More than writing code, keeping infrastructure up and running, or good QA, continuous delivery relies on collaboration. Continuous integration is all about encouraging collaboration between developers, and DevOps is all about collaboration between development and operations.

Continuous delivery can best be realized when all teams operate in a social, collaborative environment, with the necessary information from downstream and upstream teams and by working with relevant experts when clarifications are needed or issues happen.