Some of us in the software test industry write like test execution automation is a radical idea, but test automation has been around for many years, and for some projects, it is the best options. Test automation of each level of execution or type of test is not for every project, but there are software project environments where large-scale test automation is the reasonable choice, despite automation’s limitations such as cost, tool maturity, and skill of testers.
I am not talking about all categories of testing support tools, such as planning or reporting. I am specifically focused on the automated execution of tests at any level of the software, where the testing is accomplished by computer-readable scripts or keywords.
So, what kind of software execution can best—or maybe only—be tested using automation?
When the Only Option Is Test Automation
About thirty-five years ago, a space industry project had a real-time embedded software project that needed to be tested. The constraints on the testing included speed of computation input-output response cycle, which was ten milliseconds; a mission time use case that spanned more than ten hours; risks such as massive loss of money, life, and reputation; complicated embedded software; interfaces to control unique hardware, including rocket motors or sensors; and telemetry communications.
To test at any level, particularly the system level, required automation, because no human could respond with a new input for each ten-millisecond time frame. The team had to build a keyword data-driven system to input tests that covered a complete series of mission scenarios. The keywords drove test execution and controlled complex, real-time simulations, which allowed the software to experience many possible external inputs. Automation was the only option.
Other Ideal Cases for Automation
This project situation is not unique. Many in the software test industry will experience contexts that require test execution automation. Here is a sampling of software project where test execution automation may be the necessary option.
When the system computation cycle time (output to required input) is faster than a human can respond
An example is the space project story above, but I have seen similar simulation systems for airplanes, robotics, and aspects of self-driving automobiles.
When the system has complex user-machine activities where failures can have significant financial or resource impacts
An example is smart cities, which can impact many people as well as large amounts of money and time. Smart cities have vast numbers of devices and communications, and automation of testing at a city level may be the only viable option.
Just listing the kinds of users the “smart” internet of things (IoT) software systems have, you are like to see testing a smart city becomes a huge manual task. A possible list of IoT smart traffic light users includes citizens on the road, police, database systems, communication systems, city managers, operations staff, power systems, machine learning route systems, and smartphone apps. Automating the testing for the many varied devices in a smart city is the good choice.
When the system is high-risk and a manual human error during testing or ops could not be tolerated
An example is found in some medical systems where the loss of human life is a sufficient risk to require test automation. Consider a smart device that is to be implanted in a human to keep the person alive. Before implanting the device in a real field trial, of course there will be many tests.
I have seen automated “test dummies” that allow the device to be hooked into a complex test framework where the dummy simulates the conditions a real human might experience. The automation framework allows many scenarios to be tested that could be risky for a real person.
Recognize When Automation Is the Best Choice
Testers need to be thinking about situations where the manual approach is not a good option for project test planning. Too often, many testers default to manual testing because it’s what they are most familiar with, but given that there are some situations where automated execution must be included in planning, testers need to not be trapped in a pattern of thought.
However, even with automation, teams still need thinking and skilled testers. Recognizing the need for automation is a skill worth building along with the factors to make automation success.
In the space industry project, there were several items critical to success. First, the test team needed skilled programmers and testers who understood the keywords and system under test. The project had dedicated people on these areas. Next, in test project planning, management had to allocate a schedule to develop the test system and train people to use the system. This took resources that had to be included in the project-level planning. Projects that do not allocate a schedule and resources for these efforts are more likely to fail during test automation. To assure these allocations, all levels of project management need to support the test execution automation efforts. Finally, the hardware and software efforts needed to be stable and integrated with the test development. This took ongoing communication and iterative planning updates within all project teams.
As software systems grow in complexity and size, test execution automation will become expected and, sometimes, even the only viable option. This situation will be seen in IoT, embedded, and large-scale, complex systems of systems.
While some testers are unfamiliar with test execution automation, the growing trend into automation necessitates new skills for manual testers. Project test teams need to become aware of this trend, as automation represents not only business opportunities, but also increased quality and fewer risks in complex, safety-critical, and mission-dependent projects.
