Whole-Team Test Automation: Making the Move

Like many test automators, I spent years developing my automation skills. I worked with teams of testers who did not have coding experience, so my skills were special. This was good for me, but I don’t think it is good for the profession of software testing. Now, I devote my time making these special skills ordinary.

Traditional testing did not really do much to compel testers to grow technically. Too much emphasis was placed on process and manual test execution. It was often hard to figure out how to use automators effectively. Fortunately, the demands of agile testing changed that. Agile requires groups of people to work together as peers and to trust each other’s skills. It requires technical testers.

What We Are Trying to Accomplish

A quick look at testing blogs and articles suggests that others share the idea that agile testers benefit from technical training. For example, in “Learning for Agile Testers” in the September/October 2011 issue of Better Software magazine, Lisa Crispin and Janet Gregory write about technical training for agile testers. If we testers do not look out for ourselves and develop a high standard of what a professional tester should be, we risk becoming ineffective and outdated very easily. The investment to grow a whole-team test automation program is significant. Reasons for doing this affect the tester, the project, and the test team.

For the Tester

At many companies, automated tests are developed, executed, and owned by a separate group within the test team. This creates a two-tiered test team. To make matters worse, the most technical members become more technical because they have the most practice, and the least technical members have no opportunities to grow their skills.

The more technical a tester is, the more tools he has in his toolkit and the better he will adapt to new situations, ranging from new technologies with services testing to fundamental changes in how software testing is done. Adapt or die.

Never forget, agile testers are developers. As members of a software development team, I think it is important that everyone should know how to code. It is good for your self-esteem, and it helps you gain and keep the respect of the other developers on your team.

For the Project

In agile development, rapid development of automated tests is critical. There are a lot of short-cycle sprints, and no one wants to retest the previous sprints’ work to make sure it doesn’t break. While we still struggle with keeping up with automation during sprint work, it is the job of the testers and the Scrum teams to do what is best for the project. This may mean aggressively developing regression tests or other efficient test tools throughout the project.

It isn’t acceptable for one project to have good coverage just because it happens to have the good automator. All projects deserve technically skilled testers.

For the Test Team

On many test teams, the full team is not exposed to existing automated tests. This lack of exposure leads both to misunderstandings about what the tests do and to poor test design. When the team members are not intimately familiar with their tests, they make assumptions. And, the more distant a team is from the tests, the more this becomes a problem.

It is easy to assume that when we have tests for an application, we don’t need to worry about the actual coverage. Some tests might have names that suggest greater test coverage than they actually provide, and if we don’t open the test and read the code, we will never know. Unsupported rumors of good test coverage spread easily. Generally, as a team, there is no universal knowledge of what existing tests do. We wanted to solve that.

The other problem that this lack of exposure causes is poor test design. When an automator who works by himself writes tests, he often writes them for himself. Tests and the process of creating them can end up being very specialized—often an extension of the automator’s brain. In this situation, the rest of the team can become detached from the test design process, leaving no entry point for them into automated testing.

These situations are not uncommon, and both of these issues affect the sense of ownership for automated testing on the team. We looked for a solution where the entire team felt like they owned the regression tests, the maintenance, and the results.

Building a Test Team of Automators

As I think back, it is easy to forget about the work it took for us to become a team of test automators. Here are some of the things we learned.

Work with What You Have

• Work with the people you have. None of us arrived on our team with the skills we have today.
• Work with the tests you have and run them every day.
• Involve the whole team on the test results analysis.
• Learn what makes a good (or a bad) test.
• Work as a team. Sometimes this is democratic and sometimes a benevolent dictatorship, but work as a team.

We started our technical education as a team with the simple step of running our existing tests more often. We did not wait until we had some great whole-team solution; we just started. Instead of running tests once a sprint, usually towards the end, we ran them every day. We are an agile shop and practice continuous integration and frequent test builds. As a result, there is always new code to run our tests against. Of course, running the tests is the easy part. We also needed to analyze our test results every day.

Our team education took its next step when we started working together to analyze test results as a group. We shared the responsibility, with a different team member leading the task each day.

Analysis can be hard if you are doing it right, and hard things take a lot of time. Tests fail because of a number of reasons: existing features change, code breaks, tests are not robust enough (too fragile from the perspective of test data, or tests are not designed to run more than one time), or there are problems with test tools.

Learning what breaks a test is handy to know when you are learning to write tests. Further, when a test fails, some tests are easier to analyze than others. After struggling to understand what happens when a test fails, you begin to pay attention to tests that are easier to analyze and to look at the differences between those and the more difficult ones. The best answers often rise out of this type of work.

As I think about this stage in our whole-team automation development, I am convinced that the shared work of analyzing the test results was the most important step for us. It changed the team from being passive observers of the automation process to active participants, and it educated the team on the differences between good and bad tests.

Test Framework and Training

• Implement a test framework.
• Get real technical training for the automation tools.

It was team expansion that spurred our next big jump in technical growth. While we were small, we lived with our old tests and thought about what we would do next. When the decision was made to expand the team, we needed a solution that would enable us to scale. This interruption was one of the best things that could have happened to us. As we looked at growing in multiple offices, we knew we needed to share the automation work and to work effectively as a team. This required a good framework and process for automation.

There are many ways to automate tests depending on your available budget. Whichever path you choose, I recommend that you think through your long-term goals, especially the benefits of implementing a suitable test framework within which you can run your tests and also your implementation methodology. We chose HP’s Business Process Testing (BPT) framework with QuickTest Professional (the former Mercury suite) as the underlying automation engine for our team. You may find that other frameworks or tools better suit your needs.

We chose to purchase the suite so we could accelerate our growth and focus the team’s efforts on building tests instead of building a framework. Using BPT and a good implementation methodology, all members work together and take advantage of previous work and each other’s skills. With a good framework, new team members can build tests and experienced automators also have opportunities to do challenging work. The framework enables new people to have success quickly.

Success within a framework often translates into greater technical confidence. Even with a good test framework, robust test automation requires strong technical skills. Do not allow a special subset of your team to own the “real automation.” Of course there will be topics that make sense for the most experienced to develop, but do not limit it to them. Make sure that everyone can do more automation than the simple, record-modify-and-play type of development.

Everyone should be able to write code for the automation tool set. To do this, your team members may need basic training on software development—using constructs such as loops, conditions, functions, and dynamic referencing. If they do not have this experience, it can usually be quickly learned and everyone will benefit. This helps prevent all the cool work from going to the more experienced automators and empowers every team member. Everyone has the same chances.

The move to whole-team automation opened our test team to a new world of technical opportunities. As a team, we acquired the confidence to solve many of our problems and to look forward to challenges we will face. Change is always challenging. Look for the right opportunities for your team and jump on those chances.