Test Automation's Abilities, Myths, and Future: An Interview with Robert Muehlfellner

Noel: So, to give everyone a better idea of your background and what Ranorex specializes in, can you take a minute to go into that a bit?

Robert: Ranorex is dedicated to fundamentally improving the quality of software applications. The company’s products, led by its flagship Ranorex software automation framework, allows testers and developers alike to thoroughly test applications from a user’s perspective, making bugs easier to identify and eliminate.

Noel: Test automation is certainly a topic being discussed a lot these days, as well as one that I believe will only grow in usage around the world. Even though automation been around as long as it has, I still feel like there are certain aspects of it, its scope, its abilities, and even its limitations that may not be fully understood even today. Are there any areas of test automation that you feel may have more misconceptions than others?

Robert: I agree with your assessment of where test automation stands. At Ranorex, we focus on functional testing through the user interface (UI), so I want to concentrate on those aspects. I’ll be the first to admit that test automation isn’t the holy grail to achieving defect-free software. But it is an important puzzle piece that when properly connected to its surrounding pieces helps to produce a homogenous picture of software quality. In other words, test automation isn’t the solution to all software problems; instead it is becoming an increasingly valuable part of any software development project.

We are living in a fast paced world, where both consumers and businesses with their buying power demand ever-better products. Undoubtedly software is becoming a growing companion in our daily lives–think of all the time we spend online and with our electronic gadgets–and because of increased competitiveness to reach consumers, technology is fast paced and product life cycles are short. All of this is increasing pressure on software development: more users demand increased functionality, graphically richer and more intuitive user interfaces, all while having less tolerance for software defects and more possibilities to compare similar products before purchasing one.

I believe the shift towards agile development processes is a direct result of user demand for better software, being able to quickly react to changing requirements and satisfy quality issues in a timelier manner. Why does this matter for test automation? Because test automation done effectively within an agile process will lead to an increase in efficiency of resources involved and thus improve any software business’ bottom line.

So how can test automation help with the effectiveness of software quality control? This is where some misconceptions suggest it can’t, because some may say test automation tools are:

• Expensive
• Difficult to use
• Require programming knowledge

The automated tests those tools create:

• Can’t be integrated in an agile environment as they are created only once the development has been launched
• Aren’t flexible as their data is hard-coded within the tool
• Lock testers to a test automation tool due to their proprietary nature and closed architecture

Based on such claims, test automation truly would be a poor fit for many, especially smaller development teams, and early on in the history of test automation those claims were largely true.

However, test automation tools have come a long way. Their improvement has been driven by the same demands on better software as any other software.

So, the misconception of test automation being inefficient is caused by a historic weakness in tools not living up to expectations, especially in an agile process.

Consider how test automation will become much more attractive if:

• License costs are lower because one tool can be used for any UI technology of any desktop, web, or mobile application
• Test maintenance cost is reduced because the test automation is robust against UI changes, so e.g. a moved button or changed text doesn’t break the automation
• The tool is easy to use for a typical domain tester creating and executing tests through a graphical UI without scripts or code, while a test automation expert or developer can add functionality if needed
• The tool is open and flexible allowing test automation projects to be integrated in the same ALM tool, version control system and test management tool as the development software itself
• Test data can be created and maintained outside the tool using standard interfaces and thus eliminating a vendor lock

When evaluating if test automation can help you, don’t overlook the importance of the tool itself. If you are using a tool that offers the above mentioned features, not only will be your testing become more efficient – as in doing the right tests – but also effective – as in doing testing the right way!

Noel: You and I were discussing some topics for this interview and one of the things you mentioned as being an interest of yours really struck me. You mentioned how test automation for mobile applications, another huge topic these days, is often able to "easily integrate into existing processes, and toolchains can be a huge advantage for testers and developers alike." This sounds like one of those areas in my last question, and area where perhaps people may fear a lack of that easy integration." Is this the case, and can you maybe go into how that integration is done easily?

Robert: Compared to desktop and web development, mobile application development itself is comparatively young. It took a while for the underlying platforms themselves to stabilize. Once platforms were stable, tool manufacturers became more eager to invest into making continuous integration and test automation tools available for mobile development.

We hear from a lot of our customers who either already have, or are planning on offering mobile versions of their existing desktop or web application. Early on we observed a trend towards outsourcing mobile app development to 3^rd party specialists. I believe this was caused by mobile development being quite different in terms of ALM integration and testability. The downside however was a disconnect between in-house and outside development and testing groups.

The ideal scenario of course is to use the same development tools and processes – IDE, CI, version control, test management and automation etc. – for any kind of application. Luckily there have been a lot of exciting products introduced into the market allowing just that.

For instance, being able to have existing C# or Java developers now develop mobile applications using their familiar environments means in-house development can be done without retraining efforts.

Advances in capture/replay technology allow user interactions to be recorded directly on a mobile devices using the same mechanisms that are proven to be robust against changes – meaning a basic change in the UI doesn’t break the automation in desktop and web test automation. Validation of application results can also be done on the mobile device in the same way known from desktop and web test automation. As many mobile apps are connected to web applications, it is also possible to record inputs on a mobile device and validate results in a web browser.

Finally, mobile web test automation is another area where I am seeing a lot of interest right now as mobile versions of websites or mobile browsers embedded in mobile applications are getting attention from developers to satisfy demands from customers for a richer mobile experience. Using innovative capture/replay technology helps to reduce test implementation time for mobile applications, especially in an agile environment where close coordination between development and test team members is necessary to manage short release cycles.

The same applies for using existing continuous integration tools as those as well can now automatically build and deploy apps onto devices, where they can be tested using existing test automation tools. Test automation experts can use their existing skills in managing tests, even using existing test cases of a desktop application and simply rerecording it on a mobile device. Automated cross-device testing allows testing of iOS and Android versions of an application to be executed simultaneously on multiple devices all while being managed from a continuous integration process.

Another important aspect of mobile test automation is the ability to instrument the mobile application without having to modify the source code or the mobile device itself, enabling test automation to be embedded within a fully automated continuous integration process.

Noel: I was really interested in a recent blog on the Ranorex website that I read, "Keyword-Driven Test Automation Framework," that focuses on how this type of automation aims to separate the automation from test case design, as well as enabling tests to be developed earlier in the QA process. How does this type of automation allow for this earlier development?

Robert: The idea of keyword driven testing is to allow the definition of a test case to be done independently and earlier than the implementation of the test case itself. For instance keywords can be used to describe a sequence of user inputs derived from the specification of an application at a point where the application may not yet exist. These keywords have a user focus in mind and are completely independent of the technology and test tool itself. So they can be defined by resources familiar with using the application, but they don’t have to be test automation experts or developers. Once the keywords are defined, the accompanying test cases can be filled by testers, e.g. through recordings of the actual application.

The time savings result from setting up the testing structure in parallel to the development. Likely not all test cases are always suitable for automation, so those that aren’t can be done through manual testing while the remainder can be slated for automation. The keyword driven approach allows flexibility for growing both the number and modularity of the test cases as well as any automation in parallel as necessary. Especially in an agile process with short release cycles, testing can grow alongside development using the keyword driven approach.

Noel: This reminds me of TDD, Test-Driven Development, and ATDD, Acceptance-Test Driven Development. Where is automation's place in each of those practices?

Robert: Automation definitely is an important part of TDD and ATDD.

In my opinion, TDD is an interesting approach to software development and I will leave arguing for or against it to the experts in that specific field. Generally speaking, test automation can be a part of any development process. However in the case of TDD this is best done through code-based test automation frameworks for unit tests, which also require development and maintenance resources. Still, a good unit test is the foundation for a successful functional test, especially for more complex applications.

When it comes to test automation, ensuring testability to me is an important aspect of TDD and ATDD as a developer of a code unit can certainly support the test automation by providing necessary UI control methods to allow object recognition.

At Ranorex, our expertise is in functional testing through the UI and therefore we are closer to ATDD as in this development method. Testing the application to the specification has the user in mind and UI test automation simply mimics the user for more efficient automated testing. With ATDD and an agile process, a test case can directly be derived from a user story of the application’s requirements and can be implemented right away, e.g. using the keyword driven approach discussed previously. Test cases can be defined upfront, but automation can be added on an ongoing basis in parallel to the application’s development.

With continuous integration, visibility of failed test cases is provided through detailed reporting, which provides an overview of where either tests are incomplete or the application is failing the test. A key advantage of ATDD from a resource perspective is that testing and the associated automation can be performed by resources focused on knowing the operation of the application—who aren’t necessarily skilled developers. Instead testers can have more of a user’s perspective, which helps to ensure a quality user experience right from the start.

With any test-first approach to development, refactoring of the code after initial passing of the test with subsequent retesting is part of the process. Therefore it’s important for the test automation to be very robust against changes of the application, e.g. by using Xpath information for object identification to separate a UI object and its underlying methods from the automated test case. This means that even if the underlying code is being refactored, but the functionality a user experiences hasn’t changed, it shouldn’t break the test automation.

Noel: I recently did another interview with a big proponent of test automation at HP, and we were discussing the future of test automation. She predicted, due to the success that companies are having who are already doing this today, that we may see a lot more "test architects" employed in the future, do to their specific abilities in regards to test automation. What value do you see in the role of an experienced test architect when it comes to test automation?

Robert : I think approaches like test driven development that we already discussed or agile in general with its shorter release cycles, and also continuous integration all put a heavy emphasis on testing. But in order to satisfy the increased demands on testing, it needs to be well thought out in terms of approaches, processes and tools applied.

I see a test architect as the counterpart to the software architect. While a software architect has a focus on the architecture inside the application, the test architect concentrates on the outside of the application ensuring that requirements are met following a black box principle. A clear separation between development and test within a team with defined transition points within the overall process will be instrumental to meeting increasing demands on software without an increase in organizational and resource overhead. Testing is no longer an afterthought, and it requires both attitude and aptitude different from traditional development. This is something a test architect brings to the table.

Noel : And then to wrap things up, I was curious as to what you see test automation being utilized to accomplish in the future?

Robert: In the future I don’t see any drastic revolution, but more of a continuous evolution of the journey we have already begun. Software will continue to grow in importance is many aspects of our daily lives and poor quality of software will simply not be acceptable going forward just as we expect great quality in any hardware we buy – whether or not is has electronics inside. At the same time, the cost of achieving and maintaining software quality can’t spiral out of control. To achieve higher quality, testing coverage will grow in importance to ensure both initial and long-term quality. As a basic example, regression testing can help to sustain quality while both maintaining existing and adding new features.

Basically I see test automation as a key element to achieving and attaining high quality software as traditional manual testing simply will be ineffective in terms of resources and associated cost required.

From the perspective of an automation tool vendor, our future contribution to the expansion of test automation is to break down the entrance barriers and continue to make test automation more attractive by:

• Offering a test automation tool that is easy to use by a typical domain tester without programming skills
• Supporting existing and new UI technologies allowing test automation to be used for any application with a single tool
• Making the tool smarter to reduce or eliminate the manual steps necessary to increase robustness against changes
• Offering an open architecture and integrating the automation tool into any development process that is utilized, so the tool itself is really just a tool – albeit an important one – and not the focus of the greater testing process itself

Thank you for the opportunity of sharing my views and good luck everyone making better software!