Keyword-Driven Test Automation: An Interview with Hans Buwalda

Hans Buwalda's experience covers being a developer, manager, and principal consultant for companies and organizations worldwide. In this interview, Hans talks about using keywords effectively, tests that have too many details, and the changing testing industry.

Jonathan Vanian: I’m here today with Hans Buwalda of LogiGear. Hans, thank you for joining me today.

Hans Buwalda: Yes, I am glad to join. Thanks for making time for me.

JV: Definitely. Let’s begin by just having you describe your technical background and expertise to our readers and listeners.

HB: I am originally from the Netherlands. I have been in California since 2001. I joined a company called LogiGear, near to San Mateo. Basically, what we do and what I do is everything that happens to do with testing where my focus is very much automated testing and then particularly for large and complex projects.

JV: Interesting.

HB: I accidentally came into testing. It really wasn’t my career plan, like many of us, but because of a project that I was assigned to, a customer that had problems with testing and especially with automated testing. As a solution for that customer, I came up with what you would now call keywords. Then, the year after that I presented that in EuroSTAR Conference in – which was then held in Amsterdam, which is kind of the European spin-off of STARCANADA, STAREAST, STARWEST.

Among the people in the audience were Dorothy Graham as well as the founder of STAR. Before I knew it, I was in the US and I have been here ever since.

JV: I have been talking to a lot of people who just sort of fell into testing and this turned out to be their new career path.

HB: I was not aware that testing would actually be a very interesting topic to look. There was proofing, there is programming, and developing, but actually testing can be, at least, as cool, and, particularly, automated testing. It can also be particularly complex and difficult to do especially in a large scale. If you try to automate, let us say, ten test cases you are probably going to be fine and do it.

But the moment you are getting 2000, if not tens of thousands of tests, it becomes a lot more will to get it to work, to manage it, and to understand where is what, and to maintain it. Any change in the system and the test can have a big impact on your tests which is – it makes it kind of unmanageable to do so. It is a very interesting field to be in.

JV: Yeah, definitely. Do you spend a long time working on keyword-driven test automation? Can you explain this technique?

HB: Simply said, what you are doing is you are describing the test outside of the scripting language. All the modern tools have very good scripting and programming languages. You deliberately do not put your tests in the programming language. Take them out of the programming and put them in a separate document, like in a spreadsheet kind of format as a sequence of what I like to call actions.

Each line in the spreadsheet really starts with a keyword that is sometimes called an action word, and then a couple of arguments, zero or more arguments, and then the action, like create an order. That way you can create some verbiage of the test. Then, later on, you are going to automate those actions, those keywords. Each keyword gets an action script, or if you look at our own product TestArchitect—about 350 of them are already built into the product.

Then, the effect of that is if you change in the system and the test, you only have to take that single keyword while you might be using that keyword for dozens of times all over your test.

JV: You do not have to go into the actual language itself, you could just go straight into the spreadsheet.

HB: Yes.

JV: How did you come up with this?

HB: Under pressure, you really have to – because that project involved the Amsterdam stock exchange. It has now merged with the other stock exchange. It is no longer a separate company, but at the time they were moving from floor trade to Spring trade, and just a couple of months before the deadline they found out that the testing, especially automated testing, was not going anywhere. It was all record and playback and simple scripting. There, there was the situation that a small change in the system in the test and this guy’s column on the main screen got one space more; it destroyed most of their automated testing.

JV: Oh, I am sure! Yeah, something as delicate as the stock exchange, that's not very good.

HB: We went around a couple of times and then I basically found out that it looks very much like programming where you have functions, ‘if’ arguments, and so that is what we started using in the spreadsheet. At the time, it was Plan Perfect; It was not even excel. That turned out to be kind of a very good vehicle to get that project to work.

Of course, we needed to do a lot more. You need to have a lot of subject matter expertise involved. It was still a very big project, but their keywords made it manageable. A lot of talk is about in STARCANADA is about how – not only what keywords are and how to compare to other techniques like scripting and record and playback—but also how you can make them effective.

Because simply setting keywords by themselves that only you format, it is not a method; it is just a way to write tests to make them easy to automate. To make them really effective you need to have some kind of a method to know what keywords to use and how to use them. To give an example, if you are having more – if you are just interested in testing – do you know why I tend to call that an interaction test and you only need keywords like select menu, click button, and stuff like that.

If you are interested in doing a business process, like create an order, fulfill an order, close an order, then you are not interested in those details. You do not want to see them in that particular test. That is what is kind of the talk is about, how you best can organize it.

JV: The keywords—it is really dependent on the context of the situation.

HB: Your goal is with the tests. Actually, it is the goal of the test to test like an insurance premium or something like that. The goal of the test to a certain combo box has to write items in it. Both are very meaningful tests. Both tests can be very important, but you should not mix them into one test. The strange effect that it has, which I learned over the years, is that successful automated testing is not so much a technical challenge. It is much more a test-design strategy.

The test designer will drive whether the automation is going to be maintainable.

JV: Right. I was just going to ask you, why do you see it not much of a technical challenge, but more of a test-design challenge?

HB: We actually see projects with customers. We do it a lot. The testers, not really having a technical background, they do not apply methods to find out what they need to test and in which test case. The result is that they might be very clever automation engineers, and even our company might provide very clever automation engineers, but if the test design is disorganized, then there is almost nothing that an automation engineer can do to fix that.

The only thing you basically can do is just follow step-by-step-by-step everything the test tells you, because you do not know what the focus of the test is. That is all mixed in a hotchpotch of steps, and you do not really see the forest for the trees. That is what I will talk about mostly during my talk, but I will also do additional things like show little tips to make your automation more stable, like how you deal with timing and things like that.

JV: How do keywords relate to other automation techniques, like scripting?

HB: There is a very good book about that written by Dorothy Graham and Mark Fewster. I think Dorothy is going to be at STARCANADA as well and she, in their book, they have roughly organized the tests and automation into three main categories. The first one being record and playback. That is basically what the sales guy will show you when he tries to sell you a tool. It is basically only good for sales. Once you have the tool, you should not do that.

JV: Very good, and listeners take note of that.

HB: Yeah, because it is very maintenance sensitive and a small change in your test – in your system in the test can ruin your tests. The second technique, which is a bit better, is what we called scripted. It means you have tests and you give them to an automation engineer and the automation engineer starts programming. That is better because now you have a specialist for the automation and that specialist has a background of factoring out comment sequences and putting them into subroutines, so that is already a lot better.

Then, the next step is then to externalize those design steps out of the script and those test steps out of the script and put them into a spreadsheet, which is much more accessible for just about everybody, but particularly the non-technical people.

JV: Right. You do not need to know  too much advanced coding to be able to open a spreadsheet and take a look. What are some ways that people might be doing keyword-driven testing in a real, well, not very effective manner?

HB: In STARCANADA, there is a topic in the keynote that is called “Lightning Strikes the Keynotes.” I think you know about it.

JV: Yeah, “Lightning Strikes the Keynotes.” Very popular.

HB: It is moderated by Lee Copeland. I think Copeland is also doing one of those lightning keynotes; some other speakers are like Dorothy Graham. I am going to do a little five-minute talk there as well, which I call “Misconceptions of Test Automation.” One of the things I say there is that some of the worst test projects I have seen were keyword projects. Keywords can enormously backfire. It is a very powerful technique, but with great powers come great responsibilities.

JV: Thank you for that, the Spiderman reference for everyone who is not aware.

HB: That is the point. If you get that kind of power that you basically have with keywords and you start just testing away, you start just filling out spreadsheets with hundreds of thousands of keywords. I mean, we have a project that has 6,000 test case and each test case is about 4,000 lines, so you can imagine how unmanageable that gets.

JV: Yes.

HB: There are way too many details, so that is a very difficult project to maintain. We are working with the customer now to start redesigning it, essentially from scratch to make, at least, a distinction between business tests and interaction tests.

JV: How do you make sure that does not happen, you know, that you do not go haywire with your keywords?

HB: The short answer to that is, think before you do. Think about a couple of bookshelves you want to organize yourself. You should have shelves, and the shelves should have labels before you start putting the books in there. If you just put the books in there arbitrarily, as long as it is one book, it should be probably okay. If it is a library, you will not be able to find any of that. Information that you cannot find is just as valuable as no information.

That is basically what you also do with the test. You figure out what we then called test modules. You figure out what modules of tests you want to do, which modules contains the test cases. Once you know that, then you start developing those individual test modules and then you can make that effective because each of the test modules. If you design very well you can be very sizeable, quite easily very scalable.

JV: Yeah, and I guess I wanted to jump into that. Why does using keywords make tests more scalable, and maintainable?

HB: Oh! You can make it less scalable and more scalable, that is up to you. The most counterintuitive thing that I have found—I had to learn it myself, as you know—is that it is to the testers; it is not to the automation engineer. The automation engineer can coach the testers, and in many cases I have seen that actually happens, but it is the tester’s duty to organize the tests and they need to keep details away if the details do not matter, and reversely show the details if the details do matter.

JV: I think, to me it seems it is about organizing everything before you start doing—before you start performing the test. It is all about preparation, knowing what you are going to do. Do you see it pretty often where organizations just put all their trust in the tool and do not really do much in the preparation?

HB: I see that in automation, perhaps it is kind of the default behavior. I see many customers also coming our way. If they want to do automation, the first question is “Which tool to buy?” We have a great tool that we are very proud of, but at the same time, I see the tool as a footnote. It is not about which tool to buy, it is about what do we want to test and how do we want to organize that?

Regardless whether you are going to actually do those tests manually or with a tool, the same organization principles apply. The good news is, to think about it, it does not cost a lot of time if you spend today or maybe a few days, maybe even a few hours at the beginning of your project on how you can organize your tests and then you would have that for months and years to come.

It is a bit like writing a book. If you write a book, what is the first thing? Writing the Table of Contents. Once you know the chapters, then to actually write a book is a lot of work, but it is relative.

JV: Right, you have the outline.

HB: Yeah, that outline, and that is where people probably have to think the hardest.

JV: I want to ask you, how are the demands of testing in today’s world different than how it was a decade ago?

HB: Yeah. You asked me before and I said, “Well, there is not that much difference.” I would say if I look back, I can look back about twenty years now…

JV: Okay, let us go back then.

HB: I see still a lot of the same problems and challenges for the testing industry. Testing, by itself, is a very big activity. It is probably, the realistic estimate is probably about 30-40 percent of time in the project you spend in testing one way or another, which is a good thing. It is about equal as programming, as developing. It is a very big activity, however. The demand of attention that it is getting as a profession is rather limited, that is compared to development.

You even see some companies fire their test person and hire, what they call, development—developers in tests, which I do not agree with. It is the fact that testing would actually be a profession in that such things as testing techniques and books about testing and conferences about testing. It is still something that still seems to be in its infancy. It is still relatively small-scale compared to how important it is for project success.

JV: The recognition, it is not totally there as much as testers should be recognized. I see it from a writing background. It is like the developers are akin to the writers and the editors are akin to testers. Editors are the unsung heroes.

HB: Yeah, it is kind of like that. In a way I see it as if you start in a big project, the first thing you should ask is how do I test it? Not what is the architecture or how do I develop it? I think, you should ask first, “How do I test it?” Out of that you can figure out what environment you want to use and just about everything else.

Now, one more thing that has to change, in the past couple of years I think it is agile. Since I was in software, which is, like, since my teenage years, this is the most fundamental profile change and also something that I have paid a lot of attention to because the action-based testing method that we have, it is a lot better than agile. Letting the testers cooperate closely with the developers and their test automation engineers also in the same team, that can help a lot. I really believe in it.

JV: Right.

HB: I talk about that in the tutorial.

JV: Got you. Alright. Hans, well, thank you so much for joining us. You will be at STARCANADA. Any final thoughts you want to leave our listeners in preparation for your talk?

HB: Come to STARCANADA, of course, you can come to my talk, my tutorial. That is, on Monday afternoon, I think. However, I am going to be there the rest of the conference, so if anybody wants to talk to me or discuss with me based on what we have talked about it in this interview, then people are very welcome to do so.

JV: Very nice!

HB: I will basically be around talking to people.

JV: Alright. Very nice Hans. Well, thank you so much for taking the time out of your day to talk with us.