How to Adapt to the New Age of Testing and Development: An Interview with Melissa Benua

Josiah Renaudin: Welcome back to another TechWell interview. Today I am joined by Melissa Benua, a senior technical lead at mParticle, senior backend software engineer at PlayFab, and keynote speaker at this year's STARWEST conference. Melissa, thank you so much for joining us today.

Melissa Benua: Yeah, thanks for having me.

Josiah Renaudin: Absolutely, and before we kind of dig into the meat of your keynote, can you tell us a bit just about your experience in the industry?

Melissa Benua: Yeah, so I've been all over the place now. I started actually really large company. I got my start at Boeing, then I spent many, many years at Microsoft working on Bing and Xbox and a couple of other products. And then probably three or four years ago I jumped out to startups. So I started doing, I started originally as a tester and I've since moved into a sort of combined engineering role. So I do some dev work, my passion is always testing. So I've been working, now, startups to kind of balance that evolving line between dev and test and what quality do we need or more fashion—how do we balance speed versus making sure our software is what our customers need.

Josiah Renaudin: The idea of speed that you just mentioned is something that is really integral to your actual keynote, because your talk is going to tackle ultra-modern engineering technologies and how much faster software development has become over the years. So to kind of kick things off in terms of the actual keynote, in your mind, has agile really been the catalyst for this speed, or was it really something even before that that made people realize that software has to be faster in order to really make sense in modern times?

Melissa Benua: Agile was definitely part of it. It's part of the process of it, but really it's a big, just a set of as process and technology has kind of improved over time, we've just gotten faster. Remember forty years ago we were using punch cards. That was not so quick. But as the tools evolved and the tools for the tools and the tools for the tools for the tools, everything just gets faster and easier to do. Especially around monitoring and deploying your software.

So, now we can see almost immediately when you've shipped out a piece of new code, if something's gone wrong, you can roll it back very quickly or push out a patch if you have to, if rolling back isn't viable. It's just everything has become so quick. It used to take ... I remember when I was at Bing it would take us days in the early years to deploy a piece of software to all of our servers. And now it takes minutes. So the cost of rolling back or rolling out is very low. It's just become very, very quick. So now we can make these trade-offs. We can trade off delivery speed for, theoretically, some of our quality—some of the upfront testing, I should say. You don't want to sacrifice quality, but you don't necessarily need to spend as much time to achieve the same amount of quality as you used to.

Josiah Renaudin: And now that testing is happening in parallel with development, for you, what new skills do you think testers have kind of been forced to adopt in order to keep up? Because I've been talking to people recently and it's always every time you talk about agile or the fact that testing is happening with development, testers are the ones who keep needing to adopt new skills, doing new things to stay relevant. Do you at all feel that testers have gotten the short end of the stick during this transitional period because they have this new workload?

Melissa Benua: Sort of. So the flip side of that is that the devs are also having to learn how to test, which depending on the dev is either a very easy thing or a very, very, monumentally difficult thing. But testers definitely have to pick up a wider range of skills I would say. They've had to pick up pieces of what is now called dev-ops, pieces of all the new automation tools that are popping up all over the place. They've certainly changed more in their day-to-day functioning. I'll go more into depth in my talk about the specifics of some of the new key skills and tools that are really beneficial I feel for testers to start picking up on, but there's definitely a lot of new things for them.

So the most useful skill I've seen a tester pick up has been test education. And as I mentioned that dev is having to pick up some of the testing slack since there's not as much time as there was and not a lot of devs have solid test foundations. So a tester who is able to kind of evangelize what they do and talk to a dev about how to write very testable code, how to write unit tasks so there's not a lot of overlap between what the tester is doing and what the dev is doing. And to really streamline the process and make everybody's time more efficiently utilized is probably the most useful skill of all that I've seen.

Josiah Renaudin: Yeah. And again, I've talked to a lot of people recently who say it’s that double-edged sword where, yes, testers have a lot of different, kind of varied skills they need to learn, but in a lot of ways that can kind of help them lead their teams in a way that they never really had a chance to before when you do learn these new things.

Melissa Benua: Exactly.

Josiah Renaudin: You mentioned punch cards before and we both laughed. Are we at all approaching a point where maybe ten, twenty years down the line, people will use the words “manual testing” and maybe not laugh in exactly the same way, but maybe in a similar way, because so much of the future is automation? So how much manual testing does the future of software testing hold because of automation?

Melissa Benua: You know, honestly I think we will look back at all of the manual stuff that we do—I mean even now, we look back at the manual testing that happened ten years ago and go, wow. I think that manual testing is becoming less and less a part of the bread and butter skill set. For example, there's still a lot of manual testing that happens in UI testing. The tools for automating that aren't quite as mature as some of the functional integration testing tools, but what we've been seeing, especially now in the start up where we're very cost aware, has been some Mechanical Turk style stuff where one tester in your company will write some test plans and then the Mechanical Turk companies will have a bank of manual testers that will just run your test plan and then you'll get a consensus based out of results.

So it's sort of a hybrid of manual, but not. I suspect as some of the manual tools and some of the iTools involved just a little in the next couple of years even that will sort of fall by the wayside.

Josiah Renaudin: Is that something that scares some of the testers you talk to? The fact that manual testing might not be very integral? Because a lot of people that's all they've really done.

Melissa Benua: I think it does, absolutely. I mean, and it is a little scary. It's a changing of the paradigm. We've had manual testing for a very long time now and it really does seem to be contracting down.

Josiah Renaudin: Yeah, and of course again, speed is such a big focus of software right now, especially testing, and it's a big kind of sticking point of your keynote. So, creating software quickly is great of course for all of the obvious reasons, but what are some of the security and even quality risks that teams really should be aware of in this new software landscape? Where testing and development, all that stuff, the feedback loop is so much shorter. Are there security and quality risks involved with that?

Melissa Benua: There are. The biggest one that is very tempting, I've seen it several times now for organizations to say, "Oh, we'll just have the devs write their unit tasks and as long as they're running automatically and they're all passing, we're good to go." Like it's fine. But there's a lot more to good testing and quality than having a green test pass and decent code coverage numbers. So there's a lot of potential risks in there. Where there's whole areas that aren't being covered at all. Especially if there's not any one person advocating for product quality. Somebody who knows that UI testing is a thing and the benefits that come with it. Or security testing or even integration testing. Somebody who really knows the different layers and knows what they're good for and what they're not good for. It makes a big difference to have that on your team, which is what I see a lot of. Really successful testers evolving into this.

So testers are still writing tests, but they're more focused on holistic testing if that makes sense.

Josiah Renaudin: Oh, absolutely. And I of course don't want you to spoil your entire keynote, but can you briefly discuss some of the stages of true CI and continuous development that you will be covering in your keynote?

Melissa Benua: Yeah, of course. So at its most basic form, it's basically building your code immediately upon check-in, running tests on it, deploying it past its quality gates. Your quality gate can be all sorts of things. You can say all my tests have to be green, or it can be you have to have 80% code coverage and 50 percent had to be green, or whatever.

Obviously in the real world there's more to it than that, you want to be able to fill out quickly for example, because every stage in your pipeline generally takes more and more time. Building code is usually faster than your test pass which is usually faster than deploying your code which is usually faster than the integration test pass and so on.

So having each stage have its quality bar and then having each stage have a failure path, whether that's if your build fails emailing the developer who checked in bad code. Or if your deployment fails, automatically rolling it back. Or if you're doing blue-green deployments, making sure that your code doesn't actually ever go live.

Josiah Renaudin: Yeah, and earlier you did mention that you've had experience both at really small companies and really large companies. I'm similar, I've worked at Sony and PlayStation a bit, but I've also been a part of the garage studios that everyone's kind of doing a little bit of everything, but for you, have you found that both start-ups and large corporations use similar testing strategies? Like does the size of the company really determine how you should test your software?

Melissa Benua: Honestly, it really doesn't. So all the products that I've worked on, the fastest, most agile was actually Bing.

Josiah Renaudin: Oh, wow.

Melissa Benua: Which is surprising to hear, but they deploy that software constantly, many times a day. It goes out to all kinds of machines, it's super stable, it's all automated. They actually have a deployment pipeline that is to be envied by most smaller companies who should theoretically be more agile. Really the only thing that changes with company size is responsibilities of the individual inside the company. When you're a company of, say, fifty, your testers are going to have a lot more responsibility or area of responsibility than they will if you have 500 or 5,000 people in your company all working on software.

Josiah Renaudin: All right, great. Well, last thing, and then I promise I'll let you go. More than anything, what central message do you want to leave with your keynote audience? What's kind of the main sticking point you hope they take away when they walk away from your presentation?

Melissa Benua: Software moves fast. I know we mentioned earlier about punch cards were only forty years ago. Twenty years ago we were all running on mainframes. It looks very little like what we do today and what we do in ten or twenty years is again going to look very little like what we do today. So you know, software engineering is a really rapidly evolving field and the best skill that any developer or tester or dev ops or anybody involved in producing software can have is the ability to evolve and adapt, along with how quickly we're changing.

Josiah Renaudin: All right, fantastic. Thank you so much, Melissa, I really appreciate your time today and you digging into the content. I can't wait to hear more, or the full thing, at STARWEST this year.

Melissa Benua: Yeah, thanks. I'll see you in Anaheim.

From the largest companies to the smallest, Melissa Benua has worked in nearly every software development role—dev, test, DevOps, and program management. She's created and run high-availability, high-quality services at companies big and small, on products such as mParticle, PlayFab, Bing, Cortana, and Xbox One. Melissa discovered her love of massively-scaled systems while growing the Bing backend, where she honed the art of keeping highly-available complex systems up while undergoing massive code churn. Now she's passionate about spreading those productivity-boosting practices far and wide. Melissa isn’t afraid to mix traditional approaches with bold new ideas to make her products better, faster, and more reliable. She’s passionate not only about maximizing efficiency both in her product code and in her developer tools, but also about sharing best practices among colleagues and the tech world at large.