Whole Team Quality: A Conversation with Melissa Benua

Melissa Benua

If we think of quality as just a checkbox—one more hurdle we have to fulfill—it's always going to be something we do as an afterthought, and it's always going to be the first thing cut. But it's not the least important thing to our customers. So if you think of quality as something we do just to make the process happier, because we have to, then we're going to end up with products that are A. not what our customers want, and B. don't work as our customers want them to work.

Owen Gotimer

There's a clear problem if it doesn't work. What are the business implications if the product you're delivering is not up to the quality the customer expects or to what the customer actually wants?

Melissa Benua

So that's a big problem, especially when we're doing testing, especially when we talk about who's going to test this developer's project. But oftentimes, automation engineers as well will fall into the trap of we're going to functionally check that this does what I mean it to do, right? They're going to make sure that when they give it a 5 it returns a 10. But they often don't check, and in fact really can't check, that it does what the customer wants them to do because checking your own understanding of the requirements is like trying to proofread your own essay. It's not going to work.

Owen Gotimer

There are important pieces of having functional testing. Functionally things need to work.

Melissa Benua

Absolutely, functional testing is important. I am 100% behind functional testing, roll your automation. Functional testing is super important. But we as an industry tend to neglect the non-functional testing,

Owen Gotimer

That non-functional testing that you know more of the human element, what is the customer expecting this to do? What does the customer want it to do? Which is another big thing. How do you have those conversations with customers? Obviously you have people that can do those non-functional tests as part of your team. Is it important also to go out into the world and talk to the customers and have them do some testing, as well?

Melissa Benua

Absolutely. This is sort of where whole team ownership of quality comes in. So in my teams that I've worked on now, in the last pretty long time now, we don't have QA as a role, for example, which doesn't mean we don't test, it means we've divided up those pieces, right? So the devs do functional validation. But our product owners, if we're talking Scrum and Agile, our product owners are in charge of making sure the requirements are correct. And not just that they wrote the requirements, but that they met with the customers, that they understood what our product is trying to solve as a problem for our customers, but making sure that when we then deliver a piece of code that they have ensured that it meets the needs that they originally set out, because nobody knows the requirements better than the person who wrote the requirements.

Owen Gotimer

Absolutely. One of the things you mentioned is some organizations potentially moving away and have already started to move away from having people that are QA or testing specific roles. And the whole team is responsible. But that often means that there are developers now that are responsible for doing more than traditional unit testing, for example. Unit testing, obviously, is very important. Moving past that unit testing, how does that shift the paradigm when you have developers now responsible for doing the quality?

Melissa Benua

Yeah, so I've seen teams generally speaking, either do very, very well or very, very poorly when they move to this sort of combined engineering model. And there really is no in between: you either sink real fast or you learn to swim. So the teams that do it well are the ones that acknowledge that you've been a developer for you know, 5, 10, 15, 20 years, you're probably pretty good at what you do. You have a lot of domain expertise and mastery, and they approach testing as a discipline and a skill, like any other that anybody can learn. You can learn to master SQL database, you can learn to master cloud services, you can learn to master testing as a discipline, and they treat testing as a discipline just as important as any of the others. And they help everybody to learn, not just "yeah, I wrote some unit tests to bang out to match my feature, and it's probably fine." But why do you write the unit tests? How do you write a good unit test? How do you make fast unit tests? And then what do you do beyond that, right? How do you integrate your code together so that each piece you've tested also plays nicely with others?

Owen Gotimer

And you have developers now trying to master this new testing skill? What are some of the biggest challenges you think, maybe in your experience or stories you've heard, of trying to get developers on board with learning this new skill?

Melissa Benua

Yeah, so it's a lot of a matter of they don't know what they don't know. And a lot of developers know just enough to know that they think a lot. There's a term for it that I'm not going to use. But they know just enough to know that they think they know everything, but they actually don't know this huge universe that they've just never been exposed to. So a lot of developers, when you say, "Yep, you're responsible for your own testing," say, "Okay, I'm gonna write one or two unit tests." They don't really know the actual rules of unit test, but they're gonna remember how it was taught to them by whoever mentored them aat the time. They don't know any of the theory behind it. But they're gonna write some unit tests, and then they're gonna maybe click around a little bit, we'll start the application, they'll click around, they'll say, "yeah, this looks about right," and they'll consider that testing. And that is technically testing. But that is not ensuring that the product is going to do what we want and how we want and in a safe and reasonable way, right? It's not a whole solution. It's a very partial answer.

Owen Gotimer

So maybe we can walk through the process. So you have a new developer who's never been responsible for quality. You're trying to get them on board and trying to explain to them why it's important and what they can do, what skills they can acquire and skills they can learn to help the team drive quality. So unit testing is one of the first things you mentioned with developers. And that's something that developers can routinely do is unit testing. I guess maybe start with for someone new out there who's maybe not familiar with it, what is a unit test and why are unit tests important?

Melissa Benua

Yeah, so a unit test, which is the foundation of any functional testing, is a very small test that covers a unit of functionality, one function, a couple functions. If you get religious about one unit test must be one function, but it covers a very small piece of function, and, most importantly, it doesn't cross any application boundaries. So it lives on the box. Unit tests must never leave the box. That's usually my shorthand test for when a new developer is trying to show me his unit tests and I'm suspicious of them: turn the internet off, if your unit test still passes, when you turn the internet off, I'll probably allow it. So it's a very small, well-controlled, single test that takes a very simple validation, right? You pass it in a 5, you assure that the object that comes back out has field one that's an 8, field two that's a 9, and field three that's the string that looks like this.

Owen Gotimer

An analogy that I heard that helped me understand unit testing is from Nathen Harvey at Google. I'm gonna paraphrase maybe a little bit. This isn't a direct quote, but he was explaining to me kind of like sewing. When you're sewing, you're putting together a garment. That's maybe the product. You have a seam of threads. Those seams would be maybe an entire functional test. Is it functionally holding the piece together and the unit test might be looking at one specific thread. Was this thread done properly? Are we testing the the security of this thread and knowing that if this unit works, then the other units that we sewed the exact same way in the exact same string could potentially work as well?

Melissa Benua

Yeah, I like that. That's a really good analogy, actually, especially as it ties all the units together into a function or into a set of functionality. That's what our integration tests are for is making sure all those pieces connect together and line up and make sense as a whole.

Owen Gotimer

So from a unit testing perspective, we know what it is, we know why it's important, because we're checking the functionality of isolated, small pieces to make sure they work. What's the next step after unit testing? Moving through this pipeline, we have some unit tests that work. Now, we need to make sure that these pieces of functionality also work together?

Melissa Benua

Yeah. So as I said, that's the integration testing or as Nathen probably called the functional testing. It's called a lot of names. Sometimes contract testing API, like if you're especially if you're at an API level, but what you're doing is you're taking two different discrete things and making sure that they can speak to each other. And usually it's across an application boundary or across some sort of major boundary, an HTTP boundary or file boundary, or any kind of major IO distance, where you're going to need to have some extra complexity, and it's making sure that those two things can speak together. A big trap that new developers often fall into is they'll have two or three unit tests, because maybe unit testing is hard or because the architecture isn't set up to support good unit testing, and they'll have tons and tons and tons of integration tests trying to test a full test matrix of inputs at the integration level. And that's not great. It's not to say you can't do it, but generally speaking, because an integration test is crossing this boundary, they're 10x to 100x slower. You can get a unit test done in a couple microseconds. Maybe it's slow, and maybe it takes one whole millisecond. When you're crossing, say the HTTP boundary, you don't get it done for less than a millisecond, maybe 10, maybe 20, maybe a second or two, especially if you start wiring in some UI up there. It's many orders of magnitude slower and more brittle and more complex when you move to the integration level.

Owen Gotimer

That's why if you're looking at the the testing pyramid, we have unit testing at the bottom, API or functional testing, and then UI, maybe some form of AI, some level, potentially up there. But at that bottom, the unit test move fast, which is why they are well, one functionally, we're making sure things are working on their own, but two, they run so fast, which is why we can have so many more.

Melissa Benua

I run 20,000 unit tests in 90 seconds. 90 seconds. 20,000 tests. I run about 1,000 integration tests in four minutes. Those are real numbers from a real build system.

Owen Gotimer

It's so important before you move from that unit testing portion of the pyramid to the integration testing, because what you don't want to do is start those integration tests and have unit tests that fail, unit tests that are not properly written, or are not working properly. If they fail, we need to address that.

Melissa Benua

Exactly. I've often seen the other mistake, mistake number two, that new testing people make is they'll write unit tests, and all the unit test does is validate that a function was called and it didn't crash. They don't have any search. They don't really look at the output. They don't look at any side effects. They just say, "Yep, I executed the function." And they sure did and it didn't crater therefore it's tested. And that's not what tested means either.

Owen Gotimer

How important is it to have unit tests that you have tested? You want the unit test to make sure that they're working properly before you move to that next level.

Melissa Benua

We accomplished tha with in-person reviews, with peer reviews, to ensure that all the unit tests are treated the same quality by that production code is that somebody has looked at the testing effort and said, "yeah, this testing effort is what we want." The unit test is actually checking something. We're not just trying to artificially boost up our line coverage or our code coverage, we're actually checking something important, and we're checking the right things.

Owen Gotimer

Now, we know we have the right things that are being checked, we've gone through the peer review, we have unit tests that we know are running properly, whether they fail or not, we know that at least they're good tests. We get to that next level where we're doing integration tests, and we're able to connect some of these things together, then we get to that next level, on top of the testing pyramid.

Melissa Benua

So I have started calling the top of the pyramid, just our specialty tests, because there's so many things that we stuff in there. And while specialty tests generally mean anything, it's not a unit test and a sudden integration test. It's something else. Because for example, if we say our end-to-end tests are there, well, sometimes you can run an end-to-end test really fast. You can make it more than 99.9% reliable, then why not just call it an integration test. It doesn't need anything special. But on the flip side, maybe you've got a test that's super slow, that's three or four hours. We do some security vulnerability scans, they take about three hours. They're still important, and you still want to run them, but they're slow. We can't run them with every single change. Maybe you've got some end-to-end, UI-based automation, and it's very brittle. But on the flip side, there's a couple of tools out there. Now one of our favorite is called Cypress IO, that you can write true integration tests at the UI level, including validating actual elements, because you can make them very fast. You can make them very reliable, and you can use Mock to separate out the interface boundary. There are integration tests in the truest sense of the word, but they are UI automation.

Owen Gotimer

I think that you mentioned there the speed, too, and you mentioned a few times, you can run 20,000 unit test in 90 seconds. The speed is obviously a big part, as long as we have good quality tests in place that are measuring the quality that we want to measure. What are we trying to test? We have the right requirements in place, and we have good quality tests in place, and we can go fast. I think that being able to help the team understand this is not a blocker. This is part of the build process. If we make sure we have good quality initiatives in place and that we're doing it in a way that we can rapidly get these tests and this quality checked and then maybe end-to-end tested, people are probably going to be less scared of wanting to get there. They know that if there's good quality and we can help educate them. If there's good quality upfront, we'll get to limit the rework later.

Melissa Benua

The power of a good regression suite is hard to overstate, because if you have a solid functional regression suite, whether you cover especially at the unit level, but even if you're up at the integration level, your ability to make changes, save changes as a developer is something like 100x faster. Because you don't have to wonder if it's tested, you don't have to click around and try to figure out inventing some new way to test the thing because we've never tested it before. Either you're making a quick change, and it's actually already tested, you don't have to write a single test for it, or maybe you have to update a single line in a single test, or even if you have to write new tests, there's already a pattern. It's already a paradigm. When developers are writing unit tests in particular, they tend to write more unit testable code, so it's a virtuous cycle of if developers are testing, they write code that can be tested. A big complaint I see from new developers who are new to testing or automation engineers is the devs have written this code, and it's completely untestable. It's got a lot of global variables. It saves that state in weird ways that you can't, you can't cook in, you can't break it apart. It could only be tested, which is why you see all that UI end-to-end automation, it could only be tested from the UI and only end-to-end from beginning to end. They can't break it apart at all.

Owen Gotimer

We need to write testable code. As developers start to write more unit tests or code, they start to write better testable code. What are some steps to get them started in writing more testable code? What can developers do to their code to make it easier in the long term for quality to be checked and tested?

Melissa Benua

So it's always an interesting question, because if you're starting with a legacy code base, maybe your code base is 20,000 lines of code, maybe it's 200,000 lines of code, and you haven't written any unit tests for it, it's going to be very difficult. But there's gonna be some low-hanging fruit in there somewhere. So there's two potential approaches, right? If you're already dealing with a huge legacy system, find your low-hanging fruit, and then just set a really low bar. I set the bar the first time at 3%. Any change that you're making has to have 3% unit test, blind code coverage. It's hard to argue with that sort of ridiculous sounding metric, like no developer will say, "I'm not gonna have 3% code coverage." But what it does is it makes them think about, "okay, I just have to write one test. How am I going to write this one test?" And they'll subtly change how they're writing the code. If you're writing a brand new system, a lot of developers go really, really into design patterns, and there are some design patterns that are much more testable than others. So send your developers to go research design patterns, go poke around on open source projects, go see stuff that's got high powered ones, usually they have high coverage. They're all about advertising that on the front page of their GitHub. See what other people have done, go to it, go to a conference, see what other people have done. If you're starting a new project, and you're not sure how to make yourself set up for success. There's lots of really good examples out there.

Owen Gotimer

One thing I was just thinking of right now is do you have an example of a situation in your career when this happened for you or someone else that you were working with, where a developer had that aha moment and thought, "Man, if I do the test, if I write my code this way, then it's going to make quality easier." Is there like a specific example you can share?

Melissa Benua

I suppose once upon a time as an SDET, and so as we became combined engineers, we could see actually, every single developer on the team go through this aha moment. Because the joke was that every single developer who became a combined engineer and a combined engineer team wrote exactly one major bug into production. Exactly one. No more and no less. Because every time they thought they understood that their code was testable, and they thought they had tested it, and within weeks, they would have written that huge bug into production, realized "oh, maybe there's something I don't know here," and would always go back to the drawing board and say "okay, obviously I didn't test that. Here's a very clear hole in my understanding about the code that I've written and how I've designed it and how I've tested it," and every one of them figured out either who to ask or who to get advice from on how to close that hole in their understanding. And they never wrote that level of bug again. I didn't say they didn't write new bugs, right? They wrote new and exciting bugs instead of the same bug over and over and over again.

Owen Gotimer

Once they've taken that initiative now to ask for help, "I need a better understanding of why this is here and what I can do to help fix this," they're also learning, because they're not repeating that type of bug or that level of bug anymore. So I think that's super important that we're we're learning new skills as we're building the pipeline. One of the new skills that I think a lot of developers maybe have a lot of skill already, but maybe some testers who are transitioning to test automation engineers, they don't have the skill is the coding. The application of putting tests into code and automating things. So from an automation perspective, we're looking at the testing pyramid. Where should we be starting our automation? Like day one, we've not had any test automation, we've never done any, where are we starting? What should we be automating first from a test perspective?

Melissa Benua

So it's just as if you're a test automation engineer: start at the integration level, if you can grab your APIs. But the most important contract boundary you have, especially if you've got some sort of web service, and we almost all do at this point, is find your API level. That's a really easy boundary to control. And just first explore it. Don't write any automation, just explore the API, what's the documentation say it should be doing by send requests in via Postman, what is it actually doing? And make sure that you explore that, that you understand what the API is doing, and then write automation that captures what it's doing right now. Because it doesn't matter if it's a bug or not, if it's in production already, and nobody knows what it's doing, it's a feature, right? So you just want to write automation that captures what it's doing, and start running it, start running it with every change that a developer makes, that you both make yourself and you make them aware when they've changed it. Because they may not necessarily know they've changed it. That's a bug. You accidentally change the behavior or the surface layer of an API. That's a pretty big bug. Customers hate that. You're not supposed to change API contract versions, it's a whole mess. That's usually where I would start. And that's pretty easy to understand, pretty controllable, pretty narrow surface. You got your APIs done. Developers see the value of that, product owners see the value of that. It's a pretty easy, high-value, low-cost initiative to take on.

Owen Gotimer

Which I'm sure management loves to hear.

Melissa Benua

Management loves to hear. They love nothing better than high-value, low-cost initiatives.

Owen Gotimer

And at that same time, you're educating your teams, they're learning these new skills, they're making themselves more valuable to the team, because they know these things. Then when you onboard new team members, you've started to create this team culture of quality, and the new people that you bring on for one reason or another, buy in.

Melissa Benua

There is something very important in that, especially if you're an automation engineer. If you're an automation engineer, your focus should be on the automation frameworks, and your job should be helping everybody else in the team understand how to write tests within that framework, because that's the difference between having a whole QA team that just does the testing and is very siloed versus being a part of the team. If somebody's throwing something over the wall at you like old style test-to-SDET sort of situation, the team hasn't actually taken ownership of quality at all, they've just microized the process and probably severely understaffed the whole thing. You're gonna be writing tests until the cows come home. You'll never catch up. Your job is to write and maintain a really good framework as a subject matter expert just like any other and help guide the developers in making sure that they are writing good tests inside the framework.