Testing as the Driving Force of DevOps: An Interview with Adam Auerbach

Josiah Renaudin: Welcome back to another TechWell interview. Today I am joined by Adam Auerbach, the vice president of quality and DevOps engineering at Lincoln Financial Group and a keynote speaker at this year's STARWEST conference.

Adam, thank you so much for joining us today.

Adam Auerbach: Yeah, thank you for having me. I'm excited to be a part of this.

Josiah Renaudin: Absolutely, and before we really dig into the meat of your keynote, can you tell us just a bit about your experience in the software industry?

Adam Auerbach: Sure, so, I've been in the QA world for over seventeen years. I've worked my way up from being a manual tester to working in automation performance to leading teams. Most recently I was at Capital One, where I led their enterprise QA group, and when I started at that role, we were beginning our agile transformation. Then it went to the testing framework. Then it became focused on how, as an enterprise, how do we become more around product enablement, like testing management, service virtualization, and then getting people to continuous testing, and ultimately to continuous delivery.

Then, most recently, I left Capital One, and now I'm at Lincoln and I'm helping them go through that same exact journey.

Josiah Renaudin: Because you've been in the software industry for so long you have seem, like you mentioned, agile, continuous delivery, continuous integration, and kind of start to spark up, but the crux of your keynote covers how DevOps and modern software have forced organizations to change their testing cultures. Can you just detail just how invasive DevOps has become? What about software development and testing has made DevOps so essential to success?

Adam Auerbach: Yeah, I think the big thing is when agile is here, right, and people are standing up team and sprinting, and a lot of them don't realize the true benefits of why they went to agile in the first place, which is, how do we deliver increments to our customers early and often? That's where DevOps is really hitting the ground because people are saying, "What are those impediments? What is it that's causing us to have a hardening phase or hardening sprints and still have all these extra teams doing deployments, or having to do manual testing?"

The move to agile has really helped also drive the DevOps adoption, but then you also see it in the companies that are having the most success today—the Netflixes, the Amazons, Capital One. The companies that have embraced DevOps are making a difference. They're able to spend more time on product development and innovation, and they don't have to have large groups that are only operations focused, and infrastructure groups, and whatnot. You're really seeing the beacon companies, if you will, have already embraced this and that in and of itself is driving this movement, as well.

One thing that is interesting is that, depending on where you are in the country, or what industry you're in, it might be more pervasive than another, but I think everyone is hearing the word and trying to dive in to more around, “Okay, how do we get started? How does this really apply to what we're doing?" That's what brought me over to Lincoln is that, in the insurance industry, one of the industries that maybe isn't as advanced as some of the other ones, they're feeling that same, we need to be digital, we need to get products into our customers' hand early and often, and so, how do we build out automation needed to have those efficiencies in order to do that?

Josiah Renaudin: Speaking of people hearing the word of DevOps, testers are so closely involved with this, and I feel like every single day I see an article either saying "Testing is dead" or "No, you're fine, testing is okay." What roles do manual testers play in this DevOps movement? Can a tester just focus on manual testing and survive in this modern landscape, or do they have to really take on new skills?

Adam Auerbach: They have to take on new skills. No doubt. There still is manual testing, but it's going to be more exploratory testing, ad hoc testing. The role of the manual tester, having dedicated manual testers, is going away. It's an automate-first culture, movement, if you will, and that requires that testers be engineers and have some level of programming expertise. It doesn't mean that they can go out of the gate writing a Java program, but they do need to be able to understand Java, to be able to create some of the tests, be able to use some of the tools, like Selenium. There are other roles for testers, whether it's product owners, or areas of ... There is still a need for that subject matter expertise, but from a testing practice perspective, we need to be engineers. If you ask me, I guarantee that the number of manual testing job reqs that are out there has been in a decline over the last five years because of this.

Josiah Renaudin: Have you talk to testers who almost feel like it's kind of unfair that they are the ones that have to adapt, and even develop new technical skills to, maybe embrace coding, to stay relevant? Where they feel like, "Why are we the ones that who have to change so much just to kind of keep relevance in this industry?"

Adam Auerbach: Yeah, that's a wonderful question. It's true in the sense of, "Why are people looking at us? Here it goes, people are trying to squeeze testing and it's always our fault." What I would tell you is that, by embracing these skills and becoming closer to becoming an engineer, and enabling continuous testing, you're able to do so much more, and become the driving force in transformation within your organization.

Typically, when you're doing manual testing, you're becoming a bottleneck, and then you're getting squeezed and pushed and, maybe your voice isn't heard as loudly. When you're embracing DevOps, and you got a continuous testing pipeline set up. Well, now your tests are running early and often, and you're getting that fast feedback back to the developers, and you're working closer with them. You're getting the business more involved. You can actually do all of the things that you really want to do, but you get squeezed because of whatever different reason, and then you're also able to take more advantage of cloud, of some of the other enablers, like containerization.

What I have seen is that, when you start with testing, it really creates energy around this movement, around what they're doing in quality, and then that propels even further to the other DevOps capabilities that a company needs to knock down thereafter. Yes, it is unfair in some regards that people are picking on testing again, but by embracing it, you really have an opportunity to be the driving force at your company for this movement.

Josiah Renaudin: Yeah, and create a new type of career longevity for you where, instead of being holed into one thing, you now have a new set of skills that you can apply in different ways. Something you did mention in your abstract that stuck out to me was, the "You build it, you own it" idea. So can you explain the "You build it, you own it" environment testers have to keep pace with to work within a DevOps world?

Adam Auerbach: "You build it, you own it" really refers to the fact that if you have an agile team, that team is responsible for everything. That team should be a product team delivering features to production, and so that team is accountable for the development, the testing, the business sign off, and getting that code into production. When it breaks in production, that team's responsible for trouble shooting and fixing it. In that way, it keeps all the accountability with the team. There's not different hand-offs, and so, when something goes to production, the team moves on to something else. You're fully accountable, and what that does is it makes sure that the things that are happening in production, we're able to make them into our development and testing practices.

It makes sure that the things that we would do in production, like monitoring, for example, to understand how people are using it and the performance of our applications, we're taking that back into our teams and incorporating that in our non prod environment to get those team insights. It really changes those expectations of the teams, and forces you to have to better understand everything that goes into a feature, a product, for its whole life cycle, not just getting it "done" from an IT perspective, but everything that goes with that.

Josiah Renaudin: I think so often, when we get into terms like DevOps and agile, we get really far ahead of ourselves and we don't actually break down the basics, so what are some of the core values of the DevOps movement that you think are important, especially to testers?

,: This is an interesting one. The first one for me is a focus on automation. The core with DevOps is around lean manufacturing and driving flow, and so, how do you look at, holistically, the testing process and understand where are those manual handles? Where are the wait times? Where are the constraints? How do we start to knock them down through automation?

The first one is automate everything. The second one is fast feedback. How do we build out these tests in such a way that the developer, they check something in, they get a cup of coffee, and they're starting to get test results back, and so it's clear to them very quickly if something is working or not. Everything has to fall in line with that fast feedback. Fast feedback for unit tests, fast feedback for integration tests, fast feedback for performance and security. If you look at testing cycles today and what happens in hardening with performance, and regression, and security, that's usually too late. How do we shift that left and make it happen early and often so that way developers, in the moment, know if something doesn't work, and they know what they did so they can go fix it so there's not contact switching and they forget.

Then the third thing to me is just that focus on flow. The whole purpose of DevOps is to be able to deliver a high quality software product early and often. Again, how do you remove those bottlenecks through test automation? How do you remove constraints on environments through different third parties that, maybe, have Selenium grids in the clouds that allow you to do all the browsers or devices? How do you use containers to spin up and spin down environments? You have to embrace all of those different enablers so that you can focus on flow. Running all of my tests multiple times every day in order to be able to put something in production early and often.

To me, those three items are the pillars that you have to focus on from a testing perspective for DevOps.

Josiah Renaudin: You went into detail about test automation there, which is one of the definite way that testing has kind of evolved to keep up with DevOps and agile. We've also talked about how testers need to learn new skills, maybe even code. Testing has evolved in a lot of ways. In what ways do you still think it has room to grow in order to keep pace with how face everything is today?

Adam Auerbach: Obviously, the first part of that change is moving to more open source tools. Moving away from some of the big third party automation tools and adopting the open source like Selenium and Cucumber, and Expresso, Appium. Really using them, because this how you're going to be able to automate in real time and keep pace with developers.

Then obviously, the next thing about that is learning the CI tools. Jenkins, Bamboo, Circle CI. How do you get your tests to be executed through Jenkins jobs, for example, and be able to have results go back to those jobs so that developers know.

Then the next thing is around test data management. We have to have tools to be able to procure our data in an automated fashion that can be hooked into our pipeline. Service virtualization because I have downstream constraints. All of these things go into the new skills that we have to learn for continuous testing. I think the opportunity is in the containerization cloud. Really being able to take control of the environments for testing. Being able to spin them up and down. Having a good understanding of how to use them appropriately. Performance and security, there are great open source tools that allow you to do that, and how to shift them left.

To me, it's how a tester can get an understanding of that full DevOps ecosystem to be able to build the tools and capabilities that they need in order to do this. A lot of times, you'll that the focus is just on the automation aspect, but I think getting full rounded picture, to your point earlier, really will then help them open up more doors. I've seen it where a testing group is leading the way for docker implementations, because they've learned that skill and they're using it for their environment. I think that just a bigger focus on some of those CICD capabilities is the next part of this journey. That open source is, obviously, the key building block to it.

Josiah Renaudin: Earlier we kind of talked a bit about how it can almost be unfair for testers. At least, testers could feel like it's unfair that they have to adapt so much just to kind of stay in the industry they've been in for so long. A lot of this conversation makes me think that DevOps is also empowering, to a certain extent, to testers. You mentioned that they can kind of be leaders of their teams. In your mind, do you think DevOps is empowering for testers? That it can help them have a larger role in this sort of testing transformation?

Adam Auerbach: Absolutely. As a tester, you want to spend more time looking at what the business is asking for and putting on your critical thinking hat to understand how it's going to be interpreted and used, and it gets you so you're able to do that because you have the automation that's happening, and you're able to keep pace with development. Then, also, because you're getting close to development, you're getting a closer knowledge of how things are working and how they're designed.

In environments where this isn't happening, a tester might not even know what the APIs are called, or how to invoke them, or what information is being sent to and from. By embracing this culture, you're really getting into all the nitty-gritty for how an application is developed and architected. It absolutely will open up more doors from a testing perspective to really ensure that you're producing a quality application.

Josiah Renaudin: All right, and just to kind of wrap everything right here, to kind of put everything together, more than anything, what key message do you want to leave with your keynote audience that will mostly be testers in the audience, but will be a mix of people? What kind of main message do you want them to walk away with?

Adam Auerbach: I think that big thing for me is that they are key to making this transformation happen at their company. Starting with testing really creates those building blocks to then drive that transformation, and then they themselves, by embracing this and learning these new skills, and tools, technologies, they will open up so many more doors for themselves and their company. Testers have the opportunity to, instead of being at the tail end of a process, or whatnot, to be there from the very beginning. That's the biggest key that I want people to take away, is understanding what this is and where they can get started so that they can be an active participant as their company makes this journey.

Josiah Renaudin: Fantastic. Thank you so much for taking the time today, Adam. I appreciate the conversation and am looking forward to hearing the full thing at STARWEST.

Adam Auerbach: Awesome. Thank you. I appreciate it.

Adam Auerbach is the Vice President of Quality and DevOps Engineering at Lincoln Financial Group. Adam is responsible for leading the implementation of Continuous Testing and Continuous Delivery across Lincoln. Prior to joining LFG, Adam was the senior director of technology for advanced testing and release services for Capital One Financial Corporation. Adam led Capital One’s enterprise performance and automated testing departments as well as enterprise release management and test data management. While at Capital One, he provided leadership for the agile transformation of their quality assurance group and led the enterprise adoption of DevOps and acceptance test-driven development. Before joining Capital One, Adam was with Chase and other financial and insurance companies, in various leadership positions focused on quality and agile practices.