In this interview, Adam Auerbach, the senior director of technology for advanced testing and release services for Capital One Financial Corporation, shares his company's experience implementing continuous testing and describes its core principles.
Jennifer Bonine: All right, we are back with our last virtual interview of day one of STARWEST 2016, and I have Adam with me. Adam, thanks for joining me today.
Adam Auerbach: Thanks for having me.
Jennifer Bonine: Adam and I have chatted before, so you might have seen some of our talk before, but you work for Capital One.
Adam Auerbach: Yes.
Jennifer Bonine: You're doing some pretty cool stuff there.
Adam Auerbach: Absolutely.
Jennifer Bonine: Maybe one thing that's of interest right out there is just when you come to these conferences, I've noticed especially at this one, there's a lot of financial institutions here.
Adam Auerbach: Yes, there are.
Jennifer Bonine: There are a lot of banks, so if you're in that business, there are a lot of you here.
Adam Auerbach: Yeah, it's interesting because there's people from Wells, there's people here from Barkley's, from Chase and Amex and Discover, and it's interesting because we all have similar constraints, right? We're highly regulated environments. Obviously security is important to everybody, but when you talk about continuous testing, continuous delivery, there's some hesitation there because in this type of environment, how do you make sure that we're managing risk and controls. There's been some really good dialogue and I think from things that we're doing at Capital One, there's a lot of information to share.
Jennifer Bonine: Right, and when you talk about risk, when you traditionally talk about certain industries have different risk tolerances so to speak and start ups and some younger companies tend to sway towards the high risk and you guys kind of sway on the other side. Which we're all glad about that keep our money in your banks and financial institutions that you're more risk averse because you're dealing with our money, right, and our ability to use that money when we need it. We've seen historically what happens when banks lose control of that or the ability to control the money supplies. You guys have to do a lot around keeping that safe and keeping it—
Adam Auerbach: It's real interesting. The thing that we're finding now is as we move towards continuous integration, continuous delivery, we actually become better managed and less risky by decoupling our software, being able to move things into production early and often. What we're doing now is we're able to automate a lot of our controls and make them part of the process.
Jennifer Bonine: Yeah, early.
Adam Auerbach: Yes, so I can show that hey, with our groups that are doing continuous delivery and maybe they're deploying multiple times a day, every day, which would be scary on the surface, but they are much more better managed because they have automated quality gates built in that say, "Hey, if you're not meeting our controls, we stop your pipeline automatically." There's no human interaction in that process.
Jennifer Bonine: Right. No, it automatically cuts it off, yeah.
Adam Auerbach: There's things that we can do to help document and show like, "Hey, this is a better process, it's less riskier," and that's part of getting the message out and sharing those stories because at the end of the day, we're able to do more for our customers-
Jennifer Bonine: Be more responsive.
Adam Auerbach: Exactly, but you're baking quality in for performance and security and managing that risk.
Jennifer Bonine: Have you seen just interesting that there is a lot of financial institutions, Capital One has taken the stance that we think it's a good idea to build in this CI process, making sure we're putting in those quality gates, building stuff in early, have you seen any of your peers that maybe are more resistant to that or a little bit more right traditionalistic in their view of risk?
Adam Auerbach: Yeah, I have, I don't want to drop any names but there definitely is. There are some European banks who've actually stopped. Who said, "You know what, waterfall is the right process for what we're trying to do." Part of this thing about getting to continuous delivery is it takes an investment.
Jennifer Bonine: It does, yep.
Adam Auerbach: When you talk about automating everything, you have people who aren't used to programming who aren't used to doing that level of automation, and so you're going to have to make an investment in them. You're going to have to probably slow down a little bit in order to get faster and more efficient. If you don't do it right, it might go bad and so there absolutely are banks and other companies who've said, "You know what, we're going to wait it out."
Jennifer Bonine: Yeah. It's interesting in the adoption lifecycle too, you look at change and there are early adopters and innovators, and then you have the resisters. There's definitely some focus in that resister camp and it's interesting because if you wait too long, you can lag so far behind-
Adam Auerbach: Yes, you know the competition is not other banks, right?
Jennifer Bonine: Right, yeah.
Adam Auerbach: It's the Googles, it's the Apples, it's PayPal. Everybody's getting into some type of credit card processing or banking and now with Bitcoin and other technologies like it's available to people, and if they're able to do it faster and react faster to market conditions, you're going to be on the outside. That's one of the things our CEO, he recognized. There's a ton of great examples, like Uber and with what they've—
Jennifer Bonine: The disrupters.
Adam Auerbach: Yeah.
Jennifer Bonine: Yeah, there's tons of disrupters out there and I've even talked to a couple organizations here at the conference that have said two things that are pillars for them in the next year, which denotes to me they get what's happening, is disrupters and disruption, and then also quality. Understanding I have to put out a good quality product, I'm going to invest in it because it takes an investment and I'm going to look at what the disrupters are because you need to do that, otherwise you can cease to exist.
Adam Auerbach: Yeah, so we do scan trips, like go look at other companies, we go to these conferences or we host conferences or other types of hack-a-thons an events as a way to get insight into those things and then figure out, hey, how do we bring that in internally with the stuff that we're doing? We do internal events like we do something called C-Con which is a software engineering conference. We do it once a year, it's got over 2,000 Capital One associates participating and showcasing their work, but that's how you can get aware of what's going on and then figure out how do I do local experiments to then expand it.
Jennifer Bonine: Expand it, and so good point for folks out there if you're not doing that, that you should be looking at those types of things in your organization, right? Not putting blinders on to what other people are doing, but creating an awareness of what's out there and then creating the opportunity to innovate inside the organization. It does take an investment. It sounds like you have leadership that's bought in, but how were you able to sell that investment, because I think there are probably a lot of people out there saying, “Oh, that’s great. You have a great CEO that gets it. You have leadership that gets it. My leadership doesn't get it." Any tips or techniques or tricks for people that are struggling with that buy in for some of this?
Adam Auerbach: A couple things. The first part is, try experiments within something that you can control. I was just talking to a gentleman outside and he runs a group in Canada and he was talking about how do I get leadership buy-in, and I was talking to him about you have enough control within your group to do something. Make an experiment, show them how it would work within the company ecosystem.
Jennifer Bonine: It's so small.
Adam Auerbach: Yeah, and then get leadership buy in to go and just socialize what you're doing and ask other people to do experiments as well. Then if you can keep getting in front of leadership and showing them the progress with the experiments, you can create this top-down, bottom-up approach.
Jennifer Bonine: You're hitting it from both angles.
Adam Auerbach: Then all of a sudden it just becomes a tidal wave and it's done. That has been a really great way for a lot of transformation to happen here. That's probably the biggest thing.
Jennifer Bonine: Yeah, it's just getting that capability to start those small concepts or experiments to get people to buy in to what's going on and see.
Adam Auerbach: People want to see it in their terms. I always talk about branching, one of the big things about getting continuous delivery is about having like a single trunk methodology, but when you talk to people in your company about doing that, they're like, "We can't do that."
Jennifer Bonine: "We can't." Yeah. All the obstacles for why.
Adam Auerbach: Yeah, but then when you actually go through someone's branching strategy and you say, "Okay, why are you creating a feature branch and a team branch? Why do you have a disintegration branch?" You start digging into it, you start to be able to show them, "Well, what if you did more unit testing? What if you have automated tests that ran when you did that first pole request? What if you brought performance in over here? What if you used the cloud and you had more test environments?" Then all of a sudden, you're able to show them in their terms, like what it means and why it's worth investment. Things like that.
We would do value stream analysis. Being able to say, "Hey, let's take in ten from your business and go through everything that you have to go through in order to get it in production, and then through that exercise, you can show the inefficiencies. In their terms, you can show them that, "Hey, if you were to make this investment in CI, in test automation, in the cloud, hey, your productivity is going to increase." You might take that initial investment, but a team of eight who's only doing a team of four worth of work can now double their output if you made this investment. There absolutely are tools like that to at least put it I their terms and then the light bulbs start going on.
Jennifer Bonine: Yeah, because they need to see it, right, and that's where I think a lot of folks struggle is getting to that value stream and showing where the value is and just laying that out because a lot of people get stuck at that, "How do I show or demonstrate the value of what I'm trying to ask you to buy in to?" Doing value-stream analysis, I love that term for it, for a way to look at it so you have a way to say, "Here's how we're going to help improve the organizational world."
Adam Auerbach: Yeah, because people, it's just a natural tendency to say, "Well, that's a generic idea which sounds great on the surface but we're different."
Jennifer Bonine: Yeah, everyone's different. Everyone's special. Yeah, I've heard that so often, everyone is special.
Adam Auerbach: Right. Yeah, we have different problems. The things here are different so that won't quite work here, and it's like, "Let me show you."
Jennifer Bonine: Yeah. Well you mentioned service virtualization, again, that's one that just getting to that piece and there are a lot of organizations to there who aren't using virtualized environments yet or aren't taking advantage of that. Any advice for people out there who are saying, "Gosh, maybe I should be doing that, because that is a real block for me today?"
Adam Auerbach: Well, I'll tell you, service virtualization is one of those things where I tell people about it and they just look at me and they go, "Okay," and then they walk away. It's one of those things where you have to get into the details—
Jennifer Bonine: To get it.
Adam Auerbach: Yeah.
Jennifer Bonine: I know.
Adam Auerbach: If you have problems where you're waiting for teams to deliver something in order for you to be able to test, if you have constraints around test data, if you have constraints around the non-prod environments, all those things can be solved with service virtualization. Being able to create smart stubs to isolate your components and eliminate those constraints are a huge enabler in order to get to continuous testing and eventually continuous delivery. I can't talk about service virtualization enough, but still, you have to talk about it, then you have to show them.
Jennifer Bonine: They have to see it.
Adam Auerbach: Yes. Again, back to the example, if I show you, "Hey, you have a team that's doing front end work," now they're waiting a sprint for the API team to get done and then when that gets done, you have 30 percent defects. Even though the team has moved onto something else, they get pulled back to do this work and then it doesn't go out here and it still doesn't work. If I had service virtualization, it would be done in the first sprint, they're able to move on, the API team would be done, and then it would be in production and you would have saved yourself time-
Jennifer Bonine: You're moving as a group as opposed to lagging.
Adam Auerbach: Yes.
Jennifer Bonine: Yeah, and we talked about that.
Adam Auerbach: Going through that exercise, now it's like, "Okay, this makes sense. We should be doing it."
Jennifer Bonine: We should be doing this. When you guys started with that, how did you do that and how difficult was it to sell that concept? It's a little bit less tangible to people around how to do that.
Adam Auerbach: One of my responsibilities at Capital One is I run the enterprise testing and release management group. Part of that group is performance testing. One of the issues that we had at Capital One is that we didn't have production scaled environments in non-prod performance.
Jennifer Bonine: Yeah, lots of people have that.
Adam Auerbach: We created a small team dedicated for service virtualization, and we said to that team, "Okay, you're going to enable everyone else, and it might mean creating the first suite of virtual assets." What we did was start with performance testing. We said, "Let's use performance testing as our guinea pig," and we created virtual assets. We enabled all the performance testers to be able to do it themselves, and we quickly built up this huge arsenal of virtual assets. Then once we started having wins in some of the groups with that, then we were able to then take that team and say, "Hey, besides performance, look at the other opportunities."
Jennifer Bonine: Where you can leverage this.
Adam Auerbach: Yes, but that performance was like—
Jennifer Bonine: Was the key.
Adam Auerbach: Yes.
Jennifer Bonine: They gateway for folks, so for those out there watching, maybe an idea for them to then start with that performance piece, because that is an issue, right?
Adam Auerbach: Yeah.
Jennifer Bonine: I think it's been interesting to be in testing for a number of years and see performance was really big ten years ago and people were like, "Oh, we need performance testing," because they had large applications they were worried about and they tested them, but they did it once and kind of went, "Oh, we know it's good." As you're getting into more complex technologies, mobile, people wanting to stream more information on their devices, you need to make sure that's going to work and they're going to be responsive, that people have really said, "Wow, we need to do this performance thing all the time."
Adam Auerbach: We talk about continuous testing, performance is part of that.
Jennifer Bonine: Absolutely, and that's what I don't think people have quite caught onto yet. They're getting that you need performance testing, but that's involved in there, and it's coupled really tightly, so it's very important that you have the performance component built in.
Adam Auerbach: Absolutely. I wrote an article for TechWell, maybe a month ago about the performance testing bottle neck and the fact that many people, even though they have CI and they have automated testing in their pipeline, performance testing was still bottleneck. There are ways using open source tools, integrating early smoke tests is part of your pipeline, also you can leverage application monitoring and you can be doing that in non-prod. There are things you can do to get insights into performance early and often and make it standard for everybody without being a bottle neck that slows people down and people eventually opt out of it because it costs some time.
Jennifer Bonine: Yeah. Again, something if you're not looking at in your organizations is building in that performance testing into that CI process as well. We're not just talking about traditional testing, we're also talking about performance testing and that that's a key component. This conference versus other conferences, I know you've been at a lot of them, and threads or things you're hearing that you weren't hearing before? Has the needle moved since last time I talked to you?
Adam Auerbach: You know that's really interesting that you say that. My experience this year is totally different than previous years. Number one, I always, I shouldn't say this, but I really prefer West Coast conferences. I usually find that they're a little bit more progressive, so I take more away from them than some of the East Coast conferences, not that they're not bad.
Jennifer Bonine: It's funny you say that too, because I always say this is my favorite one. I didn't put together why, but that's why. It just has a different vibe with the progressiveness.
Adam Auerbach: Last year, I think people were on the edge of, "I hear something's happening in the industry, I'm not exactly sure what it is. Let me hear it," and then when they're hearing it, they're like, "Oh my God. I have to learn how to code. I have to learn these technologies." Last year, I saw a lot more fear. I had a session last year, I talked for an hour and then I spent an hour and a half answering questions. I had people writing up their frameworks. I had people who were just scared. This year, it's much different. People are nodding their heads, they're getting it. We had the opening keynote today from Netflix which was awesome. People are embracing the direction we're going in and it's more about, "Okay, this is some of the stuff that I'm trying within my company. What have you seen work? What are the pitfalls? How can I do this better or faster?" Where last year it was like, they were still kind of dragging their feet.
Jennifer Bonine: Right. It was like that fear of what's coming. Now they know it's here and they're embracing more and saying, "Okay, let me get information. Let me get on board."
Adam Auerbach: I think it's changing and my job's not going away. It's just evolving.
Jennifer Bonine: Right, it evolves, yep.
Adam Auerbach: Last year it was, "I'm hearing this and I'm getting scared."
Jennifer Bonine: Terrified.
Adam Auerbach: People are moving past that which is really great. The turnout has been really great, the talks have been really great. There are lots of wonderful conversations, it's really been one of my favorites.
Jennifer Bonine: That's awesome. Well, the time always goes so fast, but thanks for being with us.
Adam Auerbach: Yeah, absolutely.
Jennifer Bonine: If people want to get in contact with you, what's the best way for them to ask some more questions, continue the discussion?
Adam Auerbach: You can tweet me, @bugman31 on Twitter.
Jennifer Bonine: @bugman31 on Twitter. Thanks so much for being here.
Adam Auerbach: Thanks, Jennifer.
Adam Auerbach is the senior director of technology for advanced testing and release services for Capital One Financial Corporation, a diversified bank with 65-million customer accounts worldwide. Adam is responsible for Capital One’s enterprise performance and automated testing departments as well as enterprise release management and test data management. Since joining Capital One, he has provided leadership for the agile transformation of their quality assurance group, and led the enterprise adoption of DevOps and ATDD. Previously Adam was with Chase and other financial and insurance companies in leadership positions focused on quality and agile practices.
