In this interview, Gene Gotimer, the senior architect at Coveros, explains how you can test better, deploy features into production faster with continuous delivery, and arrange your tests so each round provides just enough testing to give you confidence.
Jennifer Bonine: Hi, and welcome back to the interviews at STARWEST 2016. I have Gene with me today. Gene, thanks for joining me.
Gene Gotimer: Thanks for having me.
Jennifer Bonine: We actually had one of Gene's colleagues here earlier, Tom, and we were talking about security. I was talking to the folks out there about what's interesting about security. We always knew security was an issue before, just with ... I used to like to give the castle analogy. We had our castle and we would build the moat around it. So, we were like, "Okay, if we can just keep people out of the castle, we're secure." You lock it down that way. What we see now with embedded devices, everything being connected, all of the IoT stuff, people are now inside the castle. The moat's great, but they're already in.
Gene Gotimer: Sure, in fact—
Jennifer Bonine: Now what do we do?
Gene Gotimer: The biggest threat, usually, is an insider, whether through malicious or even not malicious, just careless. Making a mistake. We have to start being a lot more protective of our stuff inside and not relying on perimeter defense.
Jennifer Bonine: Right, because the perimeter isn't the problem now. Those people are embedded. They're inside our organizations. We've see it with a lot of things where people thought, "Well, I've got all this covered and I'm doing scans and all that." Who's watching the stuff inside, too, and what's happening there?
Gene Gotimer: Right, and like I said, it could be malicious people. Someone who's disgruntled or just sees a chance to make a quick buck by selling inside data. It can just be the person that doesn't realize what they're supposed to be doing and downloads a database and leaves it in their car, which gets broken into and it walks off, right?
Jennifer Bonine: Right. Yeah, and that happens.
Gene Gotimer: It does.
Jennifer Bonine: More and more, we're seeing that happen and we think about physical security of securing homes and other places, but it's this virtual data that we've got now that we have to take care of. It's all this virtual information that we have to lock down.
Gene Gotimer: Yeah, and people are increasingly understanding that the value in the software is often the data that they collect with it, not the software itself. Trying to start from the beginning and put the controls in place that you need to make sure that you're keeping the data safe, and that even someone just through carelessness doesn't release it or doesn't make it available.
The other thing is that so much of our modern software is based on frameworks and third party libraries. Most of the software we're running, even in our own app that we've written, isn't code that we wrote. It's stuff we're borrowing from other people. If they've made a mistake or if you're using is wrong, who knows what you're opening up?
Jennifer Bonine: Look at Target. Target had that issue. They said, "Well, it's not our software. It was a vendor that provided something to us that caused the breach, which allowed all the access to the data."
Gene Gotimer: The HVAC guy who just didn't realize that leaving it on a sticky note next to his computer wasn't the right way to control security.
Jennifer Bonine: That's the whole thing. We have all these other access points that we have to be aware of and conscious of. As testers, we're all about helping manage and understand risk. Having a basic understanding of where some of those risk points can be is a good thing, I think, and seeing that.
Gene Gotimer: Right. I think the best way we see to control that is to start getting security involved right at the beginning. Just like testing is often ignored until the end, and then when you run out of development time, you just get rid of the testing. Security is even a special case of that. When the testers run out of time, they get rid of security testing.
Jennifer Bonine: Then they're like, "Oh, we're not going to do that security piece."
Gene Gotimer: We just won't do that. Don't worry, it'll be fine.
Jennifer Bonine: It'll be fine.
Gene Gotimer: Most people don't really get serious about the security processes until they've been breached. Until they've had a mistake. They learn it the hard way.
Jennifer Bonine: That's when you get the call. Is all of the sudden, we've had something happen. We're afraid. Now we'd better look at it. I think it's all about all of us having an awareness and getting more proactive in how we handle it.
Gene Gotimer: We talk about trying to get it involved, get some security testing early in your process, just when you start out. Even if it's not much, it's better than nothing. You get better and better, and you keep doing your retrospectives and you keep reevaluating your process, and you keep looking at where your shortages are and where you need more effort. Eventually, you get better and better at it. You develop a, what you might have set out to build in the first place, but you've done it over time, incrementally. Little bits at a time and build it in throughout your security cycle.
Because the other thing is, if security happens only at the end, if that's when you're doing your testing and you find a problem. It's somebody saying, "Hey, I found a security problem," versus the product owner saying, "Hey, I promised I'd ship it." Who's going to win?
Jennifer Bonine: Right, exactly. Have you guys seen at all, in some of what you're doing. It used to be, just like with testing as a whole, there was a lot of manual processes in the testing that now, with DevOps, and with some of the tools we have available and accessible to us, that there's also the option to get it embedded in part of your development cycle and embed some of that automatic checking. Are you guys proponents of that?
Gene Gotimer: Yes, absolutely.
Jennifer Bonine: Having that automatic checking that's giving you some information on every build, on every time you're checking code in.
Gene Gotimer: Right. We love continuous integration. The rapid feedback cycle, early in the development cycle, you get feedback right away. Any type of feedback we get there is great. Security is no exception to that. Static analysis tools are really good at finding certain types of security problems, SQL injection. Race conditions, which are hard to find dynamically through regular testing, but are actually easy to find through static analysis. Resource leakage, stuff like that. That's all easy to put in and it's just a matter of running a static analysis tool.
Then, after you start doing deploys, especially if you do automated deploys. I've run my unit tests. Everything passed. I've run some static analysis. Everything passed. I do a deploy and I'm about to run my functional tests, we say, "Why not throw that through a proxy?" You can do something like OWASP app and just watch the traffic flowing though. It's not even an extra test. You are just watching the functional test going to your web app looking for an cross-side scripting, cross-side request forgeries, stuff like that.
Jennifer Bonine: Have you guys done some work, too, with companies besides building some of that tooling into their CI process? The static analysis, doing some of the checking. Getting that feedback. It's that whole concept of shift left with testing but shift left with security.
Gene Gotimer: Yes, absolutely.
Jennifer Bonine: Get it in as soon as you can. Also, besides shifting left and building some of those tools in, I think what we're seeing, in general, in the market space is just a widening of awareness and making yourself more aware of what tools are out there and what tools are available to you that you may not. Even if you're not the one who implements them or has to put them in, as a tester, having some awareness of those tools is important in what's coming out. Because there's always new things coming out that are available for people to use. Have you guys seen anything that's impressive to you or that you've said, "Wow, this is something you should really take a look at"?
Gene Gotimer: Just looking at all the open source tools that are out there is the easiest way to go. Buying a tool takes time. You have to procure it. You get it approved, whatever, a lot of time.
Jennifer Bonine: Licensing, cost, and that.
Gene Gotimer: A lot of times, getting the open source is just a quicker way to start. Doing something is better than nothing. Really, the only good thing that's come out of a lot of these high profile breaches and mistakes has been that people get more aware of it.
Jennifer Bonine: Yeah, raising the awareness.
Gene Gotimer: Next time a breach hits, if you’re interested in security, that might be the perfect time to go talk to your product owners and say, "Hey, you don't want that to happen to us. Let's start looking at some of these open source tools." Like I said, we use a lot of open source, just because it's quick to get in. OWASP app, one of our favorites. OWASP Dependency Check can be run during static analysis. We love SonarQube for quality dashboard, but that helps run a bunch of static analysis tools that can fin some of these easy to find security problems.
Jennifer Bonine: We've even found, I know there's a lot of people out there running on automation and we've seen it where we'll work with different organizations that, when you try and run automation, they've now got some sophistication on their websites even, that will identify that automation as a bot. Even those types of things is good for you to start putting into your websites, if you don't have them, for identifying people coming in and trying to, either maliciously or nonmaliciously, going after the website and injecting stuff in it.
Gene Gotimer: Yeah, whether they're doing a brute force attack or just trying to map out your site to see where there might be places to inject.
Jennifer Bonine: Having that awareness of those tools and tips and techniques that are out there. You gave a session at the conference that just finished recently.
Gene Gotimer: Right. I talked about building testing into your continuous delivery pipeline. How to do it better, faster, and cheaper. Really talked about just all the different opportunities that, if you just become aware of your pipeline. What are you doing? When I say pipeline, most of the time, I'd like to be talking about DevOps or continuous delivery pipeline. Even if it's completely manual. Even if it's just traditional. However you get your software from the developer out to production or into the hands of the user, that's your pipeline. That's how you release it.
The more aware of all the things you have to do you are, the better you can balance. I want to get fast feedback at the beginning. I was to get no surprises at the end. The more types of testing you can throw in along the way. I keep telling people, "Do just enough of each type of testing early so that further types of testing are justified." If you can do a little bit of security testing up front, then the next round, you actually have to spend a little bit of effort is worthwhile.
At the end, when you finally turn it over to the security team and they have to do the sign-off on it, you're not going to get surprised. You already know that everything's going to come back clean. At least, you're highly confident that that's the case.
Jennifer Bonine: Exactly. In terms of DevOps, we've heard about DevOps for a couple of years and it's been at the conferences. There's a track on it. What is the maturity level are you seeing around DevOps as a practice in organizations and building that in now? Are you seeing more companies leaning towards that? Are a lot of people still just playing around with it and investigating?
Gene Gotimer: I see both of those. The big thing that I think has changed over the past year or so is that people are no longer saying, "If we ever get to the point where we're doing DevOps." They're at least now admitting that—
Jennifer Bonine: We're going there.
Gene Gotimer: It looks DevOps is the right way. It's the same thing we saw with agile a couple years back.
Jennifer Bonine: With agile, right? It was, "Maybe we'll be doing that. We're checking it out." Now it's, "Okay, we've got it. We're going there. Now, how do we get there?"
Gene Gotimer: Yeah, last year, a Washington Post article came out on some efforts that were going underway at the Department of Homeland Security to bring in DevOps, DevOps pipelines. One of the things that really stuck out to me, we happened to be involved in the project they were talking about. I read the article and it mentioned that the former contractors in there had been a controversial method of development called Waterfall. The fact that they were identifying waterfall as the controversial one-
Jennifer Bonine: As a controversial method of development.
Gene Gotimer: Versus DevOps and agile. That's a big change.
Jennifer Bonine: Yeah, that's huge.
Gene Gotimer: We're getting to the point where, nowadays, people are just assuming agile's the way to go. DevOps is just a logical extension to that.
Jennifer Bonine: Yeah, that's the logical extension. Oh, I love it. That's a great way. Our time goes so quick. We need to wrap up. If people have interest in some of the concepts, the topics we talked about. If they want to get a hold of you, what's the best way to reach you?
Gene Gotimer: Hit me on Twitter, @CoverosGene, or send an email to [email protected] and we'll be able to answer your questions.
Jennifer Bonine: Thanks, Gene.
Gene Gotimer: Thank you very much, Jennifer. Appreciate it.
Gene Gotimer is a senior architect at Coveros, Inc., a software company that uses agile methods to accelerate the delivery of secure, reliable software. As a consultant, Gene works with his customers build software better, faster, and more securely by introducing agile development and DevOps practices. He has many years of experience in web-based enterprise application design, and extensive experience establishing and using development ecosystems such as continuous integration, continuous delivery, DevOps, secure software development, source code control, build management, release management, issue tracking, project planning and tracking, and a variety of software assurance tools and supporting processes. Gene feels strongly the repeatability, quality, and security are all strongly intertwined; each of them is dependent on the other two, which just makes DevOps that much more crucial to software development.
