Implementing DevOps in Large, Complex Organizations: An Interview with Mike Baukes

Mike Baukes will be presenting a presentation titled "Lessons from the Front Lines: Implementing DevOps in Large Complex Organizations" at the Agile Development Conference and Better Software Conference West 2014, which will take place June 1–6, 2014.

About "Lessons from the Front Lines: Implementing DevOps in Large Complex Organizations":
Financial services are not about high fives, flashy suits, and Maseratis. Behind the scenes, the technology that powers these companies walks a delicate line, balancing regulatory risk and the need for rapid technology response to continually changing market conditions. DevOps is the perfect fit, a natural for these organizations. Getting it right, however, is quite another story. Mike Baukes describes two recent experiences of wide-scale organizational change in establishing DevOps capabilities in a trading firm and a commercial banking operation. Mike shares his experiences with straightforward accounts of wrestling large-scale technical debt, enterprise cultural change, and a constant stream of pipeline changes going to production. Whatever your level of understanding of DevOps is, you will leave with penetrating insights and pointed anecdotes of what success and failure look like—from someone who has been there as a consultant and an employee.

Jonathan Vanian: Hello. Today, I'm with Mike Baukes. Mike Baukes is going to be speaking in the upcoming Agile Development Conference West. Mike, thank you for joining us today.

Mike Baukes: Hey, thanks. How's it going?

Jonathan Vanian: It's going well. I thought we should start off by having you explain a little bit about your career, thus far, for readers and listeners.

Mike Baukes: My background is, I'm very much an enterprise guy. Mostly financial services, predominantly large-scale enterprise IT. Everything from architecture to operations to app development, I've seen a lot of it and I've had senior leadership roles, had operational roles, and a lot of those types of components, which has been great.

Jonathan Vanian: Right.

Mike Baukes: For me, it was kind of natural after spending close to 12 or 15 years in there, my co-founder and I, we came across this continual problem and that was that it's really, really difficult when you're in a large-scale organization to get visibility across how everything's operating, and the composition of how services that our business relied on are actually stitched together.

Jonathan Vanian: Right.

Mike Baukes: After hitting our heads against this problem so often, and trying to automate solutions to build these services, it became pretty evident that if we were to stop and look at the composition of these companies that the best way to get everyone engaged and understanding these services is to use an emerging philosophy, which is called DevOps. DevOps is really about getting cohesive collaboration inside an organization in order to deliver software for the benefit of the business.

Jonathan Vanian: Are you seeing this in primarily financial service industry or just across the board, any type of large organization.

Mike Baukes: It's not only large organizations. It is across the board, but it's small organizations, it's larger organizations. I think everyone has realized and come to the conclusion that historically developers and operations, they are at odds, and there's no real mechanisms in place for them to establish communication and establish shared learnings between those different groups. I think that while I took them along and there are some pretty interesting processes around—formal adherence to processes for the benefit of operations and to some degree for the system designs principles, agile came along and started talking about iterative development cycles, trying to reduce the waterfall view.

I think we're now seeing, as a result of agile occurring and has a result of the cloud, and the data center perimeter extending out to the Internet, what we're beginning to see is this need to deliver solutions faster and in a more agile way. Naturally, now the operational guys that have historically been very much looking at protecting the environment, and safeguarding it to some degree, are now beginning to learn a lot of agile techniques to effectively manage their infrastructure faster, provide consistency, and really provide clarity and visibility across all those types of environments. It's a pretty exciting time for everyone, I think.

Jonathan Vanian: Yeah, I'm sure. I want to ask you a real basic question on what is DevOps? I like asking that because people have so many definitions. What is your definition of DevOps?

Mike Baukes: It's just collaboration with a shared goal. Really about being able to collectively learn as an organization. I think foundationally for me if you're going to distill anything in a really simple format, it's really about being able to fail, but at the same time being able to learn about failure and being able to move very fast and very iteratively.

Jonathan Vanian: Right, do you see it as a means of connecting the old-school waterfall development with agile now that people have to be connected with each other.

Mike Baukes: I think, in general, waterfall has its place. it's just like outsourcing has its place. I think there's arguments for it and there's arguments against it. Just as you can say agile has succeeded a great deal in being able to get people collectively to deliver software faster, I think there's also a certain level of comfort and robustness that sometimes, under certain circumstances, very regularly, waterfall can also benefit. I think to some degree we at this weird intersection at the moment where I think the genuine knowledge around how IT systems are constructed, developed, and supported are coming into this weird confluence of organizations' ability to be able to execute on this stuff to a world-class standards increasing and at the same time the quality of software in general, over-archingly, is becoming a little bit more robust.

At the same time, accessibility to this knowledge, be it code or knowledge in stack exchange, all of this stuff is forming now so that you can get to market faster, you can stay safe, and you can go very fast doing it. A lot of organizations are trying to work out how to do that, trying to understand how they can move forward on that journey, once it works.

Jonathan Vanian: Sure. It's a subject that you could just talk endlessly about. Let's just talk a little bit about why is it hard for organizations to get DevOps right.

Mike Baukes: I think more often than not, it's there's a loss of knowledge typically that exists in any organization. People say that there's lots of silos in an organization, segregation of duties, and specialization. In particular is something that naturally happens is businesses get bigger and bigger. You can't expect that everyone in an organization will be a generalist.

Jonathan Vanian: Right.

Mike Baukes: I couldn't imagine. It just doesn't work that way. I think what is need though is very much a view of visibility across the organization. Whereas before, because a lot of systems, a lot of applications were within a data center typically and there was traditional cycles to get that software developed for the customers, I think now with increasing security threats, like Heartbleed for example, there is fewer multiple services or service oriented web architecture to some degree, all of this stuff now means that organizations aren't just looking at their own data center. They're looking at a myriad of different services that they can plug into their business for speed and for enhanced customer satisfaction.

Being able to tap those, being able to do those consistently, and then at the same time track the state of them I think is the biggest challenge that people are faced with at the moment. If you don't know what you've got then it's really hard to move any direction really.

Jonathan Vanian: That's the modern world now is that you have so many different tools, so many different things plugging into each other. You didn't see that in the past generally, you maybe had to use one tool, but now you can use two tools and each tool has several other components. Is that why you feel the need of DevOps? It's like a site map essentially of all your tools.

Mike Baukes: Yeah, the taxonomy of your business.

Jonathan Vanian: Right.

Mike Baukes: It's just so easy now to be able to plug different systems in for different particular portions of business processes. How do you even track that? How do make sure that a change isn't going to create a problem down the line for a customer? All of these things have resulted in organizations now needing better visibility, better control mechanisms, and at the same time the ability to be able to share knowledge collectively about how the systems are compromised across their IT staff.

Jonathan Vanian: This is actual configuration management, right?

Mike Baukes: I think there's a configuration management aspect, I think there's a configuration monitoring aspect, I think there's a knowledge management aspect, I think there's a process aspect, and I think there's plugs that you've got into all the aspects of an organization. From me, I always look at the value chain of an IT organization as how much waste to they create, how much duplication do they, be it in a a simple form, be it in a data field, be it in the system, and then how well often, how well aware, and how well versed is everyone in the composite state of the applicational service and how often do they try and improve the reduction of duplication throughout each of those life cycle steps to production.

I think that quite often a lot of organizations lose that visibility and they don't really strive to remove the duplication in an effective way and I think that's ultimately where a lot of the problem historically have popped up, but now are more so important because you can't spawn 1,000,000 servers for one app, you could have thousands of apps and thousands of servers and they could be here, they could be in Romania, they could be in New York, they could be anywhere.

Jonathan Vanian: It's like waste management, except it's spread across the world; you're dumping your garbage out in Romania.

Mike Baukes: It's great. The utility effect, it's great, but it's just hard to control. I think quite often companies struggle with that.

Jonathan Vanian: Let's talk about what a failure looks like when an organization tries to implement DevOps.

Mike Baukes: I've been part of a couple of failures in the past, so I'm pretty well versed now in things that we did wrong a couple of times now. I've also been part of a really good success story, which is a really good thing too. I've seen both sides, not only as a consultant, as an employee, and now as a vendor too, which is great. I think too many companies they hear DevOps and they jump to automation and then jump to tooling. I think there's a lot of people out there talking about the effect of culture and the need to prepare a culture in order to be able to facilitate faster learning.

I think too often people are just naturally hearing about DevOps and they're like I'll just look at the call list and see what's available, and then there's a myriad of options there. I think the number one thing that people have to realize is that there's got to be a cultural ambition to actually work together and deliver fast. What I've seen in the past and jumped to conclusions about in the past, and instituted to some degree, is the need to automate as fast as possible. For us, whilst it met the needs of the project at the time, we didn't achieve any of the organizational roles or cultural goals and as a result the technology got left by the wayside.

This is a common thing that happens. You hear about it all the time. Particularly, when you talk about DevOps a lot of people talk about there's pockets of automation here, there's pockets of automation there, we've got a bash group and part of that is built on Jenkins. The unfortunate thing is that people quite often, particularly when you're in a large-scale enterprise with legacy systems and legacy apps, the need to automation feels like the right move, but quite often it can actually really, really hamper your efforts to introduce a DevOps style culture to your organization.

Jonathan Vanian: You've got to really set up the automation right before you flip the switch.

Mike Baukes: I think you've actually got to know what you've got. More often than not, you'll find most people don't know how to get something to production. They've never really spoken to the ops guys. They don't know why the security guys are there, trying to ask them well why are these files open, have you broken this out into a 3-tier structure, how does the data regress, all these questions. It's just how much does an individual know throughout their chain. More often than not, particularly from an engineering perspective, the natural inclination is to close in, just focus on 'I'll just keep on building' versus actually getting out there and talking to people. I think often that's the first inclination is that we can control is using code instead and then that leads to the automation problem, and then unfortunately they lose sight of what the business actually does.

Jonathan Vanian: All right. That's a good example of failure. What does a success look like?

Mike Baukes: I can say this is for us what we saw as success and that was an organization right about 2010 there was the best program happening within this large bank, right?

Jonathan Vanian: Correct.

Mike Baukes: It wasn't DevOps at the time, but there was an application-centric group , there was a whole need to remove assets from this bank, create new assets, and then at the same time bring together a brand new team. As a result of that, there was multiple entities that had to get mashed together so there was great application guys, there was great infrastructure guys, and what happened as a result of that is that there was enough people to not only deliver a great repeatable solution, but at the same time everyone was focused on actually delivering this project and delivering the solution from the CEO all the way down to the ... everyone was aware how this was operating.

For us, the philosophy of DevOps is we've got this outrageous project deadline that we've got to meet, we've got to create all these new capabilities for a business, we've got these existing capabilities, let's understand those first, let's map out our to-be state, and then let's execute iteratively on that using agile to some degree and a lot of the operational infrastructure test-driven development processes.

For us, we got that up and running. Collectively, when you looked across the organization everyone was really aware and everyone was really open and sharing those lanes we had regular continuous improvement sessions, there were scrums, you could learn a lot. There was visibility across the organization and what state we were at. Anyone could ask a question at any time and query any decision that had been made.

Jonathan Vanian: Management was totally okay with that?

Mike Baukes: Totally open, really irregular, particularly for a banking organization. This was a big project. The great thing was is that what was a two-year deadline was delivered in 18 months, I think.

Jonathan Vanian: Wow, that's pretty amazing.

Mike Baukes: This was a significant project budget too.

Jonathan Vanian: Was this a big bank?

Mike Baukes: I can't actually say. It's like the number one or number two bank in the UK, and a couple of international divisions were involved in this. It was huge.

Jonathan Vanian: This was not just a small organization, small shop.

Mike Baukes: Yeah, I think it affected directly about 1500 people in IT was a massive push to get this thing done. That was a small project team to some degree.

Jonathan Vanian: All right, let's talk a little bit about tools. Readers love talking about tools. What are some good tools out there people should be aware of?

Mike Baukes: Definitely on the open-source side, Puppet, Ansible. Ansible is getting increased traction. I think Puppet is a mainstay on the infrastructure side. Ansible is becoming more and more available too. Our product, GuardRail, of course, which is awesome.

Jonathan Vanian: Get your pitch out now.

Mike Baukes: Yeah, I won't do that to everyone. There's a whole hub. There's so many great platforms now for collaboration, a lot of the Atlassian tools, like Jira and Confluence, they're really helpful. One that I really, really like these days is a product called Slack. It's like a chat room for ...

Jonathan Vanian: ... How do you spell that? Was it Slack?

Mike Baukes: S-L-A-C-K. It's kind of like a ChatOps product, if you've heard of it. Basically, you get all of your events injected into this long stream that you can basically go through, you can add Twitter feeds, you can add Help Scout, you can add continuous integration tools. All of this stuff becomes this just moving canvas of the business at any point in time, which is really great.

Jonathan Vanian: That's really cool.

Mike Baukes: There's so much stuff. We're focused on a lot of things. There are some great products out there. If you get a chance to ever look at a monitoring tool called Geckoboard or Graphdat too, also really great tools these days that are being made that are SaaS subscription models that are fantastic.

Jonathan Vanian: Yeah, amazing time for the tools.

Mike Baukes: I blurted them all out to everyone here.

Jonathan Vanian: No, that's totally fine. How about any trends in enterprise technology? What are you seeing that people should be aware of?

Mike Baukes: Definitely the containerization movement is taking off, right. Docker itself is becoming this huge force, so people are beginning to realize that containerization and application-centric technology like that, particularly for deployment ...

Jonathan Vanian: ... When you talk about containerization just briefly explain that for people.

Mike Baukes: Oh, sorry. Being able to take an application and instantiate it in its own space where it doesn't really have any underlying effect to the operating system. Effectively, it means that you can wrap it around this portable unit, like a shipping container, and move it from host to host without any impact on the operating system. Kind of like what a virtual machine did to the hypervisor, but way smaller, way more compact, and way more service oriented.

Jonathan Vanian: Very cool.

Mike Baukes: We're seeing a lot of that and we personally use that technology a lot, and we love it. It's moving in a really great direction. I'm really looking forward to the software find, the networking components where you can see more visibility. Not really practical application, but you can see where it's heading. We just need a few more of these other parts to come together.

Jonathan Vanian: You're seeing the seeds planted right now?

Mike Baukes: Absolutely. YANG and NETCONF and all those components now are beginning to form the stream where you can actually see, and particularly with Docker coming into the play too, you can see how this whole thing's going to fit. I think we'll actually be able to do true ... this sounds really weird saying this, but I think we'll be able to do, the reality in the 1990s of service-oriented architecture I think will actually be a reality in the next few years because the ability to be able to pick lightweight reusable components up, mash them together, create services and loosely-coupled services is clearly nearly there. I think we're just at that point now where it's about to tip over and it's going to be so cool. We're just at that point now and it's really great.

Jonathan Vanian: Very great. Winding down a bit, anything you want to leave people who are going to be attending your session? Anything you want to give out to people?

Mike Baukes: Just come with a really open mind. You can hurl abuse at any point in time, I'm quite all right with a heckle, it's always good. There's a couple of books that if you've had a chance to read, number one is Jez Humble's "Continuous Delivery" book, and number two is Kevin Behr and Gene Kim's book "The Phoenix Project". Really great practices or just explanations on DevOps, not so much Jez's book, but Kevin and Gene's book definitely.

Yeah, if you need any help or assistance or anything, or just wanted to have a chat about it, I would love to talk to anyone that wants to talk about it afterwards.

Jonathan Vanian: Great. All right, Mike. Thank you so much for taking time to talk.

Mike Baukes: Any time, thanks a lot.

Mike Baukes is the cofounder and CEO of ScriptRock, a Bay Area startup that makes QA software easier for DevOps teams. ScriptRock's flagship product, GuardRail, is a cloud-based change, configuration, and release monitoring solution that helps enterprises deliver ideas to market faster and safer. Prior to startup life, Mike held senior technology leadership roles in strategy, systems architecture, engineering, and operations dealing with billions of transactions for international banking and finance organizations including Lloyds Banking Group, The Commonwealth Bank of Australia, and  E*TRADE. He resides in the Bay Area with his beautiful wife and two children. You can find him on twitter as @mikebaukes or @devops