Top Reasons Why Agile Fails in Large Enterprises: An Interview with Mike Cottmeyer

Josiah Renaudin: Today, I'm joined by Mike Cottmeyer, president and CEO of LeadingAgile. Mike, thank you very much for joining us.

Mike Cottmeyer: You're very welcome. Thanks for having me.

Josiah Renaudin: All right, well, first, could you tell us just a bit about your experience in the industry?

Mike Cottmeyer: Yeah, sure. I started LeadingAgile about four years ago. Prior to that, I was working for a company a lot of people have heard of called VersionOne. I was a trainer and consultant for those guys. Then prior to that, I was in a company called CheckFree, building large-scale online banking and bill payment software, and just through the years of working with lots of large companies, I developed a passion for figuring out how to do agile at scale.

Really even more so than that, really trying to start to build practices for leading through the process of transformation and helping companies to understand what it actually takes to get from point A to B in this journey. I've been around for a while, done a lot of consulting. A lot of big companies, a lot of small companies, a lot of complex problems out there.

Josiah Renaudin: Now we're all trying to move toward more agile practices, but your upcoming talk, "Why Agile Fails in Large Enterprises and What to Do About It," argues that plenty of big businesses are failing to correctly implement this new methodology. What makes larger organizations more subject to failure in this department?

Mike Cottmeyer: Yeah. A big part of what we're seeing out there is that people are either trying to do one of two things. They're trying to adopt what we call small-team scrum in large enterprises, and that basically means maybe they're doing a small project within a larger context, or they're trying to transform large programs or large parts of the organization to being agile.

One of the things that we see is that almost everything that you read or all the guidance that we give within an organization is all focused on the end state, and it's not really focused on the transformation patterns. What makes large organizations unique is that they tend to have a lot of what we just simply call the tendencies of tight coupling between the different entities within the organization.

When you try to form either a small scrum team or even a large-based program team, what ends up happening is that those teams inevitably end up being connected often in unpredictable ways to the rest of the enterprise. It's those connections between the different pieces of the enterprise that actually cause the problem.

The talk that we're going to do is going to go into a lot of those different kinds of dependencies where different parts of organizations are more tightly coupled, and the kinds of coupling that you see, whether it's the organizational structures or technology or business processes and all that kind of stuff. Really what the talk's about is how to progressively decouple those over time as you go from point A to point B in your agile journey.

Josiah Renaudin: Now, who's often blamed first when a large-scale agile transformation actually does fail?

Mike Cottmeyer: Well, it's not really a blame thing. I really do believe that our industry is in a state of transition right now. If you look back ten, fifteen years ago, the people that were coming to agile conferences and the people that were leading agile transformations were largely team members that had maybe been exposed to the Agile Manifesto or common scrum training, something like that. I would say, again, ten, fifteen years ago, maybe to as recently as a couple years ago, what you were seeing was primarily grassroots agile transformation efforts.

What I think we're beginning to see a trend into now is less grassroots or maybe it starts grassroots but actually become something that's important to a senior vice president or a CIO-level person within the organization. What I think we're learning as an industry is that patterns that are coming into play in large-scale transformations really require a holistic system thinking view to changing the organization. If we're going to say why things failed maybe more so than who to blame, I think grassroots transformations tend to fail because they can't actually change the things, they don't mandate to change the things in the organization that are around the team and cause them to fail.

A lot of times, even if the executives are involved, they don't necessarily understand the breadth of the kinds of organizational changes that agile is implying for them. Again, we just spent a ton of time talking to people before we engage with anybody in terms of making sure that everybody up and down the leadership chain of command is informed, understand how it's going to impact them, making sure that they understand the implications of these dependencies, make sure that they understand the path that it's going to take to move through the transformation, how those metrics are going to change over time, just lots of stuff.

Again, it's not really blame. It's just really just holistic understanding and making sure that folks really know, again, what it is that they're trying to ... what they're going for when they go down this path of transformation.

Josiah Renaudin: Yeah, you talked about avoiding blame and more looking at the root of the problem. What are the three most common challenges associated with an agile transformation?

Mike Cottmeyer: Yeah, the three most common things for us come down to the ability to form complete cross-functional agile teams, the ability to get those teams really clear backlog, and the ability for the team to produce a working tested increment of software. If you look at every issue that we deal with at scale, it is a larger manifestation of one of those three problems.

At scale in larger organizations, it’s incredibly difficult to find a pattern that allows teams to be able to come together and stay together and be held accountable and establish stable velocity and all that kind of stuff. We look for places to align business process and technology and teams into these units that can ultimately become agile teams and become more loosely coupled from the rest of the organization.

Doing that at scale is a challenge. Creating backlog, it can be in a small environment as simple as naming somebody to be a product owner and letting them write user stories, but in large complicated environments or complex environments where you might have multiple product owners, multiple business stakeholders, the need to coordinate across different kinds of boundaries at least initially in the transformation could be problematic. A lot of time, teams become starved for backlog, especially at scale. Then the ability to produce that working tested increment. What does it look like is the output of multiple teams has to be rolled up together and integration tested and performance tested and scalability tested, all that kind of stuff.

The three problems are backlogs, teams, and working tested increments. Pretty straight forward in a small team environment, but as you scale how many constructs to be able to deal with that in large organizations is basically what we're trying to solve here.

Josiah Renaudin: Now not all organizations were really built to be agile. It takes time, money, and a behavioral paradigm shift to really make that kind of change. Even so, would you still argue that it's always worthwhile to adopt an agile framework in today's landscape?

Mike Cottmeyer: Well, one of the things that I'm going to talk about in my talk might simply be described as how agile do you want to be. I don't think that in our industry that waterfall or at least classically described waterfall, maybe even classically implemented waterfall, is ever a good idea. I think we're well beyond the place where we want to do eighteen month-funding increments and then wake up in two-and-a-half years and see that what we decided to build wasn't really the right thing for the market. I think smaller batches, faster delivery to market, I think that kind of stuff is here to stay. I think iterative and incremental delivery is here to stay. I think to some degree team-based delivery is here to stay.

I don't think it's every a good idea not to do those things, but when you look at the continuum of practices that could be considered agile, you have all the way from things like heavily governed, almost commanding control iterative and incremental delivery all the way up to highly empowered, self-organized, goal-oriented, almost like lean startup kind of agile. I think that there's a lot of environments where it makes sense to have a more heavily governed, iterative and incremental team-based delivery model. I wouldn't necessarily call that agile. It's almost like a proto-agile. Then there's a continuum between that all the way to the incredibly loose, hyper productive, goal-oriented lean startup kind of agile.

What a big part of the challenge is figuring out where your organization needs to fall on that continuum. That's the big kicker because, again, eighteen-to-twenty-four-month waterfall I think is gone. I don't think there's ever a good case for doing that. How agile you need to be past that is something that every organization needs to decide for itself.

Josiah Renaudin: You just used the term proto-agile. I'm curious. Do you think as we're moving forward, as we're seeing agile evolve and used in different situations, do you think we'll hear more terms like that, maybe different takes on agile, that people will be talking about instead of implementing agile, they're using maybe an evolved or transformed version of it and giving it different names?

Mike Cottmeyer: Well, I've really struggled with what to call that first step down the path. I happened to say proto-agile because it's not really agile if you think about it. I've been trying to untangle the word "agile" for probably the last eight years at this point. There's a lot of different kinds of views that you can take. Some people view agile as a cultural phenomenon. Some people view it as a leadership framework. Some people look at it as a set of practices. There's a lot of different things, and if you're in the camp that describes agile as this cultural value system where we respect and empower people and we let them decide and we let them self-organize and we put them at the tip of the spear as far as solving the business problems at hand, right? I mean, that's a great way to work.

It's very much like a startup kind of mentality, and I think that's what a lot of us in the agile community are going for. When I describe the first step that a lot of companies have to take to go down that path, what they tend to have to do is they tend to have to form teams. They have to be really aggressive about figuring out how to create backlog and making sure that those teams are reliably and predictably delivering a working tested increment of product every couple weeks. They have a lot more measurement. They have a lot more control. There's a lot more specificity in the requirements, and a lot of times when you describe that, in one way it looks a lot like agile, a lot like scrum.

It's team based; there's clear backlog; there's velocity; there's burn down. There's all those things, but you're doing it in a heavier, more commanding control mindset that is often antithetical to the value system that a lot of agilists are prescribing. I make the distinction that sometimes the first step is to install the teams, put in an agile governance framework, and get the mechanics of that team-based iterative and incremental delivery in place first so you can start working on the culture as you begin to decouple that team from the rest of the organization. I think a lot of times in our community, we blend this idea of well, we're doing waterfall and agile. Well, I don't think that's really the right thing.

I think that there's some certain non-negotiables if you're on this path toward agile that you don't want to compromise on, so I'm not really talking about taking shortcuts to agile, but I am talking about figuring out and distilling what is the essence of agile delivery. Then it becomes a matter of the mindset that you approach it, and the more you can decouple the entities within the organization, the more agile and adapted you can let those individual teams be. It's really hard to do in a short format interview like this, but it's very, very nuanced in terms of what does it mean to blend approaches.

We just anchor on the fact that you have to form complete cross-functional teams. Those teams have to have really clear backlog, and they have to be able to produce a working tested increment every couple of weeks. Past that, almost everything is negotiable in a first step towards greater business agility.

Josiah Renaudin: Absolutely.

Mike Cottmeyer: That was a mouthful. Yeah, sorry.

Josiah Renaudin: No problem at all. Now how long does it often take to transform a bigger more traditional workplace into an agile one? What are the best methods for sustaining these changes for the years to come?

Mike Cottmeyer: Wow, big question. There's a lot of companies that I think are on just a journey. They're probably on a journey in this forever. There are consultants obviously there forever, but this is going to be something they're going to have to constantly put energy into. What we usually do is once a team has been identified and you've got an agreement on how you're going to create backlog and you've got some basic practices or methods for producing that working tested increment of product, what we generally find is that it takes about two to three months to get a team to set around the basic practices, cultural values associated with agile.

Again, when you think of it as a journey or as a continuum, that first pass to get minimally articulate, minimally performing within an agile framework is usually a couple months. Depending upon the size of the organization and how hard you push, oftentimes you can just think about the organization is just a large collection of teams because that's ultimately where you want to go.

If you think about your deployment patterns and how you're going to put agile into the organization, if you get 800 people broken up into 100 different teams and realize that each team's going to take two to three months to get basically set into these pattern, you could probably get, with the right amount of support, you could probably get a working sourced path in agile, but again, right, probably this proto-agile I was talking about where it's delivering in that two-to-three-month period.

In practicality, that's probably not how you would implement this in an 800-person organization. You'd probably do it in waves, and you'd probably spend a couple years doing it, but over time to get to that really high-performing, loosely coupled agile, very, very dependent upon the nature of the complexities and interdependencies within the organization you're dealing with. It's just absolutely just not a one-size-fits-all approach for us.

Josiah Renaudin: More than anything, what message do you want your audience to walk away with after your talk in the coming months?

Mike Cottmeyer: More than anything, I want people to understand that dependencies between different parts of the organization are absolutely what is killing agile. If we can't figure out how to create complete cross-functional teams where those people have autonomy, if we can't agree on how to create a backlog, if we can't produce something of value every couple of weeks, we are absolutely going to fail. Almost every compromised agile you see, some scrum-butt or whatever you want to call it, is because somebody's made a compromise to accommodate some sort of dependency or external factor that they can't directly control.

When they do that, what ends up happening is that people more often than not end up going through the motions of scrum. They end up doing scrum, so they're doing daily stand-ups, and they're going through the planning rituals. They might even be really good at it, but they're not delivering in a way that's consistent with how agile’s supposed to operate. Again, the big part of what I'm talking about here is how do you systematically organization-wide begin the process of breaking dependencies so that even if it takes you a couple years to where some point you've implemented a pattern where those teams have a shot at being independent from each other.

Because as long as we're managing dependencies across teams, the more dependencies we have, the less agile we're actually going to be as an enterprise. To me, that's the story. It's like break dependencies at all costs and then manage the hell out of the ones that you've got left over.

Josiah Renaudin: Fantastic. Well, I really do appreciate you taking the time to talk to me today, Mike. I'm looking forward to hear more about your thoughts on agile in the coming months.

Mike Cottmeyer: Yes. Thank you very much for having me. Appreciate it.

LeadingAgile cofounder and president Mike Cottmeyer is passionate about solving the challenges associated with agile in larger, more complex enterprises. To that end, his company is dedicated to providing large-scale agile transformation services to help pragmatically, incrementally, and safely introduce agile methods. Mike spends most of his time leading and growing LeadingAgile, and providing strategic coaching for clients. He was on the steering committee that created the PMI-ACP certification and co-led the creation of the DSDM Agile Project Leader certification. A fellow of the Lean Systems Society, Mike served on the boards of the APLN and the Lean Software and Systems Consortium.