How to Use Your Data in an Agile Environment: An Interview with Larry Maccherone

Josiah Renaudin: Well, welcome back to another TechWell interview. Today I'm joined by Larry Maccherone, the director of analytics and research at AgileCraft and a keynote speaker at Better Software East who will be talking about making better decisions with data. Larry, thank you very much for joining us today.

Larry Maccherone: Well, thank you, Josiah. I'm really pleased to be here.

Josiah Renaudin: Absolutely. Before we really dig into the actual meat of the keynote, could you tell us a bit about your experience in the industry?

Larry Maccherone: Sure. I was working on a Ph.D. at Carnegie Mellon University in 2009, where the Ph.D. was really on software engineering metrics; it wasn't very agile-specific. But Watts Humphrey and I, Watts was my advisor, came up with a way to reintroduce metrics back into the agile world, that it largely rejected it—throw the baby out of the bath water, so to speak—because it was hurting agile transformations. We came up with a way to actually do metrics in an agile environment that didn't actually harm the agile transformation.

I gave a talk on that at the 2009 agile conference and I got three job offers out of that talk. The tool vendors wanted me to come work with them and essentially implement this concept into their tools. Their customers were asking for metrics but the coaches were resistant—being the agile coaches, were resistant—so I seemed to have a way of bridging that gap, so that's how I ended up moving over to the agile world, and I was there for seven years or so.

Josiah Renaudin: I think so often with these discussions, if it’s agile or DevOps, we start talking about the actual topic without defining anything. Can you define metrics and measurements, especially within an agile environment?

Larry Maccherone: I really use the words metrics and measurement more to sort of connect with what people are asking for, but really, it's about feedback. Think of a Fitbit, you wear a Fitbit. What if you were to tie the Fitbit to the ceiling fan? That would be silly, right? Because you defeat the purpose of the Fitbit in the first place. The purpose of the Fitbit is to give you feedback so that you can help improve yourself, and I like to think of metrics as feedback, primarily. The other way that I like to think of metrics is in terms of insight.

I have this mnemonic called ODIM. ODIM is basically what I'm about to say backwards. Measurement leads to better insight, better insight leads to better decisions, better decisions leads to better outcomes. And the reason I like the mnemonic ODIM instead of MIDO, which would have been in the order I just said it, is that while that's the effect of measurement, you actually have to start with the outcomes in mind when you're building a regiment.

The NBA has a star. I won't name who it is here because some people get upset, but someone, he's one of the top scorers in the NBA, but some statisticians noticed that when he was out sick or injured, that the team had a better chance of winning. But he was still a high scorer, so how did that happen? Well it turns out that, really, how many points you score in a game are a function of how often you take the shot times what the percentage likelihood it is to go in, and he was a high scorer for the NBA only because he took more shots than all of his teammates who had a higher percentage likelihood of sinking the basket. He was literally stealing balls from his teammates who had a better chance of getting it in. The outcome that the team wants is winning games, and the outcome that maybe the individual wants is to be high scorer so he can renegotiate his contract. But really, you need to pick you measurements in such as way that they drive back to the right outcomes.

Josiah Renaudin: It sounds like you're talking about Kobe Bryant, but I understand if you don't want to actually reveal which NBA player you are talking about, but if I had to guess, you know.

Larry Maccherone: It's not Kobe Bryant. Carmelo Anthony.

Josiah Renaudin: Oh, that's another good one, too, actually. Just give Porzingis all the shots anyway. Why is it—and I was reading through your abstract and you mentioned this—but why is it that some people see measurements in agile development as not just bad, but actually destructive? The way you're describing it, it sounds really beneficial, but you also argue that some people see it as this bad thing for the business.

Larry Maccherone: Yeah, I have this other talk that I call The Seven Deadly Sins of Agile Measurement, and what it is is that it's not the seven deadly sins of agile measurement; it's the seven deadly sins of traditional measurement that if you do in agile measurement, you're dead. They really don't work in a traditional environment already and I already mentioned the key one, and that is that it should be all about feedback rather than driving behavior directly.

When you try to use measurement to manipulate your employees, they will manipulate the measurement and it will become useless, so what you have to do is create a culture where they will work the measurement to help themselves improve or to help themselves make better decisions and that is sort of the key difference between the harmful way and the good way. There are a few others as well. Spending too much energy on it, having the metrics be driven by what's most convenient to measure as opposed to ones that lead to the best outcomes. There are seven of these things that I talk about.

Josiah Renaudin: Are there any tools that you're bringing up during the keynote that can help the process of instituting measurements and metrics within your team more seamless?

Larry Maccherone: Tools not as in software, but tools as in techniques and mindset, yes. The ODIM mnemonic is essentially the outline of a workshop I do for folks. You can basically have an exercise that tries to identify the outcomes you're looking for and then works your way back to the decisions that people need to make on a day to day basis to reinforce or deviant from that outcome, so you want to discourage those decisions but you need to have that. You need to know which decisions that would lead to which outcomes and then what insights they would need to help make those decisions. You can have exercises, I have set of exercises for each of those things, and the end of the they come out with a measurement regiment. They've decided on a measurement regiment for their organization that they're going to be implementing.

Josiah Renaudin: You're going to be mentioning anti-patterns in your keynote that often derail a metrics program. Can you give us a teaser on that? What do you exactly mean?

Larry Maccherone: Well, I mentioned the seven deadly sins and I actually have more recently been referring it to it as the eight dragons but those are the energy patterns. The reason is I came up with an eighth one and I was like oh well it doesn't have the same ring to it, the seven deadly sins but anyway those eight things are the things that actually are the anti-patterns. Using measurement as a lever to directly drive behavior as opposed to feedback. Not having a balanced regiment, so for instance, if you try to push real hard on productivity, well what's going to happen to your quality or responsiveness? They're going to go down so you really have to make sure that your metrics regiment is balanced. These are essentially the mistakes, the anti-patterns, these eight things.

Josiah Renaudin: When you're instituting new, let's say methodologies for example, if you're talking about agile or using DevOps, it usually requires some kind of culture or mindset shift. With metrics, measurement, and really starting to use data within your software environment, are there any shifts you have to make to effectively make that transition? Do you have to actually change the way you do things in order to properly institute it?

Larry Maccherone: Yeah, so you have to change the way you do things and you have to change the way you think before you change the way you do things otherwise you won't be able to. A couple of mindset shifts that I really push for, one is to think of every decision you make as a forecast. You might not realize you're making a forecast, you might just think oh well last time we did this it didn't work out so well so we're not going to do that, this time I'm going to do something else and you make a decision. Really, you've forecasted the outcome you've chosen, or the alternative you've chosen, I should say. It's going to have better outcomes for you and your organization than all the other alternatives. Once you admit that that's what you're doing every time you make a decision, you can start to introduce some easy, simple, but still a little more methodical ways of making those decisions, of predicting what the outcomes will be of each alternative.

Josiah Renaudin: We have an almost near-unlimited amount of data available to use these days; at times it can seem a bit daunting. How can you not only pinpoint the valuable data you want to use in your team but also put it to use in a way that strengthens your teams and your projects?

Larry Maccherone: I talk about a different way of doing forecasting. Management constantly wants to know, when am I going to get it? When is it going to be done? That's the quintessential question that management asks of the team. When the team gives an answer, like a date, then whether anyone did it or not, they've essentially made a contract. And agile is about relationships and interactions over contracts, right, so people and interactions over tools and collaboration over contracts, I should say.

Really, you need a different way of giving that forecast that encourages collaboration and instead of feeling, as a contract that you're negotiating here. The way I propose that you do this, is that you don't give a date, you give a probability distribution. There's a 50 percent chance you'll have it by this date, a 75 percent change you'll have it by this date, a 90 percent chance you'll have it by this date and a 99 percent chance you'll have it by this date and management will be like, wait a second I asked you for a date and you gave me this probability distribution, what the hell am I supposed to do with that. Well, how much risk can you bear? Well, the really pointy headed manager will come back and say, I can bear no risk, we're going to make whatever date you say and then you say well we have that date for you, that's this 99 percent percentile one way out here. I don't like that date, management say. Well, really how much risk can we bear. Well, marketing is ramping up. There's a conference we have to have something at but we don't necessarily have to have the full working product by the conference.

Well, why don't we take this model we've created to create this probability distribution, why don't we take this model to marketing and let's do some what if analysis and have a comfort level, there's maybe a 20, 30 percent risk we'll miss a certain date but we'll have something before that date and that will be good enough and you'll feel comfortable with that risk. It's measurement that brings management and the team together as opposed to measurement, the old way of forecasting this, essentially a wedge driven between management, us vs. them.

Josiah Renaudin: I think that's a good overview of the keynote, Larry, but to close things out again—have this act as a trailer, not a compete spoiler about what you're going to say. What central message do you want to leave with your audience? What's the goal of this keynote?

Larry Maccherone: The central message is, essentially, that measurement is for improvement and decision making, but that seems a little too trite and a little too obvious to be the most significant of that. I would say that it's either fundamentally change the nature of the conversation, like I just described, or I would say that ... let me give you three, I'm going to give you three.

Josiah Renaudin: Okay.

Larry Maccherone: That every decision is a forecast. And then another one that I haven't mentioned actually today but I will talk about in the keynote is that when you're making decisions, they need to be about what is right. All too often we have a conversation about who is right, and that's essentially argument, not decision making, so decision making is about what is right, argument is about who is right. These are some key ideas that I'm going to try to highlight in the keynote.

Josiah Renaudin: All right, I'm looking forward to it, and hopefully people will be there to hear you talk more about metrics, measurement, and making better decisions with data. Thank you very much, Larry.

Larry Maccherone: Thanks, Josiah, glad to have the interview and looking forward to the talk.

An industry-recognized leader in agile, metrics, and visualization, Larry Maccherone currently helps a number of companies with the design of their analytics products including AgileCraft and Pendo.io. Previously, Larry led the insights product line at Rally Software which enabled better decisions with data, leveraged big data techniques to conduct groundbreaking research, and offered the first-ever agile performance benchmarking capability. Before Rally, Larry worked at the Software Engineering Institute for seven years conducting research on software engineering metrics with a particular focus on reintroducing quantitative insight back into the agile communities.