Deliver Value; Don’t Measure Efficiency

A couple of weeks ago, I received an email from a ScrumMaster friend of mine. A group of managers at her company had determined they needed a metric to measure project efficiency, so they came up with the idea of comparing the number of forecasted story points for a time frame with the number of story points that were actually delivered by a project team. They felt the ratio of actual story points to forecasted story points for all the sprints in a given time frame would demonstrate whether the project is delivering to its capacity for work according to business priorities. The managers thought the measure would indicate whether business value was being delivered according to forecasted timelines, and it could be used for guiding continuous improvement efforts.

Here's how I responded:

“I don't like it at all.

It's measuring output, not outcome. I'd be much more interested in whether the team is delivering value, and in some cases you may be able to provide more value by delivering fewer story points. Of course, the problem here is that value can sometimes be very difficult to measure, so people like to fall back to measures like this, which in all reality is just a distorted "earned-value" calculation.

It's measuring the wrong thing—this is really telling you how "accurate" estimates are, which I think is the wrong thing to look at. See my first point.

The team did not come up with the measure, so they most likely were not able to weigh in with what it actually means.

It will drive bad behavior. "Oh, we're being measured on how many points we get done. Okay, we'll inflate our point estimates, and we'll do easy stuff that we know we'll be able to get done." The story points lose their usefulness as a tool to help the team predict.

What value does this metric bring? What will you do with this information? What are you trying to accomplish with this measure?”

My friend replied that she agreed with my assessment and appreciated the backing. She also asked if I had any ideas on measuring business value. She mentioned that she had heard about assigning value points to stories but wasn't sure she liked that, either.

My response was not as direct as the one above, but I did point out that starting with stories and then trying to derive value was, in effect, doing things backward. Ultimately, delivery teams should be concerned with helping their stakeholders solve their problems (i.e., delivering value). In the case of software development efforts, a delivery team should also strive to solve those problems with as little code as possible. The more code they write to solve the problem, the higher the likelihood that today's solution will be well on its way to generating tomorrow's problems that will be solved by—you guessed it—more code. It's this vicious cycle that generates the bowls of spaghetti commonly known as "legacy systems." Establishing a backlog of stories that the delivery team will deliver before determining their value will inevitably drive teams to produce more than they need to. Even if you assign relative value scores to stories with some stories getting higher scores than others, the team will more than likely assign "value points" to everything, even those stories that really don't accomplish anything. This is a concept described as feature injection.

Instead, delivery teams should work with their stakeholders to understand what a good measure of value is first, before identifying any stories whatsoever. Since value is such a subjective concept, you can think of this as answering the question "Why are we doing this?" In this question, “this" is usually some goal that the organization is trying to solve, and it should usually be expressed in a specific, measurable, agreed-upon, realistic, and time-framed (SMART) manner.

For example, if a health insurance organization wanted to streamline their claims processing, it may set as one goal to reduce its received paper claims from 5,000 per month to 100 per month by the end of the year. After setting this goal, the delivery teams and stakeholders identify who can impact reaching that goal and who could stand in the way of reaching that goal. In the health insurance example, this may include office managers, billing personnel, and individual patients. Then for each of these actors, the team should consider what behaviors those actors need to change in order to drive closer to that goal. For example, we may want individual patients to send us their claims electronically rather than filling out paper forms. Finally, the team thinks about the solutions they could provide that would help the actors change their behaviors. The delivery team could produce a tool that scans electronic forms emailed to the health insurer extract information from those forms,and load it into the claims processing system, or the delivery team could produce an online claim form on the patient-facing website. This process is called impact mapping, a technique created by Gojko Adzic that, as he puts it, helps teams "make an impact, not just ship software."

After the members of the delivery team create an impact map for their effort, they can begin selecting a feature at a time to deliver and gauge its impact on the final goal. They may find that by simply creating the web-based claim form for patients, they immediately are able to reach their end goal and can stop there. Whereas, had they gone the route of brainstorming a set of stories, they may have come up with a slew of possible features and felt the need to develop most., if not all, of them, because that was the definition of their scope. Impact mapping, on the other hand, helps teams to identify the assumptions they are making and test whether those assumptions are correct. For example: We think a majority of our paper claims come in from patients. We assume that if we provide a web-based claim form for our patients to fill out, they will use it instead of using paper. The delivery team can develop that one feature, test their assumptions, and then move on to decide whether further development is needed.

If the delivery team members in our health insurance example were being held to the metric that my friend was having imposed, they would most likely create a backlog filled with stories that introduce many of the features identified via the impact mapping exercise. However, because they are acting more from the scope/time/cost paradigm, they are likely to commit to a certain number of stories and strive to get those done. Even if they deliver the web-based claim form first, they may actually be incented to keep delivering functionality. Even though the web-based claim form solved the problem, their actual delivered story points may end up being considerably lower than their forecasted points, especially if they have to provide a forecast at the beginning of a quarter. The same would happen if the delivery team assigned value points to their stories. It's true that this relative valuing may help with making priority decisions, but it wouldn't stop the team from potentially delivering more than they needed to in order to meet the specific metrics. The trouble is those specific metrics are looking at the wrong things. The number of value points I deliver in a quarter doesn't tell me whether I helped a stakeholder solve the problem.

My ScrumMaster friend doesn't work at a health insurance company, but she does work at an organization that has just started adopting agile approaches and is typical of many organizations making the move, in that delivery teams seek to understand and utilize all the practices, but their leaders are reluctant to adopt the mindset shift that is necessary for effective efforts moving forward. They tend to ascribe to the idea that you can't manage what you can't measure, and they seek to equate things in agile approaches based on their existing paradigms of managing scope, time, and cost. Unfortunately, that paradigm does not always lead to the best outcome for the stakeholder.

A focus on value—truly seeking to solve your stakeholder's problems—will lead to the best outcome for the stakeholders. By starting with value as defined by solving your stakeholder's problem and seeking to solve it in the most efficient way possible, it is very likely you will be able to keep within scope, time, and cost constraints much easier than when you start with a laundry list of things that you want to do. Starting with value first is truly the way agile approaches can make things cheaper, faster, and better.