An easy way to gauge progress might be to compare yourself to someone else, but what does that really mean? Johanna Rothman explains why the most valuable comparison is to compare what you’re doing to what you could be doing.
I often receive emails from readers asking questions such as “We have ten developers and two testers. How do we compare to our industry?”; “We have six-week iterations and a two-month hardening iteration at the end of our release. How agile are we?”; “We finish twelve stories in a four-week iteration. How many stories do other people finish?”; or “We fix problems in less than a week post-release. How long does it take other people?”
My question to them is usually this: Why do you care? Is what you are doing working for you? If it is, fine, continue what you are doing. But, if you are worried that what you are doing now is not quite enough for your needs, reconsider what you need. Are you finding defects quickly enough? Would you like to forgo the hardening iteration at the end of your project? Would you like to see more throughput in your project? Would you like to fix problems faster—or not have them at all?
To know how you compare, you have to know what you want to compare yourself to. The real question you should ask is: “How are we doing compared to what we could be doing?”
What Results Do You Want to Achieve?
If you want confirmation that you’re great, here it is: You’re great. If you don’t have a meaningful comparison—and an industry comparison may not be meaningful—then you can stop right now and assume you are great. And, if all you want to do is tell someone how great you are, you can surround yourself with sycophants who will listen.
But, if you want to know that you can be better, here it is: You can be better. Unless you are a robot manufacturing the same thing every day, you have room to improve. Since you are a knowledge worker, you are not doing the same thing every day. So, the real questions are: What results do you want to achieve, and what results does the organization desire or need?
The real question is not how you compare to someone else but how you compare to what you could be. Have you ever looked at your bottlenecks and your concerns? That’s where a quantitative and qualitative assessment of your current state can help you to see where you are and where you might decide to go.
In order to do that, you need to know what’s important to you. For example, let’s address the issue of how long it takes you to fix a defect post-release.
I’ve written about the cost to fix a defect post-release (“What Does It Cost to Fix a Defect” www.stickyminds.com/s.asp?F=S3223_COL_2 or manage.techwell.com/article/what-does-it-cost-fix-defect), but I’ve never discussed the duration of fixing post-release. You can imagine where the cost of fixing a defect post-release is almost irrelevant. Say you are a bank or an insurance company or some other regulated industry—or an oil company fixing a broken well. If you’re a software organization, the cascading effect of the defect is much more expensive than the actual cost of fixing the defect. But, the duration of that defect is what’s keeping you up at night, because it’s not just the time that you have the defect—it’s the bad press that goes along with that defect.
This is a case where if you had one hundred people in development and test, you might even put all of them on finding and fixing the problem. Putting everyone on the case is cheaper than maintaining the cost of the defect for one day.
You can see where a regulated industry might have these problems more often than a non-regulated industry.
But, what about the fact that the defect escaped system test? Or, that the defect escaped integration test or unit test? Do you want to do some root-cause analysis and see where it would have been more reasonable to find the defect, given the way you develop and test your product? I think so, and I think it’s reasonable to seek alternatives. No matter how you look at it, any cascading defect that causes you to lose millions or to not care how many people you assign to fixing it fast is a huge problem. If you might have even one of those, you ought to consider how to eliminate it and significantly reduce the risk of those kinds of defects altogether.Look for the Causes and the Context
When people ask me about comparisons, I often discover they want to compare apples and frogs. The response to “How agile are we?” is often “Not very,” which is not helpful to the person asking. She is doing her best, given her context.
A better approach is to ask other questions, such as “What is preventing you from shorter iterations, and why do you need hardening iterations at the end?” The other person may describe the project context and why the organization came to have long iterations in the first place—reasons that may have been applicable at one time but no longer are.
When I asked one colleague these questions, she said, “We didn’t think we could fit enough into shorter iterations, and our testers are twelve hours away. So, we have the too-much-to-do problem on both the development and testing ends of the project.” Lengthening iterations isn’t the answer, though new agile teams often think it is. I used the Five Whys to walk my colleague through the effect of her thinking, and she learned why longer iterations aren’t such a good idea. A value stream map would have shown her the same thing.
Compared to What?
When someone asks you how she is doing compared to others, the best rejoinder is to ask, “Compared to what?” The only relevant comparison is how well you are doing with respect to yourself. Your value stream map will show you if you are wasting time somewhere. Your data can tell you if your defects escaped release. Look for causes of outlier data and the context for it. You don’t need to compare yourself to the industry. You are unique. Revel in your uniqueness, and understand why.
