Agile works. Early adopters, working largely by instinct, have seen good success. To go to the next level, instinct alone is not enough. As we face more complex and uncertain environments, as we face the need to scale to the enterprise, we need to apply intelligence and knowledge, guided by experience. Knowledge about why Scrum works.
Knowing the whys behind Scrum lets us perceive opportunities for improvement that might otherwise be overlooked. It lets us evaluate practices to identify good and better practices. It helps us adapt practices—based on principles—to new or similar situations. It is how we have started realizing success at helping our customers deploy Scrum in their enterprise. I have been doing some form of Agile development for over 20 years. For the first 10 years, my approach was somewhat informal. While I did use techniques such as automated testing, I did not fully understand its general value, seeing it more as a solution to a particular problem. Failing to see the why, I overlooked countless opportunities when it would have helped. Not until 1999, when I started studying Agile methods in greater detail, did I appreciate the value of this approach. I believe in the value of understanding "why." For myself and for teams I work with.
Over the last eight years, I have had the privilege of being a trainer, coach, and mentor for dozens of teams migrating to Lean-Agile methods.This requires a fundamental shift in their behavior, rather than simply knowing what to do next. It requires a confidence that the new approach will make a difference. This confidence is essential because when things get tough, people begin to fear that they will not succeed with the new approach, so that usually revert to their previous (bad) habits. Again and again, I have seen that when teams finally understand the whys, they finally gain the confidence to work through the fear. To know that they can work through the uncertainty that is always present when you are doing something new. And not only teams. Management seems to be much more comfortable when they understand the rationale for a particular practice. When was the last time you convinced someone with "well, it worked over there!" Or imagine telling a manger to use self-directed teams because that is what makes Scrum work...Manager: "You mean I should just let my team figure it out?" The same guys who can't deliver software on time? Yeah, right!'"
Why Does Scrum Work?
Understanding why Scrum works involves understanding where a team is in its evolution. This seems to involve four main phases:
- Getting organized (starting with Scrum)
- Using Scrum to manage a project and improve the team's process
- Sustaining Scrum by attending to quality assurance and code quality issues
- Extending Scrum into the enterprise with Lean thinking
Based on what I have seen, most teams that have adopted Scrum are in Phase 2 (Phase 1 being a pre-cursor to Phase 2). Much of the success attributed to Scrum is may simply be due to the team organizing itself better. Let me illustrate. Suppose you are the team lead in an IT organization that uses a somewhat stifling, heavy-handed process. You have to work on five or so projects at the same time. Now, your manager offers you the chance to pilot a new methodology called Scrum. He states: "Since you are doing a new process, you can ignore our current procedures so we can see how Scrum does it. We will co-locate your team. And you will only work on this one Scrum project instead of the five or six projects you are working on now. We will make sure management doesn't interrupt you with other things. And, we'll give you access to our marketing manager who knows what the customer wants so she can answer your questions." What would your reaction be? Hallelujah!
Would you expect your team's performance to go up? Of course! Even if they still did development in the same old way, they would still do better because:
- The team would be able to work together as a unit, eliminating significant communication delays and challenges.
- You would have direction from someone who could tell you what was important to the customer.
- You would be working on only one project at a time.
- You would not waste time following procedures that were not useful.
You almost cannot fail to see some positive results quickly, but it will only take you so far. When you actually implement Scrum, you'll get additional benefits because:
- The iterative natureof Scrum development (including the re-assessment of priorities between iterations) enables you to build code that is of value to the customer.
- Thefocus on defining acceptance tests concurrent with requirements improves both the quality of the code and the clarity of the requirements.
- The ability of the team to determine how to remove impediments as they arise allows them to improve efficiency.
Phase 1 produced significant improvements by changing the organization of the team: eliminating delays in communication and unproductive procedural overhead. The additional improvements in Phase 2 come from using a better process. Note that the first two benefits of Phase 2 (iteration, test-early) come more from the team's general approach than from the team figuring out how to solve things.
Structure and Process
Over the last eight years, I have worked with dozens of companies, helping them lay out roadmaps for adopting Agile practices. Sometimes, I encounter teams who have show dramatically greater productivity than their counterparts, even though they otherwise follow the same development process. These more effective teams always have the following elements:
- They are organized as work-cells.
- They are co-located.
- They work on only one or two projects at a time.
- They have access to people who can tell them what needs to built.
It is improved structure, rather than process, that accounts for their superior productivity. These are the same structural elements that take place in the first phase of adopting Scrum. I have seen this so often that I can conclude that first dramatic improvements from Scrum are more likely due to structural improvements rather than iterative process or self-managed teams. Why is this important? Because I suspect many teams who cannot adopt Scrum processes are missing out on the chance to improve effectiveness simply by adopting the same Phase 1 structures that Scrum teams employ. They can enjoy these benefits even if they cannot go to Phase 2.Lacking the knowledge of what drives Scrum, they fail to gain the experience.
Where to Go Next
Where do teams look to see what works? It sounds great, as some Agilists advocate, to "let the team decide." But what happens when what is needed is beyond the team's capability to change? Or what if the team does not want to follow good procedures? How do we decide when practices are good and must be followed and when practices are not clear and the team should decide? Our industry is very far from having an established set of best practices even though thought leaders align fairly strongly on them.
In the past, this has been done through mandates by management which led to wasteful procedures and bureaucracy; however, this does not mean that going the opposite way, giving teams full freedom to do whatever they want, is the right way to go. This is fraught with its own set of problems. What is needed is a combination of best-practices across an enterprise implemented in a way that supports the team.
Assuming that teams can discover their own best practices assumes that teams are properly trained in many disciplines: process, technical, quality, technical. Too often, this is not the case. Most still do not understand design patterns. Too many focus on story cards as their sole analysis tool. They lack knowledge to guide their experience.
Long term success with Scrum does not come simply by having teams discover answers for themselves. Instead, it comes by discovering answers that support the team. It requires both management leadership and team involvement and a willingness to learn and improve continuously.
This requires looking both at what the team does and at what constraints are placed on the team over which they have no control. When the team and management understand the principles behind Scrum, they can work together to improve the process of the team.
Looking for Standards
How do we decide what should be standard work for teams and what should be left to the team to decide? Do we tell the team, "you must do this no matter what?" Probably not. But clearly, they must be guided and strongly encouraged. They need to be aware of practices that would be beneficial. They need to know principles to guide their practices. Where should they look?
For software development, one place to look is Lean methods. Lean is the name given for Toyota's management philosophy. Lean tells teams to look for practices that improve systems by lowering errors and making us more productive. Lean practices focus on eliminating delays, avoiding work on multiple projects, continuously improve processes, optimizing the whole, and follow a "Plan-Do-Study-Act" cycle of continuously getting feedback and adjusting to it. All this sounds a little bit like Agile/Scrum development, doesn't it?This should come as no surprise, as, according to Jeff Sutherland, the creator of Scrum,
"Scrum was the first concrete implementation of lean thinking to software development that allowed organizations of all types and sizes to start up lean teams in a couple of days using a standard pattern that was easily understood. What was hard was explaining why and how to implement the pattern to generate continuous quality and productivity improvement.
Today, the writing and courses from Mary and Tom Poppendieck provide a proven set of principles that organizations can use to adapt tools, techniques, and methods to their own specific unique contexts and capabilities. Now we can explain how to use Scrum to lean out software development."
Lean provides an essential foundation for anyone wishing to employ Scrum in the organization.
Summary
Many of the dramatic improvements in Scrum are due more to changes in team structure than to the (iterative, team-led) processes that the team follows. Understanding what affects improvement is important both to improve the confidence of developers in these practices and to increase the likelihood such practices will be adopted by teams in the first place. This understanding can be used to create standard practices, practices that all teams should be aware of and follow. We need to look at the systems we follow and not just expect each individual developer will do the right thing based on their motivation alone. While they may try to do the right thing, not knowing certain basic standards will result in lost opportunities. Lean thinking can give us insights into how to find these best practices to improve software development processes. Understanding the connection between Lean and Agile is essential to enjoying the benefits of Scrum beyond the single team (Phase 2) to the rest of the enterprise.
[1] Implementing Lean Software Development: From Concept to Cash, p. xviii, Mary and Tom Poppendieck.
