The Lean Paradigm Shift – Lean Science
Lean represents a paradigm shift from focusing on increasing productivity to focusing on shortening the time from the beginning of work to the completion of it. While Lean adopters are looking for higher productivity and lower cost, they have learned that going after these directly actually results in lower true productivity (value delivered per person) and higher costs. The reason is that productivity measures too often are geared toward how much work people are doing rather than how much true value is being delivered and cost, alone, is inadequate for deciding on process and/or product improvements. For example, measuring how much work people are doing leads to keeping people busy. Unfortunately, this leads to overworked analysts, developers and testers are incredibly busy while seemingly taking forever to deliver what the business stakeholders need. It does not translate into true added value.
Lean science could be said to be the set of testable knowledge based on the foundations of Lean thinking. To summarize, these foundations are
Many companies have heard that the concepts and methods of lean production would be of use to their organization; however, they do not see how something that sprang from manufacturing practices could apply to software development. This article presents a different way of looking at Lean Software Development—one that is independent of Lean’s manufacturing heritage.It begins by presenting Lean as a collection of a body of knowledge applying Lean principles to software development. It then shows how this creates a new paradigm of management, one that does not inevitably lead to micro-management or chaos. Finally, it concludes with a discussion about how organizations can use Lean to improve their ability to learn.
- Attend to the system in which development takes place; that is where the bulk of errors are
- Respect the people doing the work
- Attend to the time from when work starts until it is consumed by the customer (“cycle time”)
- Get to the root cause of errors when they occur
The foundational tenet of Lean is, “focus on shortening cycle time.” Productivity will follow. In other words, Lean is based on the hypothesis that shortening cycle time raises productivity by reducing the delays that make error detection more difficult and it lowers cost because this delay in error detection increases the amount of work needing to take place.
Consider the type of work we do in software development, as illustrated in Table 1.
Table 1: Activities performed during software development
These items are not listed in any particular order; I am not implying any particular process. The activities on the left represent work that provides real value while those on the right represent work which we often must do but which don’t really add value. In the Lean world, these latter activities are called “waste.” And it is good to try to eliminate waste.
The problem with trying to eliminate the waste is it is often created by people who are not doing the work. Consider the problem of two ditch diggers. Both are working hard. One is digging away and throwing his dirt into the other guy’s hole. Stepping back, you can see the first guy is creating extra work – “waste” - but down in the trenches, the second guy just sees still more dirt that he has to remove. Sometimes waste can’t be seen until you look at the entire picture.
It is easier to say “eliminate waste” than it is to actually eliminate it. We can’t just stop doing wasteful things; we must stop the need for doing them. Let’s consider the wasteful activities on the right-hand side of Table 1 and what is driving the need for them. It will become clear that the root cause involves delays, and that is what we want to eliminate. [1]
Re-doing requirements is a common practice in software development. It occurs because we often get requirements too early. There is a delay between when a requirement is first mentioned until the development team starts working on it. Inevitably, the requirements will have to be updated or redone; otherwise, we will have another activity, working from old requirements, which clearly causes wasted effort.
Consider “fixing” bugs. The quotes around the fixing are intentional: although most developers believe they spend a considerable amount of time fixing bugs, in reality they spend more time looking for the bugs. This is not just semantics. Imagine a developer who writes a bug but is immediately told about it. She can probably fix the problem fairly quickly. Now, consider the situation where the same error occurs, but the developer is not told about the problem for 2 weeks. It will take her considerably longer, even if nothing else changed. It is even worse if it falls on someone else to fix it. Where did this new work come from? It comes from the delay between creating and detecting. I call this “induced work” because it is caused by delay: Just like moving magnets induce electricity in wires, delays induce work in development.
A similar phenomenon happens with “integration” errors. Virtually every “integration” error I have seen in my years of development has been caused either by communication errors between two groups or one group not doing what they said they would do. While such an error is detected at integration, it doesn’t make it an integration error.
Note how all of these “wasteful” items, including building unneeded features is exacerbated by delays in information or in people being available. Overbuilding frameworks is perhaps the only item in the right-hand column that is not caused by delays. [2]
Let’s look at the work in Table 1 again. How much time do you spend on the left hand side and how much on the right hand side? Seriously, stop and think about it a minute. In my classes, when I ask people this question, people tell me they spend between 30% and 70% of their time on the left hand side, doing useful work. This means that between 70% and 30% of the time is spent on the right hand side, doing work that is essentially useless!
Now, we can speed things up either by doing the work on the left hand side faster or by figuring out how not to do the work on the right hand side at all. While you might be able to do the work on the left hand side faster, with the exception of automated testing, it is not likely you will realize significant improvements doing so. On the other hand, getting better (faster) feedback and having the right people work on the right tasks at the right time can virtually eliminate most of the work on the right hand side.
Shift your focus from productivity to working on the right things at the right time with the right folks. Better productivity and lower costs will come as great by-products. How to do this in software development is exactly what Kanban focuses on. [3]
Working on the Right Things at the Right Time
So how to shorten the delays involved in our work? In large organizations I would assert that we must include the work done prior to the development teams. We can think of product development flow to have four stages. These are shown in Figure 2.
Figure 2. The stages of product development flow.
Identification and sizing of business capabilities. This is the first step. Business stakeholders (typically business executives or product managers) identify the business capabilities needed. They size these by refining them to their minimum marketable features (MMFs), sometimes called minimum viable features (MVFs).
Prioritization of these capabilities. For organizations with more than one product line, the business stakeholders and product managers that are associated with those products must decide which of these MMFs are the most important. This is product portfolio management. An organization must look across all of its products in order to determine what to work on next. Some products will return most of their value in a short period of time while others may continue to deliver large returns with each investment. It is important that teams not get locked into their products but rather switch to a product with a higher return if it makes more business sense.
Creation of stories for the teams to build these capaibilities. Once the MMFs have been prioritized, they need to be broken down further before the teams pull them to actually build the capabilities. It should be apparent that for large organizations this flow will be somewhat complex. Each business capability will likely be split out across several teams which must build different components that have both technical and business dependencies. A technical dependency between components means that one component requires another one to work. A business dependency means that although the components may work without each other, their value won’t be realized until all of them work. This is not unlike when you wait for your bags after a flight. If you have three bags, getting the first two doesn’t give you any value. It is only when that third bag arrives that you can leave.
A Short Lesson in Pull
To understand the flow in Figure 2, you need to understand a key Lean management concept: pull. Planning work is very difficult in high-variability situations – which software development certainly is. The idea of pull is that rather than planning the timing of our work, we have each step pull work from the output of the previous step. For example, in Figure 2, the development team (depicted as ‘building the capabilities’) would pull from the work area preceding it that creates the stories. [4] The people doing the work here need just enough work to ensure that when a team is ready to start a project there is enough work there for them to do so. Perhaps a little more for safety’s sake since sometimes projects are longer or shorter than expected. In other words, the development teams should be working as fast as they can, working to their capacity, and pulling work whenever ready. The upstream folks know to create another project to be ready when the developers pull one off the ready queue. [5]
By managing with pull, the organization focuses on lowering the queues between the different steps. As the queues get smaller, the work flows faster since the biggest delays in most software development is not so much the work itself as much as the time between the steps. To see this, consider how much time people are waiting for answers or approvals.
The Need for Management
It should be apparent that trying to coordinate this sequence of events with its accompanying business dependencies and technical dependencies requires a larger perspective than a team view. Not one of micro-management from the top. But one of recognizing global interactions are different than local ones. How do we reconcile these different perspectives – the one from the top that is attempting to make a cohesive flow and the one from the bottom where the actual work takes place? Lean science provides a basis for aligning both global and local efforts because both should be looking to shorten cycle time in whatever they do.
Current management places a lot of emphasis on ensuring that people are working productively. Lean management would have management shift from trying to motivate people to providing them with the proper work environment in which to get their work done. “Work environment,” means more than simply the physical environment they work in. It includes the way they integrate with others doing development. Management’s role shifts to a larger view: helping to create an organizational structure in which the people doing the work can best function. Instead of managing by telling people what to do, managers now assist teams in getting their work done by providing the proper environment for them to do so and coaching as required.
In large organizations, managers can best assist teams by helping coordinate their work so that they are not over-burdened. Overloading teams is one of the primary causes of serious delay. The Lean manager’s role is both to focus on how work flows and in coaching teams to attend to avoid delays. Management provides the high view of the organization needed while providing assistance to work done at the team level.
In Figure 2, management’s role could be considered to be getting proper flow of work from the left to the right. This would entail having the right number of things being worked on at each step of the process. Imagine how having the right number of items hitting the team for them to pull from will help their productivity. A secondary effect will be that product owners will be more available to the teams as well. A common challenge in software development organizations is the lack of product knowledge which slows the teams down. It is ironic that the very people who can help them (e.g., product owners) are spending time creating more stories than are necessary. By limiting how much work can be upstream of the team pulling their work, the organization is also simultaneously making these people needed by the development teams more available to them.
Lean Learning
Development systems are very complex. Cause and effect are very difficult to determine. Many events are unpredictable. And since people are involved, events can even be irrational at times. This means we have to learn quickly and challenge our assumptions. Lean’s underlying systems thinking approach enables a three step approach for learning. [6] The first is to see where you are. Then, look to see where you want to be. Finally, take steps to get there. This learning approach specifies to take very small steps. If your steps are too large, many things will happen from when you start until you see results. This will make it difficult to see the results of your actions. Kanban methods follow this by using explicit policies (e.g., limitations on work in progress at each step of your process). You can readily see the results of these actions if you have your work reflected in a value stream map or Kanban board. [7]
Putting it Together
Becoming Lean is truly a journey and not a destination. It involves creating visibility where you are so you can see what you need to do to get to where you want to go. You strive to improve your work by removing delays between the steps of the work. One method of doing this is setting work in progress (WIP) limits so you do not exceed the capacity of your organization at any step. By eliminating delays, you lower induced work which raises true productivity and quality, while lowering cost. Because we are taking a scientific process here, all the roles of the organization can see if our actions are helping or hurting. Overall cycle time is our measure of efficiency of the organization. Local changes can be made with confidence of overall improvement if they lower our overall time. The goal is continuous improvement by improving how we work to get better products for our customers. We learn in small steps so we can understand the results of our actions. Visibility is not just limited to how our work is flowing but includes the rules we use for making decisions. This enables managers to assist development teams since they can work to have the proper organizational structure they need as well as coach the teams when they need it.
Lean Software Development is a combination of Lean science, Lean management and Lean learning. It provides an overall approach that helps different roles in the organization to see how progress is being made, both in the products being built and in the way they are being built By creating visibility and a common language they help create a better, more productive team.
[1] I’ll be making several assertions in the article. An assertion is a statement where I am promising to provide evidence to validate it. Except in this case, I’m suggesting you should provide the evidence by looking into your own experience. The impacts of delay in software development are somewhat universal – I do believe if you look closely you’ll find evidence for what I am saying. If not, please ping me on the leanagile Yahoo user group.
[2] Overbuilding frameworks is typically caused by lack of knowledge of how to do proper emergent design. The interested reader is referred to Scott Bain’s Jolt Award winning book, Emergent Design: The Evolutionary Nature of Professional Software Development.
[3] The term MMF (minimal marketable feature) was coined in Software by Numbers: Low-Risk, High-Return Development by Mark Denne and Jane Cleland-Huang. It is essentially the smallest piece of value that can be delivered that is worth the transaction cost borne by both the development company and the customer.
[4] Please don’t interpret this diagram as depicted a phase-gated approach. The reality is developers are represented early in the cycle as well and there is no intention that story definition is completed in the third box. Rather we mean just enough to get started building the software.
[5] This coordination is not usually done in real-time but rather by checking on a regular basis (e.g., weekly) to see if new work needs to be prepared. The regularity of this checking is called the cadence of the step.
[6] I highly recommend Mike Roth’s Toyota Kata: Managing People for Improvement, Adaptiveness and Superior Results. While based on Toyota (and therefore manufacturing) the learning process equally applies to knowledge work.
[7] Scrum boards can accomplish this but typically don’t as most Scrum teams do not explicitly state how they move work that takes place within a Sprint. Rather they limit their explicit policies to be how stories get on (Sprint ready) and off (done) the board.
