A Productivity Comparison of Kanban and Scrum

Scrum is a great agile management framework for iteratively developing complex software systems, and it works well in many circumstances.  Certain problems can arise, however, such as a highly fluid product backlog, which make kanban’s emphasis on backlog flexibility a more attractive alternative.  Software experts like David Anderson, Corey Ladas, Dean Leffingwell, and Linda Cook point to small, loosely coupled user stories with an experienced agile team as key factors for productivity gains using kanban.  Yet, few studies exist to show concrete metrics in productivity gain.  In this article, I explain our application of reasoning for using kanban and then show how I computed that our productivity gains were more than 300 percent once we started using kanban.  I conclude this article with thoughts on applying kanban in your circumstance.

The Problem
We were in trouble. Our project’s business side began to have difficulty keeping up with the Scrum team’s productivity, and stories were neither groomed nor well thought out enough to be effectively used as part of the sprint. The product owner was changing the backlog at the last minute, inserting stories into the top of our product backlog on the day of planning, reorganizing the groomed stories out of the backlog, and even cancelling stories that were in the process of being worked as part of the sprint. The Scrum team was often faced with the unenviable task of cobbling together a sprint full of unknowns.

To make matters worse, we were ending the phase of large scale stories and entering a phase where alpha clients would introduce small, discrete stories in the form of bugs. A stream of potentially high priority bugs was about to hit us  requiring faster reaction than we could handle by collecting the bugs into two week iterations.  We had to find a way to make our process more lean.

While agile encourages change, the process cannot be chaos.  Scrum requires thought out stories with enough detail that allows for a coherent conversation with the product owner.  In our case, the product owner represented the wishes of a number of diverse stakeholders and a constant stream of bug fixes. The product was a new acquisition and everyone, including the product owner, was learning what it should do as we rebranded and integrated the product into the company line.

The Solution
First of all, the problem was really a symptom of an underlying cause that was not going to be fixed any time soon. The product was new and there were no product experts – we were all learning.  The product owner, stakeholders, and development team all were unfamiliar with the product, and we had no great vision to follow.  Abandoning a sprint, rejecting stories, or chastising the product owner was not going to move the project along.

We looked to apply lean principles to Scrum’s meta processes and considered ways to shorten the sprint length, the estimation sessions, sprint planning, and sprint demo length even further. This was done with the goal of keeping people working on stories that were of the highest value and well thought out enough that the utility of the stories could be explained. We found that one-week sprints promised to be too long.

After much brainstorming, we decided to use kanban as described by Corey Ladas. We broke sprints down from two-week timeboxes into a single story flow—conceptually a “sprint” became one story and we abandoned the timebox. We limited the teams working on each story to two people—which meant multiple teams were pulling from the backlog—and then kept the work in process (WIP) limited and used the pull technique. Additionally, we developed a ten most wanted list (MWL) for our stories before placing them on a kanban board, and held meetings several times a week to groom the list.

Several stakeholders were able to contribute their priority to separate columns on a scale of 1 to 10.  Each stakeholder had their own priority column.  The columns were summed to get an initial cut at overall order and then the features were ordered based on consensus. The top ten were the focus of analysis during our grooming sessions.

Figure 1. Kanban Board

Once the features were well analyzed and written as stories, they were moved to a pre-ready list and eventually to the ready state on our kanban board, which had a limit of two stories in the ready queue.  Stories were moved to from the pre-ready list to the kanban board whenever the queue had enough capacity (less than two items remained) and the product owner felt the story was sufficiently detailed.  The kanban board was further broken down to Ready, In Process, and Done. Our kanban board is shown in figure . The In Process list was further subdivided into Started (stories that were being analyzed and digested by the team), Code/Test (stories that were being coded via test-driven development), Awaiting Test (stories that were awaiting acceptance testing), and Pending Release (stories that were acceptance tested but were awaiting a patch build). We wrote each story on a green card with an orange tag indicating release build number and a blue tag indicating who was working on the story.

Every day we held standup meetings where the day’s commitments were written on the purple tags next to the stories. There was also an issue list at the bottom of the board for technical debt or other items that needed to be addressed by either the teams or the ScrumMaster. It took less than three days for a typical story to progress through each list on the kanban board.

As you can see in figure 1, acceptance testing and patch builds were batched due to staffing restrictions and other circumstances beyond our control. While the situation was not perfect, our kanban board made the batching under Awaiting Test and Pending Release highly visible.  We used a work in progress (WIP) limit of three stories initially, which we found to be a reasonable number.

Surprisingly, the transition from Scrum to kanban was smooth, due in big part to the team’s experience with Scrum. The new process was explained as an evolution to Scrum, so the team had a context for the changes.  

Our retrospectives were the most difficult part of the process.  We started holding retrospectives every other week (as we did with Scrum), but with the flow of kanban being at the story level rather than a set number of days, the retrospectives were more disruptive than helpful.  We decided to hold retrospectives with the team that completed a story as soon as the story was completed.  This wasn’t an optimal solution either, as the retrospective outcomes tended to be very myopic.  By the end of the project, this problem was still not solved. See the link in the Further Reading section of this article for a full description of our process.

The Value
While it may seem we were comparing oranges with frogs, we were able to capture metrics that pointed to kanban’s being significantly more productive than Scrum. This occurred in our specific instance and our metrics were based on practice—this was not a controlled experiment.  Here’s how we were able to support our conclusion.

For comparison purposes, we used story points per staff week—something that should be used with the same care as lines of code—meaningful at the project or macro level though not at the individual or micro level. We found that the metrics supported our feeling that kanban is more productive than Scrum given the right circumstance.  I would expect that a reader would agree that our sheer size in productivity gain indicates that kanban merits investigation.

We had one Scrum team working for a total of twenty-six sprints (fifty-two weeks).  After fifteen sprints, we added a second sprint team for eleven sprints.  Both teams pulled from the same backlog, which made coordination less difficult than if the two teams depended on each other. The Scrum team completed 655 story points in 518 staff weeks.  That’s 1.26 story points per staff week.

After twenty-six sprints, the first sprint team moved to kanban and absorbed one person from the second Scrum team.  The remainder of the second team was disbanded.  Our kanban team consisted of eight people working for ten weeks for a total of eighty staff weeks.  During this time, we completed 340 story points. That works out to 4.26 story points per person week (340/80). 

These metrics are summarized in the table below:

A Discussion of the Metrics
At first blush, the difference seems like a tremendous increase, too good to be true. Certainly, the process alone could not account for all of this increase.  A number of factors contributed to the productivity gains.

The Scrum and kanban teams were more or less the same people, meaning that variability inherent with changing team members wasn’t pronounced, but was something of note as the kanban team consisted of eight people rather than seven.  This didn’t seem to be much of a factor since the “new” person was at one point a coworker with the original team, had worked on the product for some time, and was familiar with the agile techniques. 

There was a small impact on the process learning curve.  When starting with Scrum, the development team took about eight sprints before velocity stabilized. We were learning the process and our estimation skills took time to mature.  While this may seem to skew the Scrum metrics, even ignoring the first eight sprints had only a marginal effect on the comparison. Since the teams already collaborated well, understood many of the mechanics of agile development, and had experience estimating, the velocity’s stabilization wasn’t pronounced. Kanban was introduced as an evolutionary change from Scrum and many of the concepts carried over.

In both cases, story point estimations were based on a list of reference stories using estimation by analogy.  When a new story was estimated, it was compared to a reference list of completed, well-understood stories with already established story points.  The new story was then assigned the same number of story points as the analogous reference story.  While there was an inevitable degree of variability when estimating story points, having the reference stories on hand helped keep variability small.  In Scrum, the reference stories had to be grown.  By the time we began using kanban, the reference list was well established.

Perhaps the quality and size of stories was the biggest difference between kanban and Scrum.  With Scrum, the story size ranged from 1 point to 13 points with a mean of 5.25, while kanban’s mean story size was smaller at 3.4 and varied between 1 and 8 points. The business team, while using Scrum, was too distracted with learning the product to be able to flesh out quality stories that could be collected into two weeks’ worth of work. With kanban, stories were fleshed out and prioritized just in time.  The product owner was on his toes because he knew the story would be picked and worked immediately in the kanban world.  With Scrum, our product owner assumed that a story may not be played until toward the end of a two-week sprint and by that time, the thinking went, the story could be fleshed out.  This is usually the case with agile but in our instance, it was extreme—stories were very vague before the sprint began.  Our product owner didn’t always have the appropriate motivation to create high-quality stories before being batched into a two-week sprint, but with kanban, the story was immediately addressed and, hence, had to be higher quality.

Why Did Kanban Work
Kanban worked in our case because we had better focus.  We were able to keep many of the stories as small, discrete pieces of work—no one story choked the backlog and stories didn’t need to be completed in groups. The business had the flexibility to change the backlog of any story not on the kanban board, which meant the process allowed the business to keep up with the development team.  The business focused on fewer stories, meaning that the stories they did focus on were of much higher quality. There were only two critical stories in the product backlog at any one time (limited by WIP), which enabled a fluid prioritization of the rest of backlog. The business team, was able to focus only the top few of the MWL, leading to better time management and clearer stories, which they determined the content of.

Along with the effect on the business, kanban has a number of other advantages that worked in our favor:

• Making Scrum lean was key for us. The development team didn’t spend any time in the planning meeting discussing stories that were never pulled into the workflow.  Only the next story on the kanban board was analyzed.
• Stories were higher quality. The business team only focused on the very top stories—the high visibility made it painfully obvious if this were not the case. Better stories lead to better development because of the clear business value.
• Stories prioritized to the top of the product backlog could be worked in a relatively short time i.e., as soon as a group of developers freed up. The temptation to “stop the press” and start on a critical story was much less when the expectation was that a story could be worked on within a few days.
• Our demos were at the story level and held immediately when a story was completed, leading to a much more focused demo with a smaller audience and less preparation time.  Team members were not distracted with the end of the sprint push or focus on demo as they were in Scrum.  Fortunately, the business team was engaged and those who had a stake in the stories attended the demos.
• The developers had small stories that they could complete quickly, leading to a good sense of accomplishment and a chance to react quickly to lessons learned from a story.  The small stories were a direct result of the business’s having enough time to analyze the stories and the limited WIP, which made the negative effect of large stories highly visible by stalling the kanban flow.
• There was no slack time at the end of the sprint.  Often in Scrum when a series of stories in a sprint are completed before the end sprint, the team might slow down, the team might gold-plate because it has time, or the team might decide to work on technical debt. With kanban, there is no timebox—work begins immediately on the next story, so there is less potential for waste.
• Fewer people participated in meetings, which was both an advantage and a disadvantage.  For example, the planning sessions only had the people working on the story in that meeting.  The wisdom of the group was lost, but the productivity time was gained.  Similarly with the demo:  Only those directly involved in the story attended, meaning knowledge transfer was lost.
• We had an experienced, cohesive team that could estimate accurately and didn’t have a significant learning curve.

Kanban does not come without a number of challenges, though. 

• With teams forming and reforming on a story-by-story basis, there was a danger of team cohesion being lost. Fortunately, we did not experience cohesion problems because most team members had been together for a long time.
• Several Scrum measurements were lost: Velocity, burn down, and burn up were not meaningful since there was no timebox. It was hard to gauge productivity and see where gains could be made. Kanban uses cycle time to measure throughput.
• Stories had to be small.  A large story could easily stall the project, negating the positive effects of kanban. The right size (about 5 story points) was only determined through trial and error.  While small stories are important with Scrum, the kanban process made the deleterious effects of large stories highly visible—the flow stagnated.
• It was hard to project when a certain story not on the kanban board would be completed. New stories would be added to the product backlog and priorities would shuffle frequently.

Closing Thoughts
Kanban seems like a productivity boost for Scrum: It increased our productivity by over 300 percent and gave us the chance to react quickly to business changes.  But Kanban is not without its own set of problems. Scrum’s benefits, such as sprint planning, release planning, a sense of rhythm takt, and velocity are either lost or reduced with kanban. Collaborative design is part of Scrum but not emphasized in kanban, since in our case a subteam only worked on a particular story.  Additionally, the teamlet concept can lead to a certain level of lost collaborative design, knowledge sharing, and idea cross-pollination.

Overall, kanban is worth trying in the right circumstances: small, loosely coupled stories with an experienced agile team. Industry reports (Kniberg and Banks among others) indicate that Kanban works best in this circumstance. Kanban isn’t a silver bullet; it should be applied carefully lest the process devolves into chaos. If you want to learn more, read Corey Ladas’ book Scrumban or David Anderson’s book on Kanban.

I’d like to thank Linda Cook, Chris Farrell, and Johanna Rothman for their wonderful insights and assistance with this article.