When time is short, even a well-disciplined team may cut a few corners. They might implement a code solution or automated tests in a less-than-optimal design, or make changes in old production or test code without taking time to refactor it. Future changes to this production or test code will be harder and take longer.
Ward Cunningham coined the term “technical debt” to represent the unfinished work due to shortcuts taken by the team to deliver a feature or user story faster. If we “borrowed” by skipping important steps, we have to pay back that “interest” quickly. If not, the interest will keep mounting up, and the growing technical debt burden will slow us down or even immobilize us.
Technical Debt Sprints
One way to pay down our technical debt is by devoting an entire iteration to activities such as refactoring production and test code to make it more maintainable and understandable, learning new ways of working better, upgrading test and code framework and tool versions, and improving the “living documentation” (also known as “automated tests”). We skip delivering new business value for just that iteration so we can pay down our technical debt.
“Sounds good,” you say, “but our product owner is never going to let us spend a whole iteration and not deliver new user stories to the business.” Or, you might be thinking, “Aren’t we supposed to ruthlessly refactor all the time? Why spend a whole sprint on refactoring?”
Even a highly disciplined team may run into limits to continual refactoring. If we have a lot of legacy code that is not supported with automated tests, it’s time-consuming to refactor and too risky if we don’t simultaneously add the tests. We also need to refactor our test code to keep tests maintainable and efficient. But, it can be risky to change our tests while code is changing.
There are other considerations. The team should choose its strategy. The work done in one quarter or a six-month period is going to be the same whether the team’s technical debt keeps growing—forcing them to work more slowly—or they devote some time exclusively to reducing the technical debt. Even with continuous refactoring, we find it works better to take some time to understand the debt and take the right steps to manage it for the long term. (See the sidebar for a helpful game about technical debt.)
Article continues below.
Technical Debt: Hard Choices Nanda recently came across a game that explains technical debt and the results of not paying it back. The game, called Hard Choices, was designed by the Software Engineering Institute (SEI) and adapted from Shortcut: A Game About Speed and Risk by Elisabeth Hendrickson. Basically, the game has the aspects of taking shortcuts (creating debt by incurring penalties) and paying it back (by losing a turn). If you don’t pay your debt, it will take you longer to reach the end. Players can move in any direction and can even change directions in a single turn. They earn points by racking up “tool” cards and by being the first one to get to the end. For each bridge they cross, they must subtract one from subsequent rolls of the die. Nanda plays differently than described in the SEI rules. In his version, three players still move the number of squares to match the roll of the die; however, there are specific rules for each player:
Nanda also changes the positions of the bridges so that each shortcut only bypasses a few squares, thereby making even clearer how Player 1 is going to suffer at the end. This is more like real life—shortcuts don’t usually buy you a huge amount of time. Played by these rules, Player 2 wins and Player 3 usually reaches the end ahead of Player 1. The “Short Cut” game has similar rules, and several variations that let players experiment with different approaches to paying penalties back and changing goals from a competition to a collaborative team effort. Playing either game helps teams and their managers understand the need for technical debt sprints to “keep the backyard clean.” At a minimum, this will allow the team to maintain its velocity, and there might even be a big improvement. |
What Should You Do in a Technical Debt Sprint?
Let’s step into our product owner’s shoes and think about what she would expect from and after a technical debt sprint. A steady velocity allows the business to plan effectively. They have a good idea of how much work will be done in the next few iterations. Increasing velocity would be even better, if the team can maintain the higher level. Let’s look at some activities the team should do in technical debt sprints to achieve this goal.
Code Cleanup
Maybe you’ve left some unused code in the system, or an unused database column is lying around. Perhaps you really want to refactor some code that is bothering you. No doubt you can come up with a long list of pain points—things that get in your way any time you work in a particular part of the code.
The same rules apply to testing debt. Test-related activities in a technical debt sprint include reorganizing and refactoring automated tests, removing unused tests, and making any test changes associated with code changes. Your goal is to remove most of these pain points, resulting in cleaner and more maintainable code by the end of the sprint.
Here are some items our team chose to work on in a recent technical debt sprint:
- Refactor enums to have consistent lookup() method. In some enums, it was spelled as lookUp().
- Delete unused legacy code. Since this code isn’t covered with automated tests, we weren’t 100% sure if it was used. We were able to remove 3,500 lines of code. One thousand lines were in the last piece of legacy code, which we were able to remove from a module.
- Refactor automated API-level and GUI-level automated tests, extracting duplication and making tests easier to understand.
- Remove an unused database column from a table.
- Rename a database column in several tables and in the code for consistency.
Software Upgrades
Technical debt sprints are well suited to upgrading your third-party software, including test tools. This lets you start using the latest and greatest features of that software. If you delay upgrading software for too long, you may find yourself stuck on a version that’s no longer supported. You need to make sure that all dependent software is upgraded together, which can take a lot of time.
Build Features That Reduce Supporting Activities
The time spent doing production support is time lost for developing new features. Identify little features that will reduce support time, thus increasing velocity, and implement those during the technical debt sprint. Some of the small features our team delivered in the most recent technical debt sprint were:
- Users didn’t have any way of knowing when a long-running daily process completed, so a developer had to monitor this for them. We changed the process to send an email to the users when it completes.
- Some reports ran extremely slowly, causing users to file production support tickets. The developers and DBA tuned these.
Do You Have Enough Time?
Now you know what tasks you could work on in your technical debt sprint. But, you only have one iteration. How do you maximize your return on investment (ROI)? Some tasks may be mandatory. For example, you may be using an unsupported version of third-party software and you can’t delay upgrading it any longer. For the rest of the tasks, consider the following aspects:
Utility, or Value: What’s the value of a new feature to address the root cause of a common production support issue? Is it in a framework used throughout the system, or will it only affect a small area? How much support time will it save?
Effort: How difficult is the task? Some tasks only take a few hours, while others might take the whole sprint. A general guideline is to prioritize tasks in the following sequence:
- Easy and high value.
- Hard and high value.
- Easy and low value.
- Hard and low value.
What About the Testers?
What do testers do during a refactoring sprint? Obviously, someone has to verify that the upgrades and refactorings didn’t break anything, and any new features must be tested. In some ways, a technical debt sprint is business as usual for the testers on the team. Still, these sprints provide time for trying out new test tools and learning new techniques. Programmers, testers, DBAs, system administrators, and others on the team need to plan together and ensure that technical debt is addressed adequately in both tests and code.
On our team, testers take advantage of these sprints to upgrade to the latest version of our test frameworks and drivers. Here are some other tasks we’ve done during technical debt sprints:
- Split automated test suites into multiple jobs in our continuous build process, letting them run in parallel so we get the feedback more quickly.
- Put our FitNesse tests under source code control with the rest of the production and test code. This allowed us to track changes to test pages and tie the tests to the matching production code build version.
- Install and learn how to use an integrated development environment to create and maintain automated tests.
Testers often pair with a programmer or a system administrator for these types of activities.
If they have good coding and design skills, testers may be able to refactor automated tests on their own. No matter what their technical skill level, testers can pair with programmers to refactor test code. Going forward, new tests will be much easier to add, taking advantage of libraries of test components.
Plan Your Technical Debt Sprint
In normal iterations, spend time on each user story to refactor production and test code, leaving it cleaner than you found it. If a task needed to reduce technical debt seems too big or too risky to try in a normal iteration where your team must deliver new business value, write it on a card and keep a backlog of “refactoring” cards.
Just prior to your technical debt sprint or at the start of it, read through all the refactoring cards together. When our team does this, we put on the board any tasks we think we might do. We don’t bother to estimate them; we just choose what we feel are the right tasks to work on and keep going until we run out of time. Tasks we don’t get to are saved until the next technical debt sprint.
At the end of the sprint, you don’t have anything new to show your customers, unless you implemented some small features to reduce time spent on production support. However, you can celebrate the lines of unused code you deleted, the mis-named database columns you renamed, and the refactored automated tests that allow you to make future changes in only one method or module rather than in many individual tests. It might be useful to quantify your technical debt as a dollar amount and track your progress.
With your technical debt load significantly lightened, your team will enjoy focusing on delivering new business value much more easily than before. Your business stakeholders will be delighted by the renewal of a steady or even increasing velocity.
References
- “The Financial Implications of Technical Debt,” Jim Highsmith, 2009
- “SpamCast 112—Israel Gat, Technical Debt,” December 2010.
- Cutter IT Journal issue on Technical Debt
- Managing Software Debt—Building for Inevitable Change by Chris Sterling, Addison-Wesley, 2010
User Comments
Excellent article!
I've been in this industry since the top of my head was covered with hair. You've uncovered a disfunctional pattern, to rush into the next project NOW.
Sometimes we are under time pressure and in the trenches where the goal is to do what must be done to ship. That happens, that's part of our jobs.
You have highlighted that we should make the time so we don't repeat the same mistakes and carry ever increasing technical debt.
Mick
Do any of the following lead you to reconsider your definition of Technical Debt at all?:
"Agile and Beyond :: The Technical Debt Trap" http://www.slideshare.net/DocOnDev/agile-and-beyond-the-technical-debt-trap
"Debt Metaphor is not Technical Debt": http://ponderingobjectorienteddesign.blogspot.com/2009/09/debt-metaphor-...
"Ward Explains Debt Metaphor": http://c2.com/cgi/wiki?WardExplainsDebtMetaphor
"Technical Debt": http://xprogramming.com/blog/technical-debt-2/
"The real Meaning of Technical Debt": http://matteo.vaccari.name/blog/archives/590
Hi Philip, I have read/watched most of those before. What specifically in our article do you find in conflict with them?
-- Lisa
If you think of Technical Debt (TD)reduction as an accumulation of product backlog items (usually unrecognized, un-prioritized, and missing as items visible in the backlog) then a TD sprint is nothing more than the prioritization of some of those items to the exclusion of all other customer focused items for the duration of a sprint.
Since TD reduction provides value to the organization (via improved development performance) then making TD reduction stories part of the visible product backlog creates a context to schedule their execution into sprints on an ongoing basis rather than (or in addition to) big bangs. The variable then simply becomes one of how much capacity to allocate to work them per sprint.
Think continuous flow of value; If you really want to see some ongoing improvement, add TD reduction stories to the backlog and make it a part of regular project / sprint planning.
J. Packlick
Agile Coach
Travelocity
Hi Jay,
If the TD is sizable and could be defined as a story, we could add it to backlog and wait for it to be prioritized. We did that lot of times. However, if you have lot of smaller granularity tasks, it would be hard to negotiate with the PO in each sprint. It would be lot easier to ask for a week or two to take care of TD and let engineers decide what's the best priority rather than asking PO to prioritize TD items.
As we always say, do whatever works in your context.