Program planners in IT organizations have a dilemma: On one hand, their agile teams tell them that if requirements are defined up front, agile teams cannot operate; but on the other hand, the program’s budget and scope need to be defined so that resources can be allocated and contracts can be written for the work. How does one reconcile these conflicting demands?
Program planners in IT organizations have a dilemma: On one hand, their agile teams tell them that if requirements are defined up front, agile teams cannot operate; but on the other hand, the program’s budget and scope need to be defined so that resources can be allocated and contracts can be written for the work. How does one reconcile these conflicting demands?
Agile processes suffer greatly if requirements are tightly defined ahead of time. At its core, agile is built around having continual discussions with the customer to elicit requirements; those discussions transfer knowledge to the team. Rather than relying on requirements documents, which are the primary method of communicating to the developers in waterfall projects, agile projects rely on in-person communication. If you give an agile team a requirements document, they will have to spend months deconstructing it and learning it, and by then their entire release timeline will have expired. If you have a requirements document, you might as well not make your project agile at all.
But agile projects do need some level of up-front requirements—in the form of a release roadmap. A release roadmap is a high-level timeline that lists the business capabilities to be attempted for each release over time. When an agile team begins work on a release, the first step is to decompose the release’s planned capabilities into high-level “features” or “epics.” These are later broken down into more granular “stories” that can be completed during the iteration. This progressive elaboration of requirements is referred to as rolling wave planning.
Rolling wave planning is a challenge for program planners because traditional program planning rests on the assumption that one can create a detailed plan and cost estimate for the work to be done. For software development, this assumption is false.
The root issue here is not so much an agile issue as it is a software development program planning issue. It is mostly a settled matter that large IT projects are risky and challenging, and the best way to deal with large software development requirements is to decompose them into many smaller, more tractable—and largely independent—efforts. For example, a 2010 Department of Defense report to Congress stated (emphasis added):
A major change in the new process will be moving from large multi-year programs to portfolios of short-duration projects. This requires a new approach to project oversight. This approach will place more accountability on timely coordination, quicker decision making, and increased stakeholder involvement through more frequent performance-based in-process reviews.
Agile has been looked to as a solution for the challenge of large software projects. However, using agile on software teams does not magically fix large IT programs—as we saw with HealthCare.gov. According to an article in The New Yorker, “Healthcare.gov involved fifty-five different contractors that each delivered a piece of the final system to [the Centers for Medicare and Medicaid Services] for integration and testing. Whatever development processes those contractors used internally, the overall project was run according to waterfall principles.”
What Works
The problem is that agile methodologies like Scrum do not say how to manage a program. To figure that out, one has to go back to the first principles of the Agile Manifesto. If traditional program management is out, then what replaces it?
In the spirit of agile, we need something more tactical, local, and accessible—something that, to use the phrasing of the above mentioned Department of Defense report, will “place more accountability on timely coordination, quicker decision making, and increased stakeholder involvement.” That might take the form of a standing committee of accessible people who consider cross-team issues on a continuing basis, much like a “war room” approach, rather than a traditional oversight approach that relies on quarterly reports of progress. I have helped to implement such a tactical process, in which cross team issues are addressed in a weekly program level meeting. One of the primary dashboard tools that proved to be highly effective was a chart that showed the intersection points across all the teams, as shown in figure 1.
First of all, let’s be clear that program management has many steps to it, including (1) deciding the scope of work, (2) defining and launching work units in the form of releases or projects, (3) overseeing the work units as they proceed, including measuring progress and identifying issues, and (4) revising the plan by going back to step #1 continually. In this article I am only going to discuss #3, considering the other topics warrant articles by themselves.
Figure 1 shows a swim lane for each of four teams, and each team has a backlog of business capabilities to work on. Each team divides its work up into features and, ultimately, stories. These subdivisions are not shown in the figure, however, because they are not a concern for the purpose of figure 1, which is to depict the interdependencies across the teams.
As you can see, Team 1 has some critical events it depends on: “Need ICA [interface control agreement] from Team 3” and “Need to start testing inter-system bridge.” Because the latter is on the chart, this item must be dependent on some activity that is external to Team 1. The purpose of this chart is to enable program leadership to see at a glance all the things that are beyond Team 1’s control that might cause delay, and to see the potential impact on the primary milestones: the deployments. This chart enables program management to maintain a forward view of potential sources of delay, monitor them, and step in to address impediments to prevent those delays—or to reshuffle work.
Reshuffling work entails examining the work backlog of each team. In figure 1 that backlog is listed at a capability level, and that is usually sufficient for program leadership. If program leaders need to drill down into a more granular level, they can examine the feature backlog for each team.
Remember that the whole point here is to shift from large multi-year programs to portfolios of short-duration projects—in other words, to have some decoupling among these separate efforts so that work can be shifted around as needed, rather than being rigidly contained within a complex program that is planned ahead of time.
In order to assess whether milestones will be met, program management should use what are now traditional agile tools such as feature burn-up and cumulative flow, available with all of the major agile planning tools.
I have found that a Scrum of Scrums is highly effective for resolving impediments that teams can handle among themselves, but for addressing impediments that require management intervention, a separate forum is needed for management to track and discuss the major issues that bubble up to them, as well as to track overall progress so that they are not caught off guard.
Conclusion
Agile programs need lightweight, fast-response program management methods. The traditional approach of a hands-off review of plans and a quarterly review of progress is wholly insufficient. Program leadership needs to actively track team progress and dependencies across teams as well as external dependencies to foresee problems and be ready to step in to resolve issues with minimal delay. Program management also needs to be accessible to teams so there can be rapid, bidirectional communication about issues that pertain to milestones, cross-team dependencies, and team-level impediments.
