An Agile Approach to Release Management

Agile teams work under the same constraints as other software development teams.  They must address issues of maintenance and support, the need for external artifacts like documentation and training, and the reality that not every customer can accept change at the rate that an agile team can (in theory) deliver it. To attain the goal of a shippable product at the end of every iteration, an agile team must manage the flow of change from a customer while maintaining discipline and good engineering practices.

We will discuss how release management works in an agile project and also explain how even non-agile teams can benefit from applying aspects of an agile approach so that value can be delivered to customers in a more predictable fashion.



Agile Principles 



Agile project management practices enable you to manage schedule risk in a project. In a sense, many are simply good practice:

• Plan to work in time-boxed iterations of 2-4 weeks
• Maintain a backlog of feature requests in prioritized order, and revisit the priority at each iteration boundary
• At the start of an iteration select the highest value items from the backlog, detailed planning for these items
• Integrate often
• Verify often
• Deliver and review against a plan

These practices affect how one thinks about release management, as it is a planning activity and any planning activity has uncertainty that increases the distance you are from the present. The goal of having features complete or not at the end of each iteration allows the user to determine what will be available to ship. The prioritization of the backlog allows you to compensate for problems by having the most important features developed first, so that you can still meet a release date should that matter for market or regulatory reasons. The always shippable goal means that you can, if need be, accelerate your release schedule.



An agile approach enables better release planning by combining planning discipline, which helps to focus on the highest value work, and also engineering (including SCM) discipline, which helps to identify and fix problems early, giving you more predictability. Agile practices make shipping a release a decision that product owners can make without worrying if the team will meet a date far in the future. The role of the agile team is to enable allow the product owner to make the decision on short notice if need be.



Realities 



The IT Process Institute's report on change configuration and release performance study says that:

• Release trumps change:  Rigorous release build, testing and rollback practices have broad impact on individual performance measures and overall performance. Change tracking and change oversight practices are necessary but not sufficient to achieve performance improvement on their own.
• Process discipline matters:  There are no change, configuration and release silver bullets. Building a process-focused culture and monitoring and responding to process exceptions predict top levels of performance more than the many of the industry recognized best practices in these areas.

This is very much in line with an agile approach to software development. While the focus of much of the literature on agile methods is about single codeline development, release management often refers not merely to managing a single release, but rather the overall discipline and methods of managing multiple releases at the same time. There are all sorts of valid business reasons why one might need to support multiple releases. Maintaining multiple codelines doesn't always mean you aren’t being
agile, particularly if each one of those is generating (or preventing the loss of) additional revenue.



While it may seem that having only a single stream of development is very restrictive, in practice it can work surprisingly well.  That generally means having to deal with a change or a fix where you first have to determine which releases need it, and then how to inject it into (propagate to) possibly multiple release streams. For these cases, agile teams can use a traditional release line approach with an agile twist.



An agile process also makes it easier to manage the disruptions to day-to-day development caused by bug fixes. The key to managing multiple release streams in an agile environment is to prioritize all the work for a team together. The product owner needs to decide how important a bug fix is compared to a feature. Agile planning techniques make the cost obvious. All teams, not just agile ones, work best in an environment where change is managed. When additional work is introduced mid-iteration, the performance of the team can suffer. The iteration approach of agile
methods gives the product owner a choice when an issue presents itself: Do I fix this immediately and forego feature work or do I wait until the next iteration (2-4 weeks)?

Laura Wingerd discusses some of the basic rules of the road for "channeling the flow of change in software development collaboration" in her book, Practical Perforce. She discusses the "flow of change rules" and the "tofu scale". The tofu scale and change-flow rules/protocol discuss the relationships between codeline policies across the entire branching structure when it comes to making decisions about stability versus speed. Though one codeline's policy might make a certain tradeoff, it is the presence of multiplecodelines that determine how they work together. 

These change-flow rules, which also relate to the mainline pattern, advise us on exactly how and when to do this. The change-flow rules are also well aligned with lean principles of minimizing the form of waste known as inventory or work in progress.  This is functionality that has
been developed, but not yet released.
We have had experiences where applying an agile approach helped manage the
flow of bug fixes.

One team commented that their agile method involved a 2 weekly release cycle and they didn't need to rush bug fixes out.  It had always been OK to wait for the next release. A key factor in the success of their development method was the trust that the rest of the organization had in the process.  Work was completed then released and, when a feature was scheduled to go into a release, it almost invariably did.


While agile teams pride themselves on their ability to be responsive, being agile does not mean being chaotic and undisciplined. Change has its cost, and agile methods provide ways of making the cost of change explicit. An agile project works best when there is some sort of rhythm for release cycles. Whether or not to ship is a business decision, which is how it should be. Even though the codeline should always be potentially shippable throughout an iteration, the decision to ship or not occurs at the end of an iteration. This is important as it negates rumors that agile processes are chaotic. Releases must be planned and iterations must align or you risk hurting the efficiency of the team. 



Agile Release Management Enablers



The key enabling patterns for an agile project are:

• Private build
• Integration build
• Release build
• Continuous integration, with automated tests
• Release line
• Release-prep codeline
• Active development line

A private build enables all team members to test any changes they make with some degree of confidence that they will not break the codeline. As part of the private build, appropriate tests are run that give some degree of confidence that the code works. Because of the private build pattern, team members can commit code often and, therefore, have only small integration issues. Once changes are committed, an integration build serves as a gatekeeper for more exhaustive verification.

The integration build is run on a build server that may be using some sort of continuous integration tool such as Anthill, Cruise Control or Team City. Since the exhaustive testing is run asynchronously from the development team, tests that you expect will pass will not delay commits.

An agile team often has a discipline that a failed integration build is an “all hands" event, thus ensuring that integration problems are addressed quickly.
 The combination of small commits with frequent integration means that the cost of change is small and enables rapid development.



A release build and release line may seem like odd beasts to mention in an agile context. Release builds typically have a few extra requirements that aren't strictly necessary for all integration builds.  Release lines are structured to have policies that limit change because you want to be extra cautious with released software.

On the other hand, having a release line frees the active development line to move forward and risk mistakes. Likewise, moving almost ready to ship code to a release prep codeline means that new feature work can move forward, while still enabling the team to address integration issues that may come up in final testing. While an ideal agile team will not use a release prep codeline, it is a very useful pattern in practice.

Also the agile mindset is to think about what is possible, and decide whether or not to actually do something based on cost and relative value. Not to attempt improvements because the changes seem unattainable is an obstacle to improving your release process

Most any team can benefit from incorporating some of what we call Agile SCM Practices into their release management plan.



Common Questions About Agile Releases 


Like many processes, an agile approach to release management raises some questions. Here are two common questions, along with answers:

• Question:  What does “deliver at the end of each iteration” really mean? Answer: Being able to deploy or deliver working software at the end of each iteration is an agile principle. It is also a principle that many projects, agile and not, seem to treat as unreasonable. Let's talk about how you can think about this practice in a way that makes practical sense, and addresses some common concerns people have about this idea.
• Question:  What if my customer can't accept changes every 2 or 4 weeks? Answer:  The fact that you have working software does not mean that you need to actually deploy it into production. The benefit of having software in a working state is that your can make the release decision at each iteration, as business needs dictate.



Documentation, Integration Testing and the Meaning of “Done
”

Many organizations worry that they can't have a fully shippable release with each iteration. One thing your team needs to decide on when adopting an agile process is what it means for a feature to be “complete” for an iteration. When you are using a non-agile process, “done" may mean that you have a complete distribution package. This is an excellent goal for an agile team and it is attainable, but starting out with this as your done threshold may end up frustrating you.

Agile practices are an excellent tool for risk management, so consider what the riskiest parts of your application are.  Is it documentation or is it the software?

A perfectly valid definition of complete software is:

• All visible features complete and stable
• All in progress features implemented in a way that does not break existing functionality and which can be hidden if needed.

There are also ways that you can get closer to a completed documentation set. While certain things, such as final screen shots, may not be able to be done until a product is finished there is much that you can do, such as the basics of online help, overview documentation, etc. Part of the challenge of adopting an agile approach is adopting an agile attitude. The usual question one asks is to emphasize what is impossible.  An agile attitude is to ask what is possible.If your product lifecycle permits, you can dedicate one iteration to completing the artifacts necessary for shipping the application and performing release stabilization if needed. A stabilization iteration is a short iteration focused on final integration testing, documentation, and the like.

While not strictly part of an agile process (since you should be doing all of these things as you go), a stabilization iteration is a useful tool when you are transitioning to agile or when you have a demand for extensive supporting activities
that need to be done after the bulk of the feature work is done. You would do your stabilization work based off of a release prep codeline. 



Another common concern is how to address a need for complicated manual integration testing. An agile approach is pragmatic and also highlights potential problems with your architecture and development process. Teams have a need for complicated (and sometimes) manual integration testing because their architecture is
complicated and they have not developed their code in a way that enables automated testing. Strive to have as much automated integration testing as possible during your iteration. After, use a stabilization iteration approach to do any remaining required testing.

The ITIL framework in its new V3 release now covers release management as part of the service transition book. One of the factors for many organizations is that the process of releasing requires handing over to operations, and integration with existing live systems and services. In such circumstances, documentation and ongoing maintainability are very important.

A related issue is that in many organizations are starting to develop a totally new application. You are, instead, enhancing an existing application and delivering some extra functionality. Many project teams spend the first part of their development trying to understand the existing system and its interfaces and dependencies before being able to reliably change it.  The project team then disbands and a new one comes along.  The whole wasteful process is then repeated.

In these situations, the appropriate level of documentation is vital. One advantage of development methods that use a good amount of automation around testing (unit tests, automated system and acceptance tests) is that, as long as the tests are maintained, they provide a very real source of documentation.  They also provide a safety net for future changes are able to prove that new functionality has been added without breaking existing functionality.


Summary

Agility is about delivering value predictably. Taking an agile approach to your Release Planning process will help you meet the goals in your release plan and allow you to be more confident in your commitments.