Merging Defect Fixes into the Development Codeline

No matter what type of software you develop, from internal IT applications to web applications to commercial software packages, fixing bugs following a release is part of the software development lifecycle. If your software development process is mature you carefully control the changes made to released code. If your software application is of modest size and complexity you probably have a separate team devoted to maintenance. Realistically, software maintenance can be a project by itself. However, the process of merging production fixes into the development codeline is an equally significant effort. Unfortunately, the effort required to merge production fixes into the development codeline can be grossly underestimated and improperly staffed even by organizations that practice otherwise good software development.

Terminology
Before proceeding, it is important to review common terms used when discussing the process of merging defect fixes into the development codeline. First, the term codeline refers to the sequence, over time, of the file versions that comprise the software application under development.

If a software development organization keeps ongoing development along a single codeline, it is following a mainline. The mainline does not need to be the trunk of the software configuration management (SCM) tool. Instead, the mainline is a home codeline that is used for all development activities except those special circumstances that require the use of a branch.¹

When the mainline is kept active (with check-ins) to reflect the current state of the development process, it is referred to as the active development line.² The active development line is named such because its use is governed by a codeline policy that allows frequent check-ins, perhaps at the expense of some stability. The code checked in to the active development line does not need to pass exhaustive integration tests, but it needs to be stable enough to be used for day to day development in a dynamic environment.

Since most software applications (including web applications) are developed as successive releases, it is necessary to separate maintenance of a release in production from ongoing development work for the next release. A branch, called the release line, is created off of the active development line at the time a release ships. The release line is used to support maintenance on the production release. Specifically, the release line allows a development organization to perform maintenance work on the released version of the software application without interfering with ongoing development for the next release.³ This is necessary since ongoing work in the active development line is often incompatible with the maintenance required on the release in production.

The Need to Merge and the Cost of Inadequate Planning
Eventually, the release in production will be replaced by the release under development. However, before the release under development can be placed into production, the bug fixes made along the release line must be merged into the active development line. The process of merging bug fixes into the active development line is often time consuming and non-trivial.

When planning the release of a software application, the project manager needs to consider the time and resources required to merge defect fixes from the release line into the active development line. Too often the task of merging defect fixes from the release line into the active development line is pushed to the tail end of the development schedule. Too often the task of merging defect fixes from the release line into the active development line is assigned to those least qualified to perform the task.

The cost of underestimating and improperly staffing the effort to merge defect fixes from the release line into the active development line can be very high. This cost might appear as

• long hours and staff burnout near the end of the release,
• a slipped schedule because of the need for more time to complete merges,
• many new bugs that surface in the active development line following a large merge,
• or bugs fixed in the production release that reappear once a new release is shipped.

All of these costs are painful to the project. However, when previously fixed bugs reappear it is a visible sign to customers and end users that the quality of the delivered software is regressing. Perhaps, there is nothing more frustrating to an end user than having to deal with a previously fixed defect that reappears in a new software release.

Planning the Merging Effort
The problems associated with merging defect fixes from the release line into the active development line can be mitigated with proper planning. The first step in planning is to estimate sufficient time in the project schedule for merging, and then schedule project tasks accordingly. Clearly, the best estimate can come from past experience in a similar environment.  The project manager who records metrics on the level of effort required to merge each production fix during one release will have some data to estimate the potential effort required for these tasks on the next release.

Lacking metrics, the project manager can estimate the time required to merge each production fix to be equivalent to the time required to implement a new feature in the current release. While this guideline may seem excessive, keep in mind that merging a defect fix from the release line into the active development line may not be straightforward. It may require far more than simply copying lines of code. In the worst case, merging a defect fix from the release line into the active development line may require reworking significant logic. This is likely to be the case if the code modified to fix the defect was also modified as part of ongoing development activities. The code changes required to fix the defect in production need to be reconciled with the code changes made to implement new functionality.

To determine the number of defect fixes to merge into the current release, the best estimate may come from project metrics. Since the number of defects to be fixed in the release line (while you are developing in the active development line) is not known before the start of your release, the best estimate is likely to be determined, in part, by the number of bugs fixed in prior production releases. If the characteristics of each release (e.g., the scope and difficulty of the work, and the size and caliber of the staff) vary greatly this metric may not be useful in planning.

Staffing the Merging Effort
Equally important in planning the effort required to merge defect fixes is identifying and allocating the resources qualified to perform the merge. In their paper, High-level Best Practices in Software Configuration Management, Laura Wingerd and Christopher Seiwald state,

"The burden of change propagation can be lightened by assigning the responsibility to the engineer best prepared to resolve file conflicts. Changes can be propagated by (a) the owner of the target files, (b) the person who made the original changes, or (c) someone else. Either (a) or (b) will do a better job than (c)."⁴

This point cannot be emphasized enough. Even if the team member performing the merge is skilled with the configuration management tool and experienced with the programming language used for the software application, the ability to merge changes properly requires knowledge of the code being changed and an understanding of the reasons for the change. Too often the team member assigned the task of merging defect fixes does not have sufficient understanding of the code being changed. The team member making the change can either guess at the intent of the developers who worked on the code impacted by the merge or consult with these developers. Either approach is less optimal than having the developer who best understands the code merge the defect fix.

Better than having one developer perform the merge is involving both the developer who made the fix in the release line and the developer(s) most familiar with that same part of the code base in the active development line. The developer who made the fix in the release line can explain what changes were made to fix the problem and why these changes were made. The developer(s) working in the active development line, in the same area of the code where the changes were made in the release line, can assess how to best propagate the defect fix.

Having a developer working in the release line partner with a developer working in the active development line may seem like a staff intensive way to execute a merge. However, when the developers who best understand both parts of the merge (i.e., the changes made in the release line and the code to be changed in the active development line) work together, there is a far better chance that the merge will proceed smoothly.

Executing the Merging Effort
The effort to merge defect fixes from the release line into the active development line does not need to occur as one big activity prior to deployment. In fact, this is not the ideal approach. It is far better to perform smaller merges over the course of the development lifecycle. Integrating a few small changes over time is almost always less painful than integrating several changes at once.

In their paper, High-level Best Practices in Software Configuration Management, Laura Wingerd and Christopher Seiwald state,

"Propagate early and often. When it's feasible to propagate a change from one branch to another (that is, if the change wouldn't violate the target branch's policy), do it sooner rather than later. Postponed and batched change propagations can result in stunningly complex file merges."⁵

By attempting to merge production fixes frequently, you may encounter fixes that do not require a merge into the active development line. This will occur if the functionality fixed in the release line no longer exists in the active development line. It will also occur if the functionality in the active development line has been reworked in a way that removed the original bug.

Testing Following the Merging Effort
The testing that follows a merge into the active development line should be no different than the testing that follows any change made to the active development line. Regression tests should be executed following each merge to ensure that the merge did not break existing functionality. Without regression testing, the project runs the risk of introducing bugs as a result of merging a defect fix from the release line into the active development line.

Using Tasks to Facilitate Merging
Fortunately, the job of merging changes from the release line into the active development line can be made easier if all changes made to the release line are grouped by task. Most popular SCM tools support the concept of a task. The task allows check-ins to be grouped according a meaningful unit of work, such as a defect fix. Using your SCM tool's task feature when checking in code changes made to fix production defects can help you when you need to merge these code changes from the release line into the active development line. By using tasks, you work in terms of the high-level defect fixes that need to be merged, rather than the low-level file changes that need to be merged.

The table below identifies the task feature in several popular SCM tools. This table was populated primarily with the information from Appendix B of Software Configuration Management Patterns: Effective Teamwork, Practical Integration.⁶

To use tasks effectively with any SCM tool, developers should follow the practice of task level commit. The concept of task level commit means a developer keeps each commit to the SCM tool to one small-grained, consistent task.⁷ Task level commit is ideal for defect fixes made in the release line because it can isolate each simple bug fix to a single task and a single commit. A more involved defect fix may need to be grouped into several small grained tasks. However, under no circumstances should multiple defect fixes be lumped into a single task. Doing so would defeat the purpose of using tasks to group together the code changes needed to implement a defect fix.

Conclusion
Merging defect fixes into the development codeline need not be a project nightmare. With adequate planning, the right staff assigned to the job, and the use of an SCM tool's task management feature, the merge process can be uneventful.

¹ Berczuk, Stephen P. and Brad Appleton. 2002. Software Configuration Management Patterns: Effective Teamwork, Practical Integration, page 54. Addison-Wesley.
² Berczuk, Stephen P. and Brad Appleton. 2002. Software Configuration Management Patterns: Effective Teamwork, Practical Integration, page 60. Addison-Wesley.
³ Berczuk, Stephen P. and Brad Appleton. 2002. Software Configuration Management Patterns: Effective Teamwork, Practical Integration, page 148. Addison-Wesley.
⁴ Wingerd, Laura and Christopher Seiwald. High-level Best Practices in Software Configuration Management
⁵ Wingerd, Laura and Christopher Seiwald. High-level Best Practices in Software Configuration Management
⁶ Berczuk, Stephen P. and Brad Appleton. 2002. Software Configuration Management Patterns: Effective Teamwork, Practical Integration, pages 171-194. Addison-Wesley.
⁷ Berczuk, Stephen P. and Brad Appleton. 2002. Software Configuration Management Patterns: Effective Teamwork, Practical Integration, page 113. Addison-Wesley.