Change Management Is Essential to the Building Processes

Types of Builds

We can classify builds into three distinct categories: release, integration and developer. They might be loosely categorized as:

·         Release:  Existing build record (i.e., snapshot) used (as authorized)

·         Integration:  Repository and change status centric (typically nightly)

·         Developer:  Workspace centric (repeatedly as required)

Release Builds
A release build is one that is done for a formal or candidate release of the build outside of development. This may be for formal verification, for field trial, beta trial, or production. It differs from the other two builds in that there already exists a snapshot of what is to be built in the CM repository. You specify which snapshot you wish to build and then, using only data from your CM repository, your build tools/process produce the required outputs.

Integration Builds
An integration build is similar to a release build except for a few distinctions.

First of all, part of the integration build task is to assemble the correct objects into a prospective snapshot which will be used for the build process.

Second, there is generally some latitude for modifying the prospective snapshot prior to freezing it. This might occur, for example, if the build attempt caused compile errors, or perhaps if the built product would not successfully initialize during basic sanity testing. Generally, urgent changes are produced to "fix the build". Different shops have different rules for such latitude within a build cycle.

Some will say to leave the initial snapshot and always create a new snapshot for any corrections, especially if there is not good traceability of what changes were required to fix it. I don't generally agree with this as it may cause confusion when a developer comes in the next day and sees four new snapshots, even if they are properly marked with an appropriate status in the repository. It may make it more difficult to pinpoint a snapshot in the past. My opinion is that a snapshot is frozen when the build team puts its seal of sanity tested on it. This does not mean that all sanity tests were passed, but instead, that it is more beneficial to release this build to the development environment then to stick with the previous one. That will be a judgment call at times.

A third distinction is that integration builds are generally done on a specific timeline. In fact, in many shops, they are automatically performed in the middle of the night; that is, nightly builds. Does such a term have a meaning in your global development shop? Some of Neuma's customers do builds in the middle of the night at each separate development site. In this way, they may actually be doing multiple integration builds a day. Others focus each site on a particular development stream and so different streams of development are integrated at each of the different sites.

A potential fourth distinction might be to say that integration builds might be either incremental or full build operations, whereas the same is not true of a release build. There is some wiggle room here. I might recommend that, at least a couple times a month, full integration builds are performed. This will help to verify the continued correctness of the full build process. Most shops tend to do full builds on a nightly basis. This is dependent on the capabilities provided by the CM tools, as well as on the size of the operation. If the CM tool knows what was built the previous night and knows the impact of all changes going into the current build, there is not a necessity to do a full rebuild. Similarly, if your build process has a reliable way of ensuring, for example, that a make operation will always do the right thing, then a full build is not required every night. On the other hand, for release builds, there is always a need to verify full reproducibility so that you know that at a point in the future, if necessary, a full build may be generated without problems.

Developer Builds
A developer build is different from release and integration builds in various ways:

·         The developer may choose to build in an isolated workspace, or to continually re-sync (aka re-base) active changes with the latest (or perhaps another) integration environment, or perhaps to use a mixed strategy: isolate until the changes are in a stable known state and then re-sync.

·         The developer does not generally create a snapshot of each build, especially if the edit/build/test cycle is only a few minutes long. Instead, the developer will generally save (aka shelf) the source changes at a stable known state prior to continuing to work on the change. It is recommended, however, that a complex feature be broken down into a set of logical changes that can be independently checked-in along a strategy that sees the riskiest portions of the feature implementation absorbed into the integration cycle as early on as possible (i.e., not close to release dates).

·         A developer build is performed ad hoc and typically a lot more frequently than release or integration builds. It is not uncommon, especially with today's IDEs, to see a developer perform dozens of builds in a day on the way to producing a new feature.

·         A developer build is most often done as an incremental build. Changes are made and only what is necessary is re-compiled and built. Your build environment should rarely require a developer to do a full build. However, for smaller products (e.g., < 100 KLOC), a full build can be done in under a minute these days. So this distinction is only important in larger projects. Often the IDE is used to decide between incremental and full builds.

·         Developer builds are not released in any way shape or form. Only the developer is waiting for it. At least this is recommended practice. If another developer requires the changes of one developer, the correct approach would be to either check-in, or save (i.e., shelf) the change at the state that it will be used by the other developer.

There are, no doubt, other distinctions that can be made, but we need to get on with the process.

Automation of Integration Builds
The process of automating builds has a lot less to do with the build process than with the change process in your shop. However many shops ignore this and automate the builds regardless of the change process. When this happens, the automated build becomes king and change management, as well as configuration management, fall slave to the
build process. It is important that a proper change and configuration management process be used from which build automation will flow out rather than vice versa.

For example, consider a shop that says, "We're going to use a basic CM tool and automate nightly builds by pulling out the latest checked-in code, including 'make' files, and perform the build operation. We'll then make the resulting build environment available for anyone who needs to make/test changes the next day." The process is easy to implement: pull out the latest code (into the right places) and invoke the top-level make file (which will invoke whatever other ones are required). Its simplicity means that the process will be implemented accurately, so that's good.

However, there are numerous other implications and omissions in this statement, and some have far-reaching effects. For example, a developer will not check in any code that is not fully tested for fear of breaking the build. Good testing is good. Leaving code checked out longer than necessary is not so good. In fact, a rapid add-on to the above process is: "You need approval for checking in any code changes that are not required to fix urgent bugs". In the context of the process, this is reasonable and even good. But overall, code is going to be checked out for even longer periods. When code is checked out longer than necessary, more parallel changes are required. When the developer does not have full control over when they are checked in, even more parallel changes are required. So starts a complex maze of branching and merging and a Configuration Management and labeling strategy begins to grow for management of such branches and merges.

Another implication of the above is that the make files checked in are going to be used for the build operation. That's not bad in itself, but does it force a serialization of changes to the make files, and implicitly the corresponding code changes, that is other than optimal. In other words, the structural changes of the product are tied to the code changes of the make file(s). One frequent response is to segment the make file into many little pieces. Now anyone who has worked with make files knows that they can be complex beasts. So now this complexity, along with all of the maintenance of the complexity, is distributed and/or replicated into many files.

You could probably identify some other repercussions from the above nightly build policy. For example, what happens when a build breaks? What happens if the server is down prior to the build and required changes don't make it in? How does a change control board figure into this? What about parallel stream development so that work on the next release can be started? These questions all hinge on the CM process and tools used to implement your change management Process.

CM Requirements for Build Automation
To keep change management working properly we need to take a different perspective. To effectively run an automated build and effective integration shop there are a number of requirements that must be met by the CM tool and process. Let look at some of these:

Change packages (aka updates): Ability to treat code changes as logical changes, rather than as file changes. Let's face it, a designer will fix a problem or implement a feature. The files being changed, as well as the problem(s) or feature(s) being addressed form a logical unit of change. The change must move through the system as a whole, from check-in through all levels of promotion. File-based CM is an old concept and if your tools push you this way, it's time to look around.

Snapshot: Ability to snapshot a build record and to easily reproduce the build from the snapshot. A build record must be something that drives the build process rather than a record of a build being performed. Reproducibility is the goal. The snapshot can be in the form of an
enumeration of file revisions (including tool/process components), as is done for a typical baseline, or perhaps in terms of an existing baseline plus a set of change packages relative to that baseline.

Push: Ability to check-in code without forcing it into the next build(s), completed code that's out on a developer's disk is a liability. It's prone to disk crashes; it's prone to becoming out of date; it's likely to go through fewer verification cycles. Completed code belongs in a
repository where it can be used by others, simply to review, or as a basis for future changes, without the need to have a parallel change branch. A developer should be able to put a source change into the repository even before dependent changes are checked in. The developer then pushes the change to the "ready" state after confirming it's ready
to go into an integration build.

Pull: Ability to differentiate between check-in authorization and build acceptance [similar to (1)]. Often, especially at the beginning of a release development effort, any code that has been marked ready is automatically picked up by the nightly integration build. Although this may be sometimes sufficient, it is not sufficient as the release approaches and changes are more selectively pulled into the build for integration. The developer pushes, the integration team pulls.

Roll-back: Ability to roll-back changes that broke the build in an easy manner. A change breaks the build. The best solution is to put in a new change to fix the broken part of the change. This provides full traceability of the problem. However, sometimes, it's too complex to fix and the best solution becomes roll-back of the change so that it is not part of the build. Ideally this is a simple state change on the build. If changes are already stacked on top of this change, it may require yanking changes. More complex CM tools/processes use parallel streams for promotion (as opposed to state changes). In such cases, rolling back a change may entail additional steps, some of which may even be required in more than one of the parallel promotion streams.

Fix-the-build: Ability to adjust the snapshot (i.e., continue to (re)define it) by adding or removing changes, until it is frozen (extension of roll-back). More generally than a roll-back, it must be possible to arbitrarily modify a build record (i.e., notice) so that additional fixes may be added to it or errors in definition of the build record may be corrected. This must be easy to do so that the build can be successfully completed for the team's use. It should only be done on a change-by-change basis - never on a file basis. A change must always be in or out, never partially in.

Dependencies: Ability to keep the build definitions in a consistent state for the build operations. As changes are added or swapped out of a build record definition, the CM tool must notify you of violations in dependent changes. For example, if a second change has already been built on the one being pulled, the tool must notify you of this and allow you to take appropriate action to ensure a consistent definition results.

Structure: Ability to associate structure changes with a change. It's no good pushing and pulling changes if the structural changes that go with the change are not also pushed/pulled. For example, if a change moves a data file from one directory to another, or deletes an existing file, these structural changes must be packaged with the source code changes so that everything moves together through the system, whether forward or backwards.

Promotion levels: Ability to support multiple levels of integration, especially in larger projects. Promotion to integration is a single promotion step. In larger projects, there are several levels of integration, especially in task-based development environments where a major task might be sub-divided into smaller tasks which are then integrated prior to integrating the larger task into the mainline. Larger projects will have more promotion levels. I am often reluctant to use parallel streams for promotion because there is a natural reluctance to running through an additional merge cycle when a new promotion level is added. Instead, I prefer state changes with a process which tends to support a higher percentage of changes moving through the promotion levels in the same order as they are checked-in to the repository.

Get: Ability to retrieve files for a build into an integration workspace. Given the definition of a build record, it must always be possible to easily retrieve the files from the repository so that they can be readily and reliably built. This includes support for both incremental and full workspace retrievals. Alongside this is the ability to easily identify differences between any snapshot and the contents of a workspace.

Stream: Ability to identify changes and do builds for multiple development streams of a product. A stream is a separate codeline targeting a different release of a product. For example, release 1 and release 2 would generally have separate development codelines, with release 2 slightly, or not so slightly, overlapping release 1. A stream is basically a codeline that culminates in a new release. The CM process and tools must support this function. Many processes will use a single mainline which switches at some point from release 1 to release 2. I do not recommend this. It introduces a number of arbitrary decision points and more complexity into the development process. For example, the process for making a change to release 2 changes at least twice in the single mainline approach: once when the switch of the mainline to release 2 is completed, and again when the switch for a subsequent release is completed.

Product: Ability to identify changes and do builds for multiple, possibly dependent, products. CM tools need to support and clearly identify multiple products. Dependencies between the products need to be identified, both for build purposes and for impact analysis.

Compare: Ability to compare any to builds to identify the difference in terms of changes (as opposed to lines of code). For a successful build operation, it must be possible to easily compare any two builds. For example, the build manager should be able to take the new build definition and identify the changes that are being added to the previous definition. It's not so much the code changes that are important as the logical changes (i.e., what is being done). It's the logical changes that will be pushed and pulled in correcting an errant build. It's also necessary to be able to identify problems fixed, features added, etc. so that integration testing can first focus on the correctness of these features prior to regression testing of pre-existing features. Ideally, your CM tool will let you point to any two builds and compare them in such a manner.

That's a pretty long list of requirements. Are they comprehensive? I don't think so, but it's a good start. Design your build processes around a solid change management capability and you're well ahead of the game. Start from an inadequate base and you'll be playing catch-up forever. That was OK when we were trying to figure out what CM is. But now that we know, and now that we have a much more solid understanding of change management,

Change Management: Requests and Changes
Change management has two key components and build management must fully support these. On the one side, there's the change request, but on the other there are changes (i.e., change packages). Builds are defined in terms of changes applied to a previous build or baseline definition. This eliminates the tedium of managing file versions and labels. Each change needs to be targeted to a specific product and to a specific stream of the product. The change should exist prior to the first checkout operation. The promotion level of a change, typically indicated by its state, is used to determine which builds it will be part of. Promotion is always done on a change basis. And changes must indicate their dependencies on one another, either implicitly through file history, or explicitly through pre-requisite data. The build process must use the dependency information to alert the build team, the CM manager, or even the developer, whenever a promotion operation leaves dependents behind.

Change requests are quite different from changes and should not be used as a change container. A change request might be implemented in several changes across several releases. Or multiple change requests might be addressed by a change. The originator, or requestor, of a change request communicates the request to the product team, which ensures that it has a proper understanding of the request. Once this handoff occurs and the product team accepts the change request, it is treated as a requirement. The originator will want to know when the request will be/has been implemented and when it will be delivered. The originator is responsible for closing the request (i.e., indicating that the implementation meets the criteria of the request).

Builds as the Key Reference Points
Where does the build come into play? It must be possible to take any build and identify which requests of a particular originator (or group of originators) have been met by that build. On the flip side, that leaves the set that is outstanding. If your CM traceability cannot give you this information, you need to improve your data and processes. Ideally, this information should be available at the click of a button (or at least a simple report request panel).

When this loop is closed, your builds are more than just a set of executables. They become the key reference points for communicating with your customers:

·         What release is the customer running?

·         What requests are still outstanding for that customer?

·         What requests are addressed by the next release that weren't previously addressed?

·         What problems have been fixed in the new release that were not fixed in the customer's current release?

The above questions are answered by looking at release builds, but builds are equally important for your internal customers, especially your development and verification teams. You can start to identify a whole range of questions that can be answered using integration builds as the handles of the queries.

·         What features/problems should be tested that weren't available in the previous verification cycle?

·         What changes went into the nightly build?

·         What structural changes (i.e., new/moved/removed files/directories) have occurred since my last rebase operation

·         What run-time data configuration files need to be retrieved to update my test bed?

The build is a key component of your CM data. It's the one that makes it easy to work with all of the traceability data you've accumulated, as long as your CM tool supports these sorts of questions and comparisons through simple menu functions. Pay attention to your change management process and your CM data and build automation will follow.