Software Configuration Management for Best Practices

As the systems being built today increase in software content, the need for software configuration management continues to rise. Prime contractors are integrating millions of lines of code from multiple subcontractors. Companies are required to produce and maintain variants of their main product to reach out to a diversified market. Project Manager/leaders are aware of the need to better manage and control their projects.

Change is a fact of life in software development: Customers want to modify requirements, developers want to modify the technical approach, and management wants to modify the project approach. Modification is necessary, because, as time passes, all parties know more about what they need, which approach would be best, and how to get it done and still make money. The additional knowledge becomes the driving force behind most changes. But, these changes must be carefully controlled.

This article brings together most of the software configuration management concepts to be analyzed from the project leader point of view. Configuration management will be shown as a project management support function that indeed helps Project Manager/leaders manage and control their projects better.

Purpose Of Software Configuration Management
According to the Software Engineering Institute's (SEI’s) Key Process Areas definition of software configuration management (SCM) (Paulk et al. 1993a; Paulk et al. 1993b), the purpose of SCM is to establish and maintain the integrity of the products produced throughout the project's software life cycle. Knowing the state of the product that a project is developing and knowing that it satisfies the customer's requirements are of utmost importance for any project leader. SCM then can be viewed as a support function that helps a project leader better manage and control the project (IEEE 1997).

The Need for SCM
In many software companies, software support functions such as software quality assurance and SCM are not perceived as value-added by Project Manager/leaders and software developers alike. SCM is frequently viewed by developers as a hindrance to product improvements because of the overhead associated with the change control function of SCM. But on closer examination, one can see that the most frustrating software problems are often caused by poor configuration management. Some examples of poor practices are (Babich 1986):

• The latest version of source code cannot be found.
• A difficult defect that was fixed at great expense suddenly reappears.
• A developed and tested feature is mysteriously missing.
• A fully tested program suddenly does not work.
• The wrong version of the code was tested.
• There is no traceability between the software requirements, documentation, and code.
• Programmers are working on the wrong version of the code.
• The wrong versions of the configuration items are being baselined.
• No one knows which modules comprise the software system delivered to the customer.

Most of these problems were revealed during software process assessments conducted by the authors. Projects, under great pressure to meet difficult deadlines, found their scarce time resource constantly under attack because of the rework caused by these reasons. One developer stated that he had "fixed" a problem three times, and three months after each delivery the problem recurred. Project Manager/leaders cannot afford to have their software developers redoing what was already done.

The cost of rework adds greatly to the cost of developing software and can be reduced by implementing an effective SCM program. The principles behind the modern cost-of-quality concept were derived for manufacturing applications and can be found in the works of J. M. Juran (Juran and Gryna 1988). In 1988 Raytheon Electronic Systems began using a cost of software quality model derived from Philip Crosby (1984) and process improvement methods to increase efficiency. According to Herb Krasner (1997), one of the leading researchers in the field of the cost of quality, by 1991, Raytheon moved from CMM level 1 to level 3; by 1994 it decreased rework costs from 41 percent to 20 percent of project costs. In 1995 it reached level 4. Between 1988 and 1994, Raytheon’s cost of rework from both internal and external nonconformance's was reduced to less than 10 percent of development cost, and the productivity of the development staff increased by a factor of 170 percent.

A key role of SCM is to control changes actively in order to answer the following questions:
What is the current software configuration?

What is the status of the modules?

What changes have been made to the software?

Do anyone else's changes affect the software?

SCM provides visibility into the status of the evolving software product.

SCM answers

Who,

What,

When,

And Why.

Who made the changes?

What changes were made to the software?

When were the changes made?

Why were the changes made?

Project Manager/leaders must be able to obtain answers to these questions in order to manage the project’s technical activities and determine actual product evolution.

SCM vs. Change Control
SCM is often equated to change control. Indeed change control is a critical component of SCM, but it is only one of many. Following is a brief look at the components of SCM and how they connect to supporting a project leader’s ability to manage and control the project.

Configuration Identification
Configuration identification supports the identifying of the structure of the software system and identifying the related life-cycle work products. It provides a unique identifier for each work product, and it supports traceability between the requirements and all other related software products.

Two structures that SCM is concerned with directly affect a project:

• Problem-solving structure.The system concept evolves through the life cycle by successive refinement and elaboration of the resulting work products.
• Product system structure. The system is composed of subsystem components, which are themselves composed of subsystem components.

A discussion between the project and the SCM representative can help the project leader look critically at the software architecture and plan for evolutionary builds that can be controlled and tested at the developmental and system level.

Baselining
Change is a fact of life in software development: Customers want to modify requirements, developers want to modify the technical approach, and management wants to modify the project approach. Modification is necessary, because, as time passes, all parties know more about what they need, which approach would be best, and how to get it done and still make money. The additional knowledge becomes the driving force behind most changes.

The fundamental success of any development effort depends on well-defined reference points against which to specify requirements, formulate a design, and specify changes to these requirements and the resultant designs. The term baseline is normally used to denote such a reference point. A baseline is an approved snapshot of the system at appropriate points in the development life cycle. A baseline establishes a formal base for defining subsequent change. Without this line or reference point, the notion of change is meaningless.

A baseline could be:

• A specification (for example, requirements specification, design specification)
• A product that has been formally reviewed and agreed upon
• A partial system

A baseline is a record of a contract and serves as the basis for further development. It should be changed only through an agreed-upon change procedure. A baseline helps a project to control change without seriously impeding justifiable change. It will help a project to control the identified configuration items but not constrain early development excessively from the aspects of time, money, or resources. Before a baseline is established, change may be made quickly and informally. Once a baseline is established, change can be made but a specific, formal procedure must be applied to evaluate and verify each change. The items in the baseline are the basis for the work in the next phase of the software development cycle. The items of the next baseline are measured and verified against previous baselines before they become baselines themselves.

Some Authors illustrated both the types of baselines that are typical and the quality functions that may be used to ensure that the work products are of the highest quality before they are baselined (Bersoff, Henderson, and Siegel 1980; Bryan and Siegel 1988; Humphrey 1990).

The functional baseline, allocated baseline, and product baseline are most often thought of as organizational or system baselines.

The requirements baseline, design baselines, module baselines, integration baseline (component and system), and operational baseline are often thought of as project or developmental baselines.

System baselines are the records of contract made with the external customer.

Developmental baselines are agreements to assure the project leader that the product integrity remains as it moves from phase to phase.

Configuration Control
In the ideal world, once a configuration item is fully approved there would be no need to change. In the real world, new versions of a configuration item are needed for a variety of reasons:

• The requirements for the system change.
• The boundaries between items in the design hierarchy change.
• The specification of an item is incomplete or wrongly interpreted.
• An error is found that was not detected during the configuration items review.
• The software environment changes in a way that necessitates change.

In each case, a new version of a configuration item is needed that supersedes the earlier version. Without change control, a software engineer could make an important change to a configuration item or its interfaces without a lot of extra work and red tape. No record would be kept of what the change was, however, why the change was requested, who approved the change, who made the change, and who verified the change (IEEE 1997; Whitgift 1991). In addition, it would be hard to find answers to the following questions:

• "Why doesn't my software link this morning? It linked last night!"
• "Why does this regression test fail now? It worked yesterday"
• "Why does the product behave this way now? It didn’t before!"
• "hy are we running out of memory now? We did not have that problem yesterday!"

All changes made to the configuration management baselines or baselined software configuration items should be done according to a documented change control process. The change control process should specify:

• Who can initiate the change requests
• The individuals, group, or groups who are responsible for evaluating, accepting, and tracking the change proposals for the various baselined products
• What the criteria are for placing the software components under formal change control
• The "change impact" analysis expected for each requested change
• How revision history should be kept
• The check-in/check-out procedures
• The process the Software Configuration Control Board (SCCB) follows to approve changes
• How change requests will be linked to the trouble-reporting system
• How change requests are tracked and resolved
• The reviews and/or regression tests that must be performed to ensure that changes have not caused unintended effects on the baseline
• The procedure that will be followed to update all affected software life-cycle components to reflect the approved changes

To control the organizational or system baselines or contracts with the external customers, many organizations establish one or more SCCBs.

The purpose of the SCCB is to ensure that every change is properly considered and coordinated. This SCCB may include members from program management, systems engineering, software engineering, software quality assurance, SCM, independent test, documentation, hardware engineering, and may even include a customer representative.

The SCCB is responsible for receiving and initially evaluating the change requests that come from all sources (that is, customer, engineering, marketing, trouble reports, program management) and performing triage to get the most critical or significant change requests to the right people for impact analysis.

Following the impact analysis, the SCCB ensures that all affected groups are able to recommit to the new requirements. The SCCB can make the decision to implement the change request, defer it to the next release, or discard it altogether. It is also possible that the SCCB will have to seek additional information before a decision can be made.

Project Manager Approval of Baseline Changes
Once the allocated baseline is created and the customer has accepted the software requirements specification and interface specification, the control of the work products and system components comes under developmental configuration control. Normally this means that decisions to make changes are decided by the project Manager/leader. If a change to a software module results in a required change to an interface module to a hardware device, the project manager/leader may share the approval responsibility with the appropriate hardware manager. When the software passes to the integration and test stage, the project manager/leader may share approval authority to make changes with the integration and test manager. When the product is ready to be shipped to the customer and a product baseline is established, the SCCB again becomes the approval authority. What baselines, when they are created during the software life cycle, and who the approval authorities are become part of each project's SCM plan.

Configuration Management Status Accounting
Configuration management status accounting involves maintaining a continuous record of the status and history of all baselined items and proposed changes to them. It includes reports of the traceability of all changes to the baseline throughout the software life cycle, and it should identify what changes have been made to the system and what changes remain to be implemented.

Configuration management status accounting provides visibility into the system evolution by recording and reporting the status of all items and the status of all requests for change.

Few Questions that configuration management status accounting should be able to answer include (IEEE 1997; Whitgift 1991):

• What is the status of an item? A programmer may want to know whether a specification has been fully approved. He or she may want to know whether a subsystem has been tested so the programmer can test his or her modules that interface with that subsystem. A project leader will wish to track the progress of a project as items are developed, reviewed, tested, and integrated.
• Has a change request been approved or rejected by the SCCB?
• Which version of an item implements an approved change request? Once a requested enhancement of a library routine is implemented, the originator and other developers will want to know which version of the routine contains the enhancement. Without this timely information, the project leader has inadequate control as to how to direct his or her project’s technical activities.
• What is different about a new version of a system? A new version of a software system should be accompanied by a list of changes from the previous version. The change list should include both enhancements and fixes to faults. Any faults that have not been fixed should also be identified. This, of course, provides project progress information to the project leader.
• How many faults are detected each month and how many are fixed? Faults are continuously detected during the operational use of the system. Comparing the number of detected and fixed faults helps the project leader, SCM, SQE, test, and other project support teams to assess the stability of the system’s latest release. Tracking the number of faults also helps the program manager decide when to make a new release of the system.
• What is the cause of the trouble report? Trouble reports can be categorized by their causes: violation of programming standards, inadequate user interface, or customer requirements that have been left out. Sometimes when it is discovered that many faults have a similar cause, action can be taken to improve the process and stop such faults from recurring. This information can help the project leader make any necessary process improvements (Kasse and McQuaid 1998) at the project level, and provide the organization's process improvement group with information to make necessary changes to the organization’s standard software processes

Configuration management status accounting is the means by which key project or system information can be communicated to everyone. Project members can easily determine which configuration item should be used, whether it is subject to a change request, and what a build consists of. Project Manager/leaders can easily determine what configuration items passed review, which changes have been completed, which changes are still in progress, how many changes have been accepted, which modules are the most volatile, what a build consists of, and what has been delayed to the next release.

Configuration Management and the Use of Peer Reviews
Configuration management status accounting can help the project leader make decisions on what degree of formality peer reviews should follow. For example: The product the project is building consists of 50 modules or units. Status accounting information reveals that 45 of the modules have changed once in six months, but five of the modules have changed 10 times per month for the past two months. With this configuration status information, the project leader can choose to conduct formal software inspections on the five modules that are experiencing rapid change and use less formal walkthroughs to ensure the integrity of the 45 modules that change infrequently.

Interface Control
The definition of interfaces is one of the most important SCM planning and tracking activities (IEEE 1997). There must be agreement of each group or organization’s responsibility. Any proposed changes to the product or baselined configuration items can be considered and evaluated by all affected groups.

There are two basic types of interfaces that must be considered: organizational interfaces and technical interfaces. Organizational interfaces are those in which configuration management controls the transfer of configuration items from vendor to customer, project to project, and co-developer to co-developer. SCM ensures that the correct configuration items are sent to the correct people. Organizational interfaces also include life-cycle phase interfaces. Phase interfaces become critical when control of the product is being transitioned between different groups (for example, software development group to independent test group for formal testing). Technical interfaces are descriptions that should be placed under configuration management control like any other configuration item. Technical interfaces include system, user, software, hardware, and communication interfaces.

Subcontractor Control
If a portion of a software development project is to be subcontracted to another organization, the responsibility for the SCM generally belongs to the contracting organization and specifically the project leader of the project that requires this outside support. The subcontractor is normally only responsible for the portion of the work that his or her organization is tasked to perform. The integration of the subcontracted work is normally the responsibility of the organization that subcontracted portions of the work.

An effective SCM system greatly increases the opportunity to have portions of the product subcontracted out and then integrated back into a whole that satisfies the customer’s technical and quality requirements. SCM must be applied to a subcontractor to ensure that the subcontractor is able to maintain the integrity of the subsystem for which it has contracted (Paulk et al. 1993a; Paulk et al. 1993b). This includes placing necessary life-cycle products under configuration control to ensure consistency with the main development effort and maintaining a subcontractor's software library that will release the agreed-upon configuration items or subsystems to the contracting organization.

Software Configuration Audits
Configuration auditing verifies that the software product is built according to the requirements, standards, or contractual agreement. Auditing also verifies that all software products have been produced, correctly identified and described, and that all change requests have been resolved (IEEE 1997; Kasse 1995).

A software configuration audit should be performed periodically to ensure that the SCM practices and procedures are rigorously followed. The integrity of the software baselines must be assessed and the completeness and correctness of the software baseline library contents must be verified. The accuracy of the implementation of the changes to the baselines must be verified to ensure that the changes were implemented as intended. It is recommended that a software configuration audit be performed before every major baseline change.

Software configuration auditing should be continuous, with increased frequency and depth throughout the life cycle. Types of configuration audits include functional configuration audits, physical configuration audits, in-process audits, and traceability audits .

Functional configuration audits The objective of the functional configuration audit (FCA) is to provide an independent evaluation of the software products, verifying that each configuration item’s actual functionality and performance is consistent with the software requirements specification. An FCA audit must be concerned not only with functionality but also with performance. An FCA includes:

• An audit of the formal test documentation against test data
• An audit of the verification and validation reports to ensure their accuracy
• A review of all approved changes (problem reporting and corrective actions) to ensure they have been correctly technically incorporated and verified
• A review of the updates to previously delivered documents to ensure their accuracy and consistency
• A sampling of the design review outputs to ensure that all findings were completed and incorporated
• A comparison of the code with the documented software requirements to ensure that the code addresses all and only the documented requirements
• A review to ensure that all testing was accomplished with appropriate test documentation and approved test data to ensure that all configuration items meet the established performance criteria

Physical configuration audits The objective of the physical configuration audit (PCA) is to provide an independent evaluation of the system configuration items to confirm that each item that makes up the “as built” system maps to its specifications. The audit is held to verify that the software and its documentation are internally consistent and ready for delivery to the customer or end user. Appropriate customer deliverable documentation includes installation manuals, operating manuals, maintenance manuals, and release notes or version description documents. A PCA includes:

• An audit of the system specification for completeness
• An audit of the FCA report for discrepancies and actions taken
• A comparison of the architectural design with the detailed design components for consistency
• A review of the module listing for compliance with approved coding standards
• An audit of the manuals for format completeness and conformance to systems and functional descriptions. Such manuals include user’s manuals, programmer's manuals, and operator's manuals

In-process audits In-process audits are held during the design and development phases prior to the FCA and PCA to verify the consistency of the design as it evolves through the development process. In-process audits are performed to determine:

• Are the hardware and software interfaces consistent with the design requirements?
• Is the code fully tested to ensure that the functional requirements are satisfied?
• Is the design of the product, as it is evolving, satisfying the functional requirements?
• Is the code consistent with the detailed design?

Traceability audits Traceability auditing is necessary to be able to verify throughout the software development process:

• Can the software requirements be traced back to the system requirements allocated to software?
• Can the architectural design be traced back to the software requirements?
• Can the detailed design(s) be traced back to the architectural requirements?
• Can the code modules be traced back to detailed design modules?
• Can the module tests, integration tests, and system tests be traced back to the software requirements?

Traceability audits are also necessary to be able to answer the following questions:

• Can it be proven that what one is doing is required?
• How does one know that what he or she is delivering is what the customer asked for?
• Are the software life-cycle work products consistent as the system evolves and change requests are processed?

Software configuration audits check that the configuration management system and practices are effective and efficient for the project and the organization.

Software Library
The software library should contain the items that are important to a software project, including source code, user documentation, system documentation, test data, support software, specifications, and project plans. The SCM library typically stores the configuration items and prevents unauthorized changes to the baselined items. The library system (Whitgift 1991):

• Supports multiple control levels of SCM
• Provides for the sharing and transfer of configuration items among affected groups and among control levels within the library
• Provides for the storage and recovery of archive versions of configuration items Provides service functions, such as checking status, verifying the presence of all built items, and integrating changes into a new baseline
• Ensures the correct creation of products from the software baseline library
• Provides for the storage, update, and retrieval of SCM records
• Supports the production of SCM reports
• Supports the tracing of requirements, forward and backward, throughout the life cycle

A software library provides safe and secure storage for configuration items (key project components) so they cannot be changed without authorization. Acceptance of items (new or revised) into the library is strictly controlled to ensure that everyone accessing items in the library can have complete confidence in their integrity. A number of different libraries may be established to hold different types of items or provide different levels of control.

SCM Plan
The SCM plan is the document that describes how a project will manage configurations (Paulk et al. 1993b; Whitgift 1991). The SCM plan should cover:

• The scope of the plan, including the project, the software to be developed, and the life-cycle phases
• The relationship between the SCM plan and the other standards or plans that describe how the project will be managed (for example, software development, SQA plan)
• SCM roles and responsibilities
• Configuration identification
• Baselining
• Configuration control
• Configuration management status accounting
• Interface control
• Subcontractor control
• Software configuration audits
• Software library

My Conclusion
SCM is one of the most important process improvement tools that Project Manager/leaders can use to evolve and deliver their product in a controlled manner. Knowing the state of the product that a project is developing and knowing that it satisfies the customer’s requirements is of utmost importance for any project leader. Since many of the most frustrating software problems are often caused by poor configuration management, proper configuration management is critical.

Even if an organization has little or no configuration management in place and is just getting started with a configuration management program, five simple steps will add a great deal of control and project tracking information: 1) Formalize the use of reviews before a configuration item is baselined; 2) Uniquely identify system components; 3) Establish simple change control; 4) Build up a repository of configuration items, change requests, and problem reports; and 5) Restrict access to the project library.

A good place to start is with simple status reports, which may include only the versions one has. When developing a change history, one can expand to a status accounting system, which includes who changed it, when, why, how, and what was affected, as described earlier in the article. Then, a configuration audit can be performed to make sure the approved change requests have been implemented completely and correctly. Once established, FCAs would be the next step to ensure that the system matches what was approved, and nothing more. Then, a physical audit can be added to ensure that the documentation matches the changes.

It is important to implement requirements traceability from the beginning through systems testing, implementing life-cycle work-product consistency checks. This means that if a requirements change request has been accepted, one must go through life-cycle phases to determine if it is necessary to make a corresponding change. Without the ability to trace through the life-cycle process, it cannot be done. Once one has the design document, he or she needs the ability to go backward and forward to look at the other life-cycle work products.

After expanding to multisite, multicountry, and multicultural projects, one may be looking at implementing at multiple levels of control boards and need to ensure that all parts are being managed and controlled with consistency and integrity. Otherwise, it cannot all be brought together and integrated into a working and maintainable system. Ultimately, some project managers are responsible for the entire integrated product!

References:
Bersoff, E. H., V. D. Henderson, and S. G. Siegel. 1980. Software configuration management. Upper Saddle River, N. J.: Prentice-Hall.

Crosby, P. 1984. Quality without tears. New York: McGraw-Hill.

Haley, T. J. 1996. Software process improvement at Raytheon. IEEE Software (November): 33-41.

IEEE. 1997. IEEE Software engineering standards collection. Washington, D. C.: IEEE Press.

Kasse, T. 1995. Project Leader Exploratory Question Database, Antwerp, Belgium. Institute for Software Process Improvement, internal document.

Paulk, M. C., B. Curtis, M. B. Chrissis, and C. V. Weber. 1993. Capability Maturity Model for Software, version 1.1. (CMU/SEI-93-TR-24). Pittsburgh: Software Engineering Institute, Carnegie Mellon University.

STSC. 1994. Software configuration management technology report, Software Technology Support Center. (This document provided by the STSC is a checklist of criteria to help organizations develop their requirements for selecting configuration management tools.)

Walker, G. 1996. What is configuration management? Alcatel Alsthom internal presentation, Paris, France.

Whitgift, D. 1991. Methods and tools for software configuration management New York: John Wiley & Sons.

International Organization for Standardization (ISO). 1994. ISO 9000 Quality Management, ISO Standards Compendium. Geneva, Switzerland: International Organization for Standardization.