Configuration Management Primer For Hiring Managers: Two Different Jobs Confused By The Same Title

Two Types of Configuration Management
In reality, there are two types of configuration management:

1. Software Configuration Management (SCM)


2. "Classic" Configuration Management (CM)

The best way to understand the difference between the two is to look at the business goals they are trying to achieve.

Software Configuration Management
The goal of software configuration management (SCM) is to produce timely, repeatable builds. The person performing this function typically has some of the skills and experience of both a systems administrator and a software developer, although he need not be fluent in the computer language that his developers are using. He might hold the HR title of either developer or sysadmin, but his team will unofficially call him something like "SCM coordinator" or "buildmaster". In pursuit of the goal of timely, repeatable builds, the responsibilities of a buildmaster are:

1. Manage the tools that he and the developers use to create source code, store source code, and build packages, a/k/a releases. The tools typically include

· A version control system (VCS) where the developers store their source code.

· The IDE (Integrated Development Environment) that developers use to write code.

· Tools like make, ant, and maven for automating builds and managing libraries of built code, and JUnit for running automated unit tests within builds.

· A continuous integration (CI) server that automates the timing of builds to enforce that checked-in code will still build and pass unit tests.

2. Create scripts to automate builds.


3. Enforce use of the VCS by creating automated builds that pull only from the VCS.


4. Centrally coordinates VCS usage standards, including the strategies for when and how to branch and tag and merge, and their associated naming schemes. (A development lead could do this for an isolated project, but an SCM coordinator is needed when there are multiple, interdependent projects.)


5. Acts as a gatekeeper by verifying that units tests are all successful before handing off a build to the next group for promotion to a QA environment.

Note that, although the buildmaster makes sure that unit tests are automatically run within builds and that reports of unit test failures are available, it is the responsibility of development leads to ensure that individual developers fix failed unit tests - the buildmaster lacks the hierarchical authority over developers to enforce quality.

Some smaller sites additionally task the buildmaster to be a release manager. In that case the buildmaster oversees the promotion (movement) of builds up the ladder of testing environments and eventually into the production environment (or release to manufacturing for products).

"Classic" Configuration Management
Classic Configuration Management (CM) grew out of several industries, principally aerospace, which had nothing to do with software builds. Unlike software, they dealt with lots of material pieces and parts, and how a revision to one piece might need a corresponding revision to another interlocking part in order to be able to still fully assembly the product. (Notice how this sounds like a VCS for the material world?) The goal was Change Management, i.e. to manage change so that a revision to one piece wouldn't break the ability to assemble a product.

But modern configuration management is a different kind of change management - I like to call it object-oriented change control. To understand why, we'll have to look at some more history ...

Many in IT will remember when Change Control was a hot topic around 1990. The idea was that we'd define a narrative set of rules for identifying when a proposed change might have a broader impact, which in turn would trigger a review, and further tell us how many levels of approval it needed. This methodology failed miserably. It was doomed to failure because it depended on humans to create a limited set of rules to cover a near-infinite number of possible circumstances, and these rules were interpreted by error-prone humans who also did their best to avoid waiting for, much less sit through, monthly Change Control Board meetings.

Some years later, the IT industry noticed how aerospace was doing change management based on objects. And because IT had embraced object-oriented programming, it was already comfortable mapping material objects into software objects. So instead of fuzzy rules, why not turn the old change control methodology inside-out? - Ditch the rules and track the objects instead! That way, there is no room for interpretation. ("Yes or no, is this object being changed, and, if so, is it on the list of objects we review?") Even better, the decision workflow can be automated.

In practice, this means creating a list of all objects for an application or environment, e.g. configuration parameters, and putting them in a database (we'll call it a CMDB), and mark which ones are important enough that we want to do a review before we allow a change to that object. Then, provide a computer program to after-the-fact check the fielded configuration against the central CMDB; or, even better, provide a program to proactively update the fielded configuration from the central CMDB. By using lists of objects, combined with computerized cross-checking of changes, we take human judgment out of the equation while also providing scalability.

The description above of CM sounds good, but does it translate to the real world? A concrete example might help. One of my past clients was having a terrible time consistently finding bugs during QA, and subsequently bugs were slipping into production. One problem was that their many instances of Websphere (a Java application server marketed by IBM) had been manually configured and didn't match each other. So I created a CMDB of their Websphere configurations. Using about 20 lines of jython, I was able to extract the complete configuration of a Websphere cluster as a text file. With 3 apps installed, each Websphere cluster had about 25,000 configuration items. The client used a sequence of 3 promotion levels for testing, plus a final promotion level for production. Likewise, there were multiple clusters at each promotion level. The configuration for each Websphere cluster was automatically fetched each day and checked-in to a VCS as a new revision. With that CMDB and a few text manipulation tools, I could:

• Verify that all servers within a given promotion level were all configured identically, and flag any mismatches. (Horizontal consistency.)


• Verify that configuration updates required by a new build were applied to the servers at the promotion level where the new build was installed, and flag any mismatches. (Vertical consistency.)


• Detect unauthorized changes made since the previous day. This also provided a history of authorized changes, in case we ever needed to revert to a known good configuration.

So, we now see that classic Configuration Management is a methodology for performing Change Management. Moreover, the business goal of classic CM is to maintain consistency as change happens, raising a flag when the left hand tries to make a change that affects what the right hand is doing. The introduction of CM is also a form of business process improvement.

If you want to know more about classic CM processes and practices, one place to start is CMMI (Capability Maturity Model Integrated), in which CM is one of 22 process areas.

Contrasting Skills
Because the objects that are "placed under Configuration Management" can represent real materials or business documents or software, the classic CM practitioner need not have a background as a sysadmin or developer, though some knowledge of the object domain is useful, as is a high comfort level using IT tools. The CM practitioner is graded on how few "oopses" occur due to incidents resulting from configuration inconsistencies.

In contrast, software configuration management (SCM) is about getting software builds out the door. A buildmaster is graded on meeting the deadline to create a new build after developers request one, while at the same time ensuring that new builds reflect every developer change.

Conclusion
Hiring managers, particularly ones in IT, will be receiving an increasing number of requests for Configuration Managers. Because there are two types of CM, the hiring manager needs to know the difference between the two so he he can understand which skillset his team really wants. Software development managers typically assume that their team simply wants a buildmaster, which may not be the case for large projects. (See the Websphere example above.) Infrastructure and operations managers in IT are just starting to understand CM, and they will want people with classic CM skills rather than a software buildmaster.