How Configuration Management Is Changing: An Interview with Joe Townsend

Joe Townsend has been working in the configuration management field for fifteen years and is a regular contributor to CMCrossroads. In this interview, Joe discusses how configuration management has changed over the years, the trouble with tools, and trends in IT.

Jonathan Vanian: Okay, hello. I'm here with Joe Townsend. Joe, you're a regular face around our parts, our family of publications, so nice to have you talking with us today. Can you explain a little bit about your career so far to readers and listeners who might not be aware?

Joe Townsend: Thank you Jonathan. A pleasure, of course.

JV: Definitely.

JT: My career started back in 1998 in configuration management. That time, I was one of the few people that actually got to choose configuration management. It usually fell upon the worst developer. At least that's what I found when I went through my training classes, that someone would say on Friday they told me I had to have this training class on Monday and I was now the configuration manager, so I have to learn these tools. At that time pretty much everything was tool based. People were thinking version management was configuration management. That's the first class that I went to that seemed like everyone was saying that that was pretty much configuration management. Of course, that's the largest piece of the puzzle when you look at the functional side of configuration management, is version management.

Way back then in 1998 it was just in its infancy as far the getting away from command lining and going to more of a GUI interface, more configuration management tools. I ended up leaving the company that trained me after a year and started consulting, first at Thomson Electronics in Indianapolis and at Boeing Aerospace in Alva, Oklahoma, the C-17 program, which took me to United Postal Service on the Hub 2000 project in Louisville Kentucky, their air hub. I implemented, basically, the version management and issue management at their location.

At Boeing, they were more advanced than most companies. They were in aerospace and defense, so they were really light years ahead of most companies as far as configuration management, having an understanding of its importance. There, I just implemented a tool that seemed to be the going thing at the time. The tools were brand new, everything was fresh, everyone was excited about the possibilities of the tools.

JV: What drew you into that? Why were you attracted to that?

JT: One of the most important reasons was I could become certified in the PDC suite of tools at that time. Like anyone who's young in IT, certification means something. At that time, it fit. It really was the field itself, being able to help the developers and the control of the environments and coding. Really, it just drew me in.

JV: The organization aspects of it.

JT: Every bit. It pretty much fit me as a person because everyone who knows me knows that I'm a law-and-order kind of guy. CM at that time was pretty much the Wild West, as far as development goes. People were just everywhere and doing things, and disorganized. There's no talk of agile, there's no talk of lean, or any kind of structures. People who were coding at that time were getting ready for the Y2K bug. I was working on a COBOL shop. Of course, that was huge for us. Supposed to get everything done, get everything built, get everything ready for year 2000.

JV: Definitely. Order, control over that, especially for that storm that could have really been destructive.

JT: Yeah, the storm that wasn't.

JV: The storm that wasn't. Obviously, the nature of the world is changing over the past decade. Walk me through configuration management, how it's been changing. We've got the rise of the cloud, new technical advancements, new management styles.

JT: It's obviously changed since 1998. I think the biggest thing was the rise of the ALM tools, like PDC, Dimensions, ClearCase Suite. Those really didn't do what was advertised. At that point, even though CDS had already existed way before those tools, and it was being used, the ALM side never really took hold of it. These companies were buying each other out and buying add-ons and doing all these different things.

JV: When you say they weren't really doing what they were doing, what were they supposed to be doing?

JT: They were supposed to bring the whole build management, version management, release management, requirements management; all the whole ALM perspective into one tool. That didn't work so well. The tools either weren't truly integrated or they just were so complex that development teams couldn't really use them.

JV:  More of a hindrance than a help.

JT:  Absolutely. Things are supposed to get better as time progresses, but I think with the CM tool market, things weren't progressing fast enough for the development market. Of course, the rise of agile at that same time and lean concepts, it was just coming to a point where you were spending more time dealing with the tool than you were dealing with your trade, from a developer perspective. As a CM practitioner, we didn't fail, but the vendor side did. You had the advent of Subversion, yet there were other tools that were simpler. Just version management at first. These developers flocked to it because you couldn't beat the price. You get a tool like ClearCase, ClearQuest, everything else, Dimensions. You're looking at a substantial investment, and these teams weren't getting the bang for a buck; just weren't getting what was advertised.

JV:  Right.

JT:  Now, that's changing with these teams that are working with Subversions, integrating them into tools. I'm actually working on a project now that's integrated. It's got Subversion integrated into J developer, Oracle development tool.

JV:  Oh, wow.

JT:  It's seamless. It really does what it needs to do. It's simple. The developers are in their IDE. They call us. Again, you can't beat it when it's free.

JV:  One tool can speak to the other tool. They can correspond with each other easier than before. Whereas before, the promise here is the big package you're getting, but wasn't really like that. 

JT: Nothing really worked well together that time. Now with the open source and the community just thriving- they're writing the integrations between Codezilla and Subversion and Git and everything. They're writing- basically, they're writing the plug-insurance for the ALM to make an ALM solution, and it's all free. That's been huge, and it's going to continue, I think.

As you can see from Serena just recently being sold again to a private partner. IBM offering Rational to obtain licenses for free. They're seeing that they can't compete in this market because they didn't have what the developers needed. I think we're getting to see a real renaissance in the CM world as far as tools are concerned, were it's going to get much more robust than we could have ever had with the big vendors.

JV: Got you. Now we know that you know the tools are modernized. What about the role of the CM professional themselves? How are they fitting within this larger IT?

JT: Really, there's two types of CM professionals, maybe three if there's people that are field managers. it's hard to say, because we're going from being configuration managers to being more functional in nature, like release managers or build managers, deployment managers, or the change managers, or avocation changes. You've got a functional side versus the four pillars of configuration identification: control, audit, and status accounting. That still exists heavily in the defense industry, but in the private sector, it's becoming more functional from the configuration manager's side.

Me, I'm a jack-of-all-trades. I do everything. I do build, release. I take care of the tools, support, help desk. I do everything from a CM perspective. Depending on the size of the company and the size of the effort, all you ever may do in your configuration management career is do builds. You may be the release manager for ten, fifteen years. That's all you do, the release management side of the configuration management.

JV: The roles sort of been broken up into several parts. One of those several parts, that might be a person's job for maybe their whole career, as opposed to overseeing the whole process.

JT: Absolutely. Now, there's that side. Then, there's what I would call a true configuration manager, who's taking care of the process side. That's more prevalent in the defense industry, government, and larger efforts where you may never do a build, you may never do a release, you may have nothing to do with the functional side, but definitely handle the process side of holding the cabs, making sure that the processes are being followed, and writing the configuration management plans. The overwriting of the individuals for the projects. That may be your job all the time, it's just taking care of that clerical stuff.

JV: Right. Yeah, that seems like that role is better suited for, like you were saying, the projects that are huge. There might not ever be a release. When you think of the apps that are constantly being developed now with such a fast-paced development cycle, how do you see the role of CM fitting in with today’s private sector? The way apps are getting created nowadays?

JT: Well, with agile and lean concepts, you can have a two-hour change control board. You've got fifteen-minute sprints, and if you're invited, in that fifteen minutes, you may have thirty seconds to have your voice heard. I still think that developers and agile has to understand importance of configuration management, but you're not going to be able use the traditional things that you may be used to. If you came from a defense industry to someone who's writing apps for iPads, I think you'd be dreaming to think you're going to come in there and have three-hour configuration control board meetings. It's not going to happen.

I think that's one of the problems of our industry is people see, if you're not doing all of it, if you're not being a purist, then you're not doing configuration management. I beg to differ. I think you're going to have to get rid of the wasteful side of CM.

Just for example, you can talk about a configuration item. I've seen conversations going for months about just configuration items. No one's going to have time for that in a lean environment or a Scrum. You're not going to go sit there and say, "I didn't know what the CIs are." They're not going to understand what you're talking about. If you start talking about configuration control, “I need to do a status audit or status accounting,” they're going to think you've lost your mind. They would say, "What are you talking about? We need you to do the version management side of it. We need to set up continuous integration. We need bills. We need these things. We need thee functional things. I have no idea what you're talking about when you say configuration status accounting."

JV: Yeah, it's just a term to them, to some people.

JT: Bob Aiello once said that to me that if you try to say those terms where he sat up in Wall Street, they're going to run you out of there. They don't know what you're talking about. They don't want to speak about it. They need you to do something. Looking at it from a functional standpoint, what can we do? What can we cut out? What can we not have to do to make things go faster? That's the whole purpose of configuration management—to help the team to make it work better, cheaper, faster.

JV: Right. The irony is that some people just don't quite understand it, so then it becomes more of a hindrance than something that's beneficial. How does these two trains of thoughts fit in, then? How can agile and CM coexist when you got people who don't understand and you got the CM person who has all this expertise?

JT: Even though they may not understand it, you have to take care of that side. You have to the base lining of the code. You have to make sure that things are repeatable, that everything is working correctly from the configuration management side. They just may not realize it. It's not a thing where- that's the thing about configuration managers, they don't always do configuration management tasks. It's usually done by the developer and others; the QA side, the quality control. You may not actually do the actual task, you just got to make sure they're being done. More falls on you as far as making sure that things are being done above board. Rather than trying to sit there and talk theory with folks who just want code.

JV: Right.

JT: That's what it all boils down to. You have to make CM work where; you have to make it happen. That's a tall order. I think it's still possible. You just can't stick to the traditional ways and expect folks nowadays to understand that, or even want to go along with it. You just have to make them do it without realizing, basically, is what I'm saying.

JV: What are some ways that someone could go about doing that, doing what you just said? What are some ways that someone can convince someone that this needs to get done?

JT: The biggest side is going to be the version management side. Doing the code check-ins, making sure that your baseline, your code, that when the build's done, that you a, know what's in the build. That's the configuration status accounting or the audit that you can do. You can make sure that the build reputable. I actually do that now where I'm at. We do a build off of a branch, and then we integrate that code into the trunk. I do a confirmation build to make sure everything's the same. We can compare that to your file, and it works perfectly. If you said to do that in a traditional setting, there's no way on earth I think you can get that done where there would be that trust factor.

I think that's a big part. You need the trust factor with your developers. They know how to work with you. They know what you require. I think that's the big word here, it's the trust. You have to trust your developer. Your developers have to trust that what you're doing is adding value, making things easier, and giving them that safety net. Developers, again, every time I'm doing desktop support, they're not all hackers, but they hack things up. That's the nature of their business. That's what they do. You have to give that trust back to build where you can do things behind the scenes and you give them that safety net. They all believe that, eventually. Everywhere I've been, they needed that safety from configuration management,  that safety that we provide.

JV: Do you think CM fits more in with these mission-critical builds and modern-day what private sector offers? Is it more inclined to fit in with that world, as opposed to somebody who's building apps on the iPad who wouldn't know how to even respond with an audit.

JT: I think it fits better in the defense industry, because they're more used to it. I've been around folks in the defense industry and the aerospace side. They get configuration management. They understand its importance from beginning to end. It is very structured. Everything is very structured. You got to follow step a, b, c, d over and over.

When you're working in the private sector, sometimes the haphazardness of it is, you can't have the structure. From a configuration management process, it's really difficult to not have structure, because that's what we are. What makes us good at being configuration managers is that ability to containerize it and to hold things, and to control them. When you go to a private sector, you have to let some of that go.

You could still do it, but it's got to be below the radar. You have to do it where you're doing it, and the developers don't know it. When something happens, you can then provide that safety net that, "Oh you know what guys, I've been trying this. I know what's new. I know what was in that build, and here it is. What happened four builds ago, what happened three months ago, where were we at? Just one second, I can tell you." It's more behind the scenes, kind of the best boy grip, something like that. You're doing all the work and no one ever sees, or no one ever notices until the credits.

JV: You could do the whole job without actually ever explaining the configurable item, or having to explain to someone the terms.

JT: Absolutely. I never talk about those things. That's the good thing about how I grew up in the CM, is I can talk about those things. I can talk theory all day. At the end of the day, when it comes time to hit the road, that theory is not very valuable to a private sector right now, and agile isn't either. Yes there is "Did you get that continuous integration server set up? We had ten builds today. Yeah, which ones were successful?" That kind of stuff. They're not going to want to know that you can go in and give them a status update on the current issues. They can do that with a spreadsheet or a tool from Bugzilla.

JV: What do you see CM being in the next couple of years?

JT: I think it's going to continue to grow. I think there's going to be a continual need for it. Again, it's becoming more functional. It's becoming more, "You're going to be the build manager. You're going to be the release manager." You're not going to be called a configuration manager anymore. You're that person who does configuration management things, but you'll be the build manager, the integrator, the employer. I think as long as defense contracts and government control, they have the standard that they have, that'll always be in place.

I'm afraid that as these people, as these project go and the age starts getting to these folks, that I think the functional folks are going out in the end. I know there are some government agencies looking at agile. That's an oxymoron, government projects as being quick and lean and doing things in any kind of fashion or form other than the waterfall. It's starting to penetrate the government side.

When that happens, I think the CM empire is going to fall to the functional side. They're going to see the importance of it, and how our job helps. I think they'll interact well together if you have someone who's willing to step back from the theory and put that into practice, put the CM principles in practice. You can still hold true the four pillars of CM, but you can definitely change enough to where you become more beneficial to the team. That's always going to be the case. We're a support group. We're not there running everything. We're there to support the developers, support the QA, QC, and to make sure that things are being done right.

JV: That safety net that make sures everything's repeatable and nothing breaks or anything like that. Do you still see government adapting agile or is it just all talk, hearsay?

JT: I think it will increase. As the new developers grow up and as the kids come out of college, they're going to want to work on these new, exciting teams. They're not going to want to do waterfall. They're going to want an agile. To me, agile just works so much better, because you get the involvement early on in the process. In the development process, you don't wait 'til you did your requirements, you've done your design, you did your development. You send it to QA, the customer finally sees it in UAT, and they don't like it. That can't be sustained, not with the way the apps are needed, their quickness. As you can see, I think that the Flappy Bird guy is a good example. He had something there that really was a simple application. It blew up.

JV: Flappy Bird, yes.

JT: It really did. It blew up. If that was a team developing that rather than one person, there would have to be fifteen different versions of that for different languages, for different cultures. It's a simple game, but I think it illustrates that. Today, the way apps are being developed, that app didn't probably take very long. If he would have wanted to stick with it and say, "Hey, I want to blow this thing up and do it in Chinese and Russian and Vietnamese and Japanese and French and German, I'm going to make tools more for their environments. “

You can't do that during  waterfall. You have to do everything quick, get everything done. I know Flappy Bird, you don't really think of as being language-specific, but you know what I'm saying? If an app needs to be changed quickly, you can't do waterfall. You're going to have to do agile and get the people involved early. That's the duty of agile. It gets the tester, it gets the business user, everyone involved early so you don't wait for the last minute to find out that, wow, this is not really what I wanted.

JV: I just wonder, we now have medical apps, apps that are really sensitive for life. Do you think you might need some of that safety net of CM to make sure that these things are built correctly. Apps are rapidly built, but then as we're making more mission-critical apps, where is that safety net?

JT: If you go that route, honestly, you're talking about life or death for someone. I don't know how much agile would be suitable for that. You want to make sure it works. You can't test it in the field. I think it would be even more important to have configuration management there to make sure you know what you're releasing, can repeat it, and you can go back should something go wrong. If someone dies on the operating table because an application crashes in the middle, that's going to be very important to see what happened. There's going to be a lot of lawyers interested as well as to what happened and why this application crashed in the middle.

JV: I can imagine that, yes.

JT: Again there, the safety net of configuration management becomes even more important. Depending on the app, what it's going to be used for, I think that's the biggest boost for CM is the fact that no one else can in the field can give you that safety net, can give you that way of protecting things the way configuration management can.

I've seen applications, very simple ones, where developers have tried to use what I call poor man's CM, where they just make copies of folders. It gets to a point where they don't know what's happened. They don't know what they released, or what they've done. CM gives you that ability to say, "Hey, you know what? I know what went on that day. Here it is. Here's the release. Here's what was in it. Here's the executable. Here's the code behind that executable."

JV: All right. Well, we're coming close to the end here. It's been a really good talk, chatting about this. Any final thoughts you want to leave to readers and listeners?

JT: I think the main thing about configuration management is as the field has—one of the things why we don't have a bigger seat at the table is the fact that I think there's a schism in CM between hardware and software configuration managers. They, really, you can't apply hardware configuration management principles to software. One of the things I saw the other day, someone saying, "Configuration item has fit form. Will you tell me what the fit and form of a piece of code is?" What does that mean?

JV: Yeah, what does it mean to anyone who doesn't know those terms?

JT: Yeah, I was thinking the whole time, it's said in there that in this standard, the fit and form of the product. Just like in hardware, yes, many parts make up a larger part. Same thing in software. A knob is a knob. A dial is a dial. A altimeter is an altimeter. The software behind that altimeter is very different than the actual piece itself. We have to get past that. That's the one argument that I see over and over in forums and groups is, people want to apply same principles to hardware that you do software.

JV: It's not going to fly.

JT: It's not going to fly. It's not going to work. We have to understand, as configuration managers, that's different. To explain that to folks here who are not used to that, I've done it before at a conference. It's a lot of heads nodding, but at the end, I don't think they really understood what I was talking about and didn't really agree what I said. I got to keep saying it. We got to quit looking at these things the same.  Software doesn't have fit and form. It's all over the place. We have to just control it as best as we can.

JT: Right. All right, well thank you Joe so much for taking time out and talking. People can check us your stories that you write for us on Techwell and on CMCrossroads for more information. Thank you so much, Joe.

JV: Thank you, sir.

JT: All right. 

Joe Townsend has been working in the configuration management field for fifteen years. He has worked for CNA Life Insurance, RCA, Boeing, UPS, and INPRS. Joe has primarily worked with Serena tools, including PVCS Version Manager, Tracker, SBM, Dimensions, and Subversion is also an administrator for WebFocus, Service Now, and supports Eclipse users. He is responsible for building all of the applications at his current location, which includes a desktop and web-based client.