App. Infrastructure Configuration Management Keeps Good Apps from Going Bad

You eagerly anticipate the day when your application can leave the house and enrich the world. But when that day comes, that’s not what happens at all. Instead, you get an angry phone call, because your application isn’t enriching anyone. In fact, its poor behavior is causing dread and anguish in your IT department. You’re accused of being a lousy parent. And you can’t believe your ears, because your application passed every test with flying colors before you unleashed it on the cold, hard world.

The problem isn’t that you’re a bad parent. In fact, you prepared your application to perform perfectly in your own environment. The problem is, when the application left the house, it was faced with foreign environment for which it was totally unprepared. As a result, it did not function as an "upstanding citizen" within that environment and became a problem child.

For developers of J2EE applications, this is an all-too familiar scenario. Your code is perfect, but the infrastructure team hasn’t configured its production environment properly, so it does not match the configurations in your development environment. As a result, your application misbehaves and has to be taken off line. It’s not your fault, but it is your problem.

To date, reconciling configuration settings has been a manual process; one that is error prone, resulting in application downtime and massive consumption of IT department resources. It also is a constant source of friction between developers and infrastructure teams.

Many IT organizations today are eliminating these issues by implementing new automated Application Infrastructure Configuration Management (AICM) solutions. These solutions impose strict configuration-management discipline on IT personnel across all environments - development, test, certification, production and hot standby - so that applications always behave the way they should.

Intervention Counseling

The first step on the road to stopping applications from being deployed into improperly configured environments is admitting that the problem exists. Unfortunately, many organizations continue to live in denial.

The reality of J2EE development is that sending the application itself out into the real world is never enough. In order for it to behave reliably and predictably, you must also provide very specific instructions on how to configure the environment in which it runs.

Feel like you’re doing that already? Then why do your infrastructure people call you all the time complaining that your child is incorrigible?

If you have a good release process in place, you probably already have a method for capturing configuration settings and passing them along to players in the process downstream. This method can be as simple as a spreadsheet that describes various property settings, release notes, or some other document that describes configuration settings.

One problem with this approach, however, is that it still relies on human beings to translate what is in the document to what is actually configured in the application environment. As we’re all painfully aware, humans are an error prone species. And as long as we rely on humans to manually input and manage configurations, there are going to be frequent configuration errors.

A second problem is that this method relies on you, the developer, to update the documents that describe configuration and keep them accurate. With application environments becoming more complex, the opportunity for "fat fingering" configurations grows exponentially. Furthermore, even if you deliver a perfect configuration document, someone downstream is likely to screw it up and point the finger of guilt at you anyway.

In rare instances, developers pass along the actual configuration files they used during testing to drive the application environment. The problem with this approach, however, is that even when the application and the associated configuration files arrive at the next stop in the lifecycle, the configuration files need to be modified to accommodate the differences in the new environment (hostnames, port settings, etc.). While this is a good step toward ensuring a more accurate application environment, the fact that manual edits are involved still introduces a tremendous risk of human error.

And finally, let’s say your infrastructure team has gotten around the problem by writing some perl scripts to automate the deployment of the configuration files and make the changes to accommodate the differences in the environment. What happens when a new version of WebSphere, WebLogic or JBOSS comes out? The scripts most likely have to be rewritten. This solution might work while the number of machines, app server instances and layers in the infrastructure are relatively small, but will it scale?

Enter Application Infrastructure Configuration Management

Application infrastructure encompasses the tools, products and servers that run, support and maintain the operation of an application. Obviously, application servers such as WebSphere, WebLogic and JBOSS are the most common infrastructure elements that require configuration management. However, anything in the environment that has configuration files or settings associated with it, falls into the category of "Application Infrastructure." While the application itself is not considered part of the infrastructure, all artifacts containing configuration information about the application are.

With so many elements requiring configuration management, it raises the question: Who in IT would actually use an AICM solution? While the answer to this question will vary widely among IT shops, the general rule is this: any person that either creates or manages configuration data should have access to AICM tools. This includes developers and infrastructure teams (in both pre-production and production phases of the lifecycle).

In many shops, most of the infrastructure - including systems in the development environment - is guarded zealously by infrastructure teams who would never allow developers to make changes. However, if they are to overcome configuration-related application problems, these organizations must change their culture and allow developers to adjust configuration artifacts if necessary.

AICM solutions eliminate the need for infrastructure teams to be so stringent in their practices, because they provide built-in work-flow capabilities that ensure proper processes are always followed. This enables developers to make required changes in the configuration model, and then send those changes to infrastructure teams for approval and deployment.

Where Traditional CM Tools Fall Short

At first glance, AICM appears to be similar to software configuration management (SCM); controlling, versioning and grouping software-related artifacts, and providing the capability to build processes around them.

While they are similar, SCM and AICM are not the same. To illustrate the difference, consider the evolution of Web content management. We all remember in the 1990s when SCM vendors made a mad dash to the perceived lucrative field of content management. And, we all remember how they fell on their collective faces by trying to address SCM and content management as similar problems. A billion dollar industry did spring up around content management. But it was new companies like Interwoven and Vignette that dominated the market, not retro-fitted SCM solutions.

What these SCM vendors failed to realize was that although the core repository was the same, the contributors and consumers of Web content had a set of requirements that were fundamentally different from those of software developers. The reason that content management vendors like Interwoven and Vignette were successful was that they crafted a solution from the ground up with the specific requirements of Web content contributors and managers in mind.

AICM at its core looks like it might solve a problem addressable by SCM vendors. However, upon closer examination it becomes clear that this is not the case, because AICM has different data, different audiences and different requirements than SCM.

AICM Core Requirements

There are multiple consumers and contributors of configuration related data across the application lifecycle, so AICM must satisfy the needs of multiple audiences. Since most organizations already have an SCM solution (ClearCase, PVCS, CVS etc.) it’s not a requirement that AICM systems provide their own version engine. However, they should provide clear, well defined integration points to third party CM tools for versioning and archival purposes.

The high level components of a competent AICM solution include:

• A repository for configuration artifacts.
• A robust data model that enables advanced configuration analysis, so users can discover and map dependencies, validate entries, search and replace, manage changes between application lifecycle stages, etc.
• Features to improve efficiency and support cross-organizational configuration management, such as filters, compare, automated deployment and auditing.
• Support for multiple file and data types.

These core capabilities enable AICM to virtually eliminate the "problem child" scenario described earlier, because all environments are governed by a single system, rather than multiple humans. For large IT departments, this can save thousands of man-hours each year, simply by eliminating errors and sparing personnel from repetitive and tedious manual configuration work.

And for the developer, it means no more phone calls complaining about your application, when in fact it was an improperly configured infrastructure element causing the problem.

In addition to these core requirements, a complete AICM system should also provide additional advanced features that:

• Allow users to continue to use whatever process is currently in place without imposing a specific "one size fits all" process.
• Allow users transparent access to scripts and automation tools that may currently be in use.
• Provide the ability to quickly "clone" or make exact copies of an environment.
• Provide templates that expedite the addition of new resources into the configuration hierarchy.
• Allow the system to be easily extended to accommodate new technologies or software in the infrastructure stack.
• Provide a framework that allows repetitive tasks to be automated.
• Allow for check pointing and roll back of configurations.
• Provide complete change audit information including the who/what/when/where/why of a change.

AICM Stops the Blame Game and Fixes Bad Behavior
Developers are constantly held accountable for their programs, tools, data and processes. However, IT culture allows infrastructure teams to perform ad-hoc management of mission critical configuration data with virtually no accountability. And, when something goes wrong, the finger of guilt is pointed at you, the developer.

Imagine how much easier - and fair - all of your lives would be if infrastructure teams could simply roll back to previous configurations when something isn’t working properly? Or do a simple compare to identify what is different from environment to environment after changes have been made? What if they could roll out your applications much more quickly by reducing the time it takes to get the application environment right at each phase in the application lifecycle?

AICM makes this possible by automating these processes and eliminating the potential for human error. This results in faster application roll outs and fail-safe management of infrastructure changes, such as application server upgrades.

By eliminating human error, automating configuration management and providing an audit trail of changes made, AICM is the best friend your "children" could have.

Raman Sud is Vice President of Engineering for mValent. He has 20 years of experience delivering mission critical software for enterprises and telecommunication service providers, and has significant expertise in building and integrating distributed teams in the US and India. He brings extensive technical experience in systems software, information management, distributed systems, and unified messaging. Prior to joining mValent, Raman was a co-founder and VP of Engineering at PurchasingCenter.com / Excara. Previously, he was Vice President of Commerce Applications at Open Market, Inc. In addition, Raman has held technical and management positions at IBM / ROLM Corporation, Centigram Communication Corporation and Octel Communications.