A pragmatic approach to Build and Deployment management

Getting Started - How?
When it comes to configuration, build and release management, the first challenge is how and from where to start? Most of the development teams fail at this step itself. The SCM person in the team is asked to identify and document the change management process, to select a version control tool and to start writing the required build scripts to build the product.

What is wrong with this approach? The team has failed to plan and in turn planned to fail. It is imperative to set the objective before taking any step further. Let us look at why the creation of a configuration and release management plan is necessary.

Planning and setting the goals
According to the 2003 Standish Group's CHAOS report, two-thirds of all software projects fail. Improper usage of configuration, build and release management process is one of the key reasons of such a high failure rate. Still, today we can find software product development teams which have no defined configuration management process in place. Some teams are using a version control tool but are not aware of the branching and tagging guidelines. Most of the teams are not using any automated build and deployment framework but are creating builds manually. Some Teams do not manage the release area and fail miserably when it comes to defining the common build and release management process for teams working in a distributed environment. The most common symptom of above mentioned diseases is that these teams have no configuration management plans in place.

You need to create a configuration management plan in the beginning of the project. The purpose of this plan is to create the processes, roles and procedures for managing the project assets or the metadata.

At the same time you need to identify the goals of build and deployment management and need to document these goals either within the configuration management plan or in a separate document.

A new product development team would like to start with a humble beginning by taking the baby steps first and then to move to high-end areas of automation of entire build and release process.
A reasonably simple objective for a new development team can be encapsulated as follows:

• A standardized change management process


• A standardized build creation process


• Adoption of a contemporary version control system


• Provide training to development team on branching and tagging guidelines


• Creation of build environment to facilitate builds on regular basis.


• Publish the deliverables and release notes so that stakeholders can pick the deliverables easily.

The above benchmark can ensure a favorable set of practices and discipline but still does not suffice if the expectation to build and deploy the product with a single click. Projects need a better goal setting to achieve such a high-end objective.
Let us add few more goals to the above mentioned objective.

• Creation of Continuous Integration Framework (setting up hourly/nightly builds)


• Execution of unit tests along with nightly builds and publication of the results of these nightly builds on the commonly used wiki in the organization


• Combining the test automation with nightly build process


• Automatic Tagging of source code at the time of each release build


• Extending the capability of build scripts so that the deliverables are automatically pushed to release area


• Extending the capability of build scripts so that the deliverables are automatically deployed in the staging environment


• Reduce deployment time to minutes from hours/days


• Creation of source code matrices along with nightly build. This might include statistics such as static analysis, dependency analysis and cyclomatic complexity


• Creation of release notes along with the build itself and publishing of these release notes on the preferred wiki.


• Maintaining the historical records of the build.


• Integrations between source code management system, requirement management system and build server so that each requirement is linked with the respective source code change and the build in which it is delivered.

Better? Let us add another challenge to it. What if software product development teams are distributed and working from different locations? The above objective might fail to suffice. We would have to look at some additional things

• Identify how to address the issues related communication issues which is the common and biggest challenge faced by distributed teams.


• Identify how the teams can effectively collaborate while merging their source code


• Decide whether to centralize or not to centralize  - How to support multiple sites and teams


• Share the build status reports, probably by publishing these reports on the commonly used wiki in the organization


• And so on...

The story does not end here. There is always the need to continue to innovate, automate and optimize the build and deployment process. Therefore the build engineer should continue to look for opportunities and should accordingly update the goal of build and release management.

At the same time, there is the need of proactive risk visualization and assessment plan. The objective is to get away from juggling and super-heroics at a later stage.

While preparing risk mitigation plan, do not just look at the challenges related to environment and tools usage but look at the behavioral aspect of risk as well.

In my experience not everybody in a project understands the real need of having an automated build and deployment system. The simple method to address this problem is to communicate with such people, understand their perspective and share the positives of having a common build and deployment process.

You should also look at how to calculate the Return on Investment (ROI) by automating build and deployment management process. Contrary to the expectations of few people, Cutting cost and building products at a very high velocity is the goal of build automation. And you should be able to identify a case when build automation is not required. If project is of a short duration and it is seems that the cost of build automation would outweigh the benefits then the build automation might not be necessary.

Time for a Case Study
Enough of sermons on planning and Goal setting! Let me share a real life example which might work as a case study to understand what I have discussed so far. Here I am talking about the project I worked on few years ago but it is still fresh in my mind.

When I joined the team as the build and release engineer, the project was somewhere in the middle and the reason why I was asked to join the team was that the former build engineer had submitted his resignation. I was given the responsibility not only to manage the current state of affairs but also to do some magic which would make the build management an end-to-end automated process.

It was not an easy task. I called for a meeting which was attended by the product owner, and the entire the project team. The agenda of meeting was to identify the pain points of the existing build management system, and to do some goal setting for the next few months.

Problem identification:

• The development team was working in parallel on three product versions.


• The product development teams were distributed and there were communication issues.


• Build creation process was defined although not automated


• A continuous integration tool was being used for hourly builds.


• A Nightly build (daily) was executed but this activity was happening by scheduling the execution of respective shell scripts.


• Creation of Release builds was a manual process which normally used to take an entire day.


• Intermittent build failures were seen on regular basis.


• Product deployment was extremely manual and took more than 8 hours


• Overall, the time taken to build, package and release the product was around 12 business hours or more.


• No build history was maintained.


• Branching and Tagging Strategy: Not everyone in the team was familiar with the concept which was a challenge. There was no standard branching and tagging guideline document for the team members to refer.


• The capability of build scripts was limited to just creation of executables. Rest of the process was completely manual.

Goal Setting
After taking inputs from all stakeholders, we identified certain objectives and used a phased approach for implementation.
Phase 1

• Creation of a standardized change management process


• Handholding the development teams on best practices of configuration management which included the following topics at high level
  
  
  • Branching and tagging guidelines
  
  
  • Source code committing  guidelines (within the private branch)
  
  
  • Code merges guidelines (when to do and how to do?)
  
  
  • Usage of a reasonable and sensible repository structure
  
  

• Setting up a cross platform build engine which would execute builds of all levels such as continuous integration, daily (nightly) build, QA and release builds.


• Integration of source code management system with requirement tracking system and training the development teams on how to use the issue tracker wisely


• Setting up configuration management audit controls at the time of each release.

Phase 2

• Extending the build engine capabilities to configure a highly automated build and deployment process which would cover the following

•  
  
  
  • Would move the deliverables to the release area at the end of the build
  
  
  • Automated deployment of the deliverables on various staging environments (located in various time zones) and to reduce the deployment time by automating each and every step
  
  
  • Automatic Tagging of source code at the time of each release build
  
  
  • Creation of release notes (automated) along with the build creation process.
  
  
  • Integration of build engine with requirement management system

• Extending the nightly build capabilities to include the test automation


• Extending the nightly build capabilities to execute the automated code reviews on the newly created change sets and publish the results via email and/or in preferred wiki.


• Creation of code coverage reports (along with the nightly build) which indicate the percentage of code covered by unit test cases.


• Creation of build dashboard in the preferred wiki.

We estimated that it would take around 3 to 4 months to meet the objectives identified in the meeting. We agreed that we would have a meeting at every two weeks to review the system and to seek feedback from the stakeholders.

Actions Taken:
Following are the high level action points taken during the entire process.

• Replaced the existing build management tool with a powerful cross-platform build management server.


• Trained the product development teams on the various configuration management practices and documented the required steps and strategies.


• Configured all types of builds including continuous integration build, (hourly/nightly) QE and release builds using new build management server.


• Written numerous build scripts which helped in smooth execution of various build automation activities.


• Builds simplified to a single step process and configured capability to schedule builds in advance


• Implemented automated builds notification process (emails, wiki etc.)


• Implemented process of deploying builds automatically in QE environment


• Ensured automated release notes creation along with build process; release notes were pushed to wiki


• Configured automated code review process (using automated code review tools) as part of build process


• Integrated source code management system, requirement management system and build management server which ensured that each requirement is linked with the respective source code change and the build in which it is delivered.


• Created a build dashboard in the preferred wiki which showcased the statistics related to build progress, unit test cases, code coverage and lines of code along with other parameters.

We continued to improve the goals as and when we found an opportunity to automate. Some of the interesting additions were following

• Addition of user guides in the build automation process. The build system first picked the respective documents and converted these documents (.doc files) into the pdf format before creation of respective packages.


• Notification of build failures to only those people who changed the source code since last successful build


• Lots of changes in the notification email messages (automated) to include relevant information such as time taken by the build, tag applied and the location from which deliverables could be picked.


• Automatic creation of the task in case of a build failure

The results:
Following are the key highlights of what we achieved in terms of return on investment of this whole exercise.

• A standardized and automated build and deployment management process which helped the team in expediting the build process


• The system helped in reducing the incidents of build failures by a fair margin.


• Release builds could now be created in less then 30 minutes


• Release notes published in the wiki were visible to all stakeholders and because these were automatically created there was no chance of any manual goof-up.


• Deployment time was reduced to less then 30 minutes


• The partner now had access to every build log and build outcome through a web interface


• Transparency was maintained in the entire system and everybody could pull any information as and when required.


• Creation of Build environment for new development streams was possible within few minutes.

Summary
Though might be sometimes challenging, but it is not hard to automate the entire build and deployment management process. Having an automated build process is critical to the success of your project and therefore can not be ignored.