In this article the author discusses the time-tested Hub-N-Spoke Release Management process that ensures a holistic view of the code (new or modified) is considered before releasing it into the production environments. This process was used successfully by data management groups of a large financial company. Subsequently the use of this Release Management process resulted in improvements in the code release process.
The Viscious Problem of Bad Code Releases into Production Environment
How many times have we come across situation where the code is developed, passed the QA but somehow doesn't seem to get into the Production Environment, only to realize it was a minor documentation glitch or a missing QA report or an ill-coordinated code release schedule. We end up cutting a sorry figure in front of the Change Control Board which eventually brands us as not so production reliable.
Hence we end up in the viscious cycle of painstaking scrutiny by the Change Control Board which results in lot of re-work effort by the development, QA and Release Management teams.
In this article we discuss a unique Hub-N-Spoke Release Management Process which is not only time tested but also ensures that a holistic view of the code (new or modified) is considered before releasing it into the production environments. This process was used successfully in one of the Data Management groups of a large financial company. This company was struggling with frequent Change Request rejects and large number of Trouble Tickets being raised due to poor quality Software Code releases into Production resulting in increased re-work effort. Subsequently the use of this Release Management process resulted in improvements in the code release process.
The Hub-N-Spoke Release Management Process
The Hub-N-Spoke Release Management Process is explained in detail in this section. The scope of this Hub-N-Spoke Release Management (RM) process is limited to the release of the software code from the development team into the Production Environment, post the approval of the QA team. Further the scope is restricted to the Data Management environment.
This RM process ensures that the Software Code released is consistent, cost-effective and a delightful experience to the software community in general.
Creation of a Separate Release Management Team
This process suggests the creation of a separate team for performing the RM function. The RM team is mainly focused on effective job releases into Production and is not very much focused on the internal details of the software code to be released. The RM team is supported in it's operations by various pre-defined checklists and dashboards.
In order to have a holistic view of the code to be released the RM team coordinates with various other teams like the development team, QA team, Production Support team, DBA team, Change Control Board, Operations team.
Let us now discuss the sequence in which the RM team coordinates with these various teams in eight stages to ensure that the release process moves from one stage to the next stage seamlessly, accurately and consistently.
In Stage 1, Once the software code is given the approval by the QA team , the Development team passes this code to the RM team along with the associated release documents and artifacts (as per the defined standards).This is followed by a Knowledge Transition (KT) session between the Development team member and the RM team member to understand the basic details about the software code to be released.
In Stage 2, The RM team gets an approval from the QA team about the software code to be released. This approval focuses on the quality/defect aspects of the software code.
In Stage 3, The RM team gets the approval of the Production Support team w.r.t the impact of the release of the software code into the Production Job Streams, of the applications that the Production Support team is responsible for. This approval focuses on how this software code is going to be positioned in the Job Dependency diagram and also how it is going to behave in the given settings and planned run schedules.
In Stage 4, The RM team gets the approval of the DBA team w.r.t the database impacts of the release of the software code into the Production environments. This approval focuses on the performance aspects of the software code at points where it interacts with the database.
In Stage 5, Once the RM team gets an approval from the QA , Production Support and DBA teams , the RM team presents the release of this software code to the Change Control/Request Board (CCB) for approval of the release of this software code into production. This approval focuses on the Enterprise level impacts of the release of this software code into production.
In Stage 6, The RM team performs the relevant Configuration Management processes for complying with the Release after the Change Control Board approves the release of the software code into Production.
In Stage 7, The RM team co-ordinates with the Operations team to release the software code into the Production environment.
Shown below is the RM process flow diagram:
Finally in Stage 8, Post the release of the software code into Production, the RM team (in conjunction with the Development team) monitors the behaviour of the newly released Job/Software code in Production for a period of "N" consecutive successful runs. After "N" successful runs, the RM team flags off the release as a success and hands over the support of this job/software code (in conjunction with the Development team) to the Production Support team.
The RM team profile
The RM roles that can be defined are Release Coordinator , Release Manager.
The RM responsibilities can include:
- Being well versed with the various documentations and artifacts needed to be presented to the Change Control Board
- Represent the Software Group to the Change Control Board
- Enforce the various Release Guidelines and Processes internally with the Development , QA , Production Support and DBA teams
- Interact with the Operations team for the physical code release process ( i.e migration processes ) into the Production environment
- Being well versed with the various Operating systems , the programming languages and software that is being used to develop the software code
- Equipped with positive and aggressive communication skills
Tracking the RM process
The RM team can track the status of the various software codes being released into Production by using a Dashboard. This Dashboard can give information like Name of the Software Code, The Developer who developed it ,The Software Group he belongs to,The Development team member who gave the KT to the RM team member ,The RM team member who performed the RM process The approval status of the QA, DBA, Production Support teams, The approval datetime and status of the Change Control Board ,The status of the software code after the release/migration into Production, The status/history of the software code until the "N" successful runs are achieved, The Handover status of the software code to the Production Support team.
The RM team also carries out a periodic analysis of the status of the various software code releases that were done. Based on the analysis, the RM team gives a constant feedback and learning experiences, back to the Development, QA, Production Support and DBA teams, which in turn helps them to improvise their internal approval and documentation processes.
Critical Success Factors for the successful implementation of the RM process
The most important factor for the successful implementation of the RM process is the management buy-in and support for a dedicated staff to perform the RM function.There should be a willingness of IT staff to perform this role, which can be mitigated by following a job rotation policy. Strict adherence to the RM process by the RM team, against the pressures from the Development teams is essential.The cooperation of the Development, QA, Production Support and DBA teams in giving their approvals to the RM team for the release is very much needed. The incremental and iterative Knowledge base that the RM team would gather over a period of time w.r.t the release process can ensure faster, accurate and consistent code releases into the production environment.
The feedback mechanism to the Development QA, DBA and Production Support teams about the trends in the software release states will help them to improvise their internal approval processes.
Benefits of this RM process
a) A very consistent release process as this release function is being done by a dedicated team which is well versed in the technology and human aspects of the Software Code release process
b) A dedicated RM team ensures that an exhaustive internal check is done on the software code to be released, before being submitted to the Change Control Board. This minimizes the expensive Change Request Rejects and the associated delays in releasing the software code into production.
c) All changes to the Change Control process is communicated by the Change Control Board to only one focused group which is the RM team. This ensures that the communication is complete , accurate and latest. This again ensures that the Change Requests Rejects are minimized. Also the dedicated RM team can spend more time to educate it's Software teams about the changes to the Change Control Processes
d) The analysis and feedback about the software code release states, by the RM team to the development, QA, Production Support and DBA teams led to further constant process improvements internally within these groups.
e) The direct monetary benefit is in:
- the reduced trouble tickets (which are raised due to poor quality software releases into the production) and hence savings in the maintenance costs
- reduced delays in the software code releases due to consistent RM processes and hence savings in the development variances (effort and schedule)
- The Development teams relieved of the operational aspects of the software code releases thus giving them more time to focus on the software programs and algorithms
- The Production Support teams have to spend less time doing fire-fighting as they are involved with the software code, much early in the release process
- The DBA teams can be better prepared for any rogue queries much earlier in the release process, thus avoiding surprises and hence expensive database downtime
Conclusion
The Hub-N-Spoke RM process can provide a cost effective, consistent, process-oriented and metrics-driven approach to software code releases into production.This is done by involving the various support teams like the QA, Production Support and DBA much earlier in the Release Process. In fact, with a strong management support the RM process can be the most effective and powerful governance mechanism in the Software Code release process evolving into a central hub for all software code releases.
