In his Behaviorally Speaking series, Bob Aiello discusses hands-on software configuration management best practices within the context of organizational and group behavior.
Much of what is written about DevOps—a set of principles that helps development and operations teams work more effectively together—is delivered from the perspective of developers. In my opinion, DevOps needs to also take an operations point of view in order to be effective and practical. This article is all about putting the “ops” back into DevOps, so to speak.
Operations professionals are responsible for ensuring that IT services are available without interruption or even degradation in services. IT operations is a tough job and I have worked with many technology professionals who were truly gifted in IT operations with all of its functions and competencies. Many IT operations staff perform essential, albeit repetitive, day-to-day operations tasks that are essential to keep critical systems online and operational. In some organizations, mainframe operators are not as highly skilled as their development counterparts. When developers observe that operations technicians are not highly skilled they often stop providing technical information because the developers conclude that the operations technicians can’t understand the technical details. This dynamic can result in disastrous consequences for the company.
I have also worked with top-notch Unix/Linux gurus in operations who focused on keeping complex systems up and running on a continuous basis. IT operations professionals often embrace the IT Service Management Forum (itSMF) ITIL v3 framework to ensure they are implementing industry best practices for reliable IT services. If you are not already aware of ITIL v3, you probably should be.
The ITIL v3 framework describes a robust set of industry best practices designed to ensure the continuous operation of IT services. The ISACA Cobit and the SEI CMMI are also frameworks that are used by many organizations to improve their process along with both quality and productivity. CM professionals should particularly focus on the guidance in the transition section of the ITIL pocket guide, which describes change management, build and release, and configuration management systems (including the configuration management database, or CMDB). With all of this guidance, do not forget to begin with an understanding of the application and systems architecture.
The first thing that I always require is a clear description of the application and systems architecture. This information is not just for my entertainment. For build and release engineers, understanding the architecture is fundamental because all of my build, release, and deployment scripts must be created with an understanding of the architecture involved. In fact, development needs to build applications that are designed for IT operations.
Many developers perform on test-driven development (TDD) where code is designed and written to be testable, often beginning with writing the unit test classes even before the application code itself is written. I have run several large-scale automated testing projects in my career and I have always tried to work with the developers to design the systems to be more easily testable. In some cases this actually included hooks to ensure that the test tools could work without finding too many cosmetic superficial issues, which we usually call false positives. Test-driven development is very effective and it is my view that applications also need to be designed and written with operations in mind. One reason to design applications with IT operations in mind is to implement IT process automation.
Effective IT operations teams rely upon tools, including the automated collection of events, alerts, and incident management. When an alert is raised or an incident is reported to the IT service desk, the IT operations team must be able to rely upon IT process automation to facilitate the detection and resolution of the incident. IT process automation must include automated workflows to enable each member of the team to respond in a clear and consistent way. In practice, it is very common for organizations to have one or two essential subject matter experts who are able to troubleshoot almost any production issue. The problem is that these folks don't always work twenty-four hours a day, seven days a week. IT process automation, including workflow automation, enables the operations team to have well-documented and repeatable processes so that IT services are consistently working in a reliable and consistent way. Getting these procedures right must always start with the application build.
Effective build automation often includes key procedures such as embedding immutable version IDs into configuration items to facilitate the configuration audit. For example, a C#/.net application should have a version identifier embedded into the assembly. You can embed version IDs via an MSBuild script or using Visual Studio and ClickOnce. The Microsoft MSIL Disassembler (Ildasm.exe) can be used to look inside of a .net assembly and display the version ID. You can use similar techniques in Java/C/C++ along with almost every other software development technology. IT operations staff need these techniques to be able to confirm that the correct binary configuration items are in place and that there have not been any unauthorized changes. Builds are important, but continuously deploying code very early in the development lifecycle is also a critical DevOps function that helps IT operations be more effective.
Additionally, application automation is a key competency in any effective DevOps environment. Continuous delivery enables the IT operations team to rehearse and streamline the entire deployment process. If this is done right, then the operations team can support many deployments while still maintaining a high level of service and support. The best practice is to move the application build, package, and deployment process upstream and begin supporting development test environments. These automated procedures are not trivial and it will take some time to get them right. The sooner in the lifecycle you begin this effort, the sooner your procedures will be mature and reliable.
Having effective IT operations is key to any DevOps transformation within an organization. It is all too common for development to miss the importance of partnering effectively with operations to develop effective procedures that guarantee uninterrupted IT services. If you want to excel, you must include the “ops” in your DevOps endeavors.
