DevOps is more than just a buzzword; it’s about delivering value to customers faster and more reliably. Jonathan Thorpe explains eight mistakes that can dramatically increase cycle time and slow the flow of releases to customers, which is counter to the goals of DevOps.
DevOps is more than just a buzzword; it’s about delivering value to customers faster and more reliably. Over and over again I have witnessed organizations that seem to go out of their ways to introduce practices that seem designed stop value being delivered efficiently. In the ten years that I’ve spent developing and releasing apps, I’ve learned that there are critical steps every IT organization should follow that ensure DevOps success.
While many of these may seem obvious, experience has taught me that at least one or more of these steps are overlooked. In this article, I will explain eight mistakes organizations make that can dramatically increase cycle time and slow the flow of releases to customers, which is counter to the goals of DevOps.
1. Not Planning For Scale
Think long term when you embark on the process of bringing development and operations closer together. Create processes that will scale from day one. An example of a process that won’t scale is having a single gatekeeper to approve all changes in an automated process. Having a single person able to approve work may work with a few releases per year but that person will become a bottleneck if there are many releases per week. If you’re successful, you’ll need to scale faster than you ever imagined. Not planning for scale is a fundamental problem I’ve seen over and over again. Simply put, even if you have amazing success when there isn’t as much work going through the system, if your processes don’t scale the processes will quickly become a bottleneck. Allies may quickly become disillusioned and your DevOps initiative may be seen as something that only works for small teams.
2. Not Preparing Your Codebase For Frequent Releases
Another problem I’ve witnessed, even in projects using the latest technology, involves the codebase. All too often there are a few key files in a codebase that need to be worked on for many, many changes. These can cause significant merge problems, which can result in time being lost on a daily basis. Refactoring is key to moving forward at this stage. Cries of “I can’t make this change because it will affect x,y,z," and "It will take several weeks/months...” are strong indicators that a codebase may have problems that can be solved by refactoring. Stakeholders must be involved when the product or feature needs to be refactored; the change involved can take time and isn’t without risk, although this risk can be reduced by having unit tests.
3. Not having Extensive Unit Tests
I’ve been astounded by how many times I’ve witnessed teams doing continuous integration (CI) without having unit tests. A CI build only proves that the codebase still compiles. If you don’t do unit tests, your automated system tests aren’t proving anything. And if you’re doing manual tests executed on a build that would never have passed unit tests, you’re just wasting valuable time and resources
4. Not Designing Automated Tests To Scale
A common problem encountered when writing automated tests is that before long test suites take too long to execute. Even seasoned veterans mistakenly create test cases that are dependent on the outcome of other tests. Tests with dependencies on the outcome of other tests do not scale. Soon, test suites that used to run in minutes will take hours to execute, and as the test coverage goes up, so does the cycle time.
Fortunately, many test cases can be written so that they’re not dependent on state from previous tests. While this isn’t always possible, tests shouldn’t have any interdependencies. When done with virtualization, many tests can be executed in parallel by spinning up multiple environments and executing tests across machines in each environment. This can result in massive time savings and help scale your DevOps solution.
5. Not Managing Your Capacity
Virtualization, including use of the public cloud, can provide enough capacity to service the needs of the development teams, but a common complaint from R&D and QA is that there is never enough capacity. Virtual machines (VMs) are so easy to create that they are built without a second thought. Unfortunately, VMs are frequently not destroyed after use, which quickly eats up capacity and leads to performance problems. Make sure you clean up after yourself. Use automation that allows for VMs to be either destroyed or restored to a clean snapshot and shut down when the VM is no longer in use. A common use case might mean that if QA tests are successful, a VM can be destroyed or reverted back to a clean snapshot and shut down. If the QA tests fail, VMs can be left running or snapshot until there is time for investigation before being destroyed or cleaned up.
6. Not Using Your Resources Effectively
Spinning up a VM automatically is useful, but what’s even more useful is configuring each VM appropriately and putting the latest release onto the system. Use deployment automation tools like Puppet and Chef to ensure that the base VM is appropriately configured. Then use application deployment automation to ensure the latest release is installed on the newly created environment.
7. Not Using One Set Of Deployment Processes For All Environments
How can the software your team is building be reliably deployed to all environments? Custom deployment scripts for each environment won’t help out with the situation. You should execute the same deployment process over and over again so that by the time the product ends up in production the deployment process will be executed many times; model-driven deployment it key to this. The application model should be aware of the environment it is being deployed to. This way, the appropriate values are inserted into the deployment configuration for the environment being deployed to. The end result is that by the time an application is deployed to production the application deployment model has been tested many, many times.
8. Not Knowing What, Where, When, Who, and How
A CI system such as Jenkins could be used to perform a build, spin up environments, deploy software, and execute tests for each environment. However, a CI system is primarily meant for small-scale builds and deployments.
The problem with enabling this level of automation and self-service is that we are left with many unanswered questions: How do we know what build is deployed to where? What’s in the build? What were the test results? There are many other questions that could be raised, mostly related to what can be done with all of the data produced by a large number of builds, deployments and tests, all potentially being executed in different tools.
Having a higher-level business process model allows for this information to be tracked much more easily and drive future decision-making. By adding business process orchestration and having the higher level process execute the CI build, you can start a deployment, run tests, and easily gather and process information about how each build in production made its way to production. Audits become much simpler, and people working in development, QA, and operations will all have a common set of data. From this common set of data staff at all levels can see what is in a release and how the changes in the release have been managed throughout the entire process.
Summary
Automation is the key to a successful DevOps initiative; it reduces cycle time and moves products to customers faster. But without a good process management and reporting framework, the amount of releases that can be pushed through a system can quickly get out of control. Providing a layer of orchestration that can report on data throughout the entire lifecycle of a release makes it easy to provide metrics that show how your DevOps initiative having a positive impact on business goals.
