From Build Management to DevOps: Lessons Learned

In 2010, I joined HP Software’s research and development team for application delivery management to form and lead our DevOps team, providing solutions for development, build, and release pipelines. Focused on HP application lifecycle management, I arrived in a highly dynamic environment, full of ideas but with little time to invest to turn those ideas into reality. The challenge was obvious: The team was working across multiple build systems along with a plethora of tools, conditions, and approaches. A vision was needed to move forward and be successful.

Taking the Agile Approach

I knew we needed to adopt agile in order to first be able to deliver small changes in minimal intervals and to build on top of them. This would fuel the appetite of all relevant stakeholders to show how agile speeds release cycles while increasing quality. Agile is a modernized approach, and as my team expanded, it allowed for a DevOps team to work with and for business stakeholders.

We engaged with potential stakeholders right from the start. Developers, project managers, group leaders, architects, and QA managers are all useful in determining some common challenges organizations are struggling with:

• Aligning development and build environments
• Streamlining the process of product creation
• Creating a robust, trusted, repeating process that can scale up to meet future demands
• Using best practices and tools to deliver artifacts
• Transparency and clarity

Out of the above pain points came a vision: one process—by developers, for developers, with developers.

Getting from Then to Now

Understanding the environment and challenges and armed with our new vision, it was time to begin the journey by focusing on a few strategic areas. We wanted to research industry-leading tools and adopt them; automate anything that could be automated; and make stakeholders part of the process, not just the end-users, which meant making sure stakeholders are at every possible checkpoint, from requirements to delivery.

We chose Git as our source management tool. We wanted Git to branch off the centralized model of Apache Subversion and manage our own repositories, on top of providing a performance boost due to the distributed nature of Git and the offline work mode it offers.

For a build tool we chose Hudson, as many of our developers are Java natives and it seemed like a great starting point for the “by developers, for developers” part of our new motto. Once Jenkins split from Hudson, Kohsuke Kawaguchi continued to develop and support it, so Jenkins was selected. For build management, we selected Apache Maven to align efforts with development and drive the same flow. Getting funds to create a production environment for the build was an important decision backed by management, which let us expand and scale up quickly.

We used the continuous delivery model for on-premises as well as software as a service products in order to shorten DevQA cycles, add testing and automation as frequently as possible, and ensure better quality of the product.

We employed a three-layered build approach:

• Continuous integration (CI), which gives fast feedback to developers and contains compilation, unit tests, and a small subset of sanity tests and occurs every five minutes (any commits in that time period are being evaluated)
• Continuous build, which is based on the last successful CI build and runs a more thorough test suite (such as REST tests and schema upgrade tests)
• Continuous delivery, which is based on the last successful continuous build and runs every three hours with a complete set of tests

After choosing the right toolset for us, we ended up with the following processes.

Continuous integration:

Continuous delivery:

What We’ve Accomplished

As a result of our DevOps approach, our developers have added a “security blanket” in terms of automation and reliability in the process. Developers now actively contribute to the ever-growing automation framework and pursue solutions to failed builds. This benefits the team’s overall experience by:

• Reducing total cost of ownership for the company
• Increasing developers’ velocity and making sure they focus on the important things: coding and innovation
• Producing quality builds and products
• Collaborating on knowledge, tools, and practices for everyone’s benefit

We have an internal instance of ALM holding about a hundred gigabytes of real data (our own requirements, defects, reports, and so on), which we are now able to automatically upgrade with every sprint. We have a staging instance of the same ALM, which is being upgraded nightly for rapid development by our teams. They can see their changes on a nightly copy of the production server.

By automating hundreds of tests, we have reduced the overall time of execution by removing outdated, long tests and replacing and refactoring tests to run swiftly and in parallel. We reduced continuous delivery build cycle time from eight hours to three, full testing framework time from six hours to two and a half, and delivery time (and deployment of a working sprint release) from once every major release to every two-week sprint.

Lessons Learned

Stabilizing the process took time, and the team learned some valuable lessons during that period—mainly that you have to monitor your own process.

You shouldn’t rely on IT for centralized DevOps services. Git, Stash, Nexus, and Jenkins can’t be handled by IT because that department can roll out infrastructure in a large scale, but it can’t compete with the swiftness and agility of research and development organizations.

We also learned to decentralize everything possible and rely on our own resources. We still work with a centralized IT, but DevOps is a distributed model necessitating collaboration between teams and conducted on a culture basis, not on the infrastructure basis.

Everything is code. We use source code management to control configurations, environment setups, conventions for release names, version numbers, Nexus label schemes, and anything dealing with code.

Going forward, we are working on adopting advanced continuous integration models. Currently, we are working on a gated check-in approach to ensure our protected release branch is always green.

DevOps is a journey and a culture. Investment in infrastructure will give a much larger return on investment. (As a former manager of mine repeatedly said, “Every dollar I invest in infrastructure is ten dollars I get in product quality.”)

The passion to make a change, not being afraid of skeptics or bumps along the road, and engagement and collaboration from developers will lead to successful implementation.