Electric Cloud, a leading provider of software production management tools, has always employed an iterative, agile approach to its software development. But like many fast-paced technology companies, a script-driven, manual approach and once-daily integration builds could not scale effectively to support its growth. Through a mix of tools and best practices, Electric Cloud was able to deploy an end-to-end build and change management system with fully automated continuous integration (CI). This new software production system has reduced the time Electric Cloud engineers spend on software production by more than 30 percent and has saved the company an estimated $700,000 per year.
Introduction
In most ways, Electric Cloud is a typical Silicon Valley start-up. We have experienced 2X year-over-year growth, mounting pressure to meet market demand for our popular software, and the well-known development "crunch" -- where the need to develop technology faster and with few resources can challenge even the smartest engineers and systems.
Electric Cloud is perhaps unlike many start-ups, because it specializes in developing software for software developers -- specifically, tools for software production management (SPM), or the critical "back-end" of the development process, where code is built, tested, and deployed. Despite this expertise, our development team initially struggled with a manual, script-driven software production process. To improve development productivity, visibility, and quality, we developed and deployed a CI and validation process that ties together automated builds, software configuration management (SCM) and defect tracking.
Situation
By using our own ElectricAccelerator® build acceleration solution, we have always enjoyed very fast builds, but our developers still had to contend with a script-driven, manual approach and once-daily integration builds, which proved not to scale effectively with company growth. "Like most development teams, we were using homegrown scripts and open source tools for our software build and release processes," said Anders Wallgren, Electric Cloud's CTO. "This was acceptable in the beginning with only a handful of developers, but as our company and products grew, this system became more and more limiting. It was having a negative effect on our overall productivity."
Our homegrown build scripts quickly became strained as the company grew. Initially, the script could only build one version of the product at a time, so "adding a lane," or a new branch, was a major undertaking. Moreover, only the developer who wrote the scripts could understood them completely, making the learning curve for newly hired engineers a long, steep climb.
{sidebar id=1} Broken builds were another recurring issue with script-driven builds. Full of new code that may not have been fully tested prior to check-in, the daily integration builds often had problems, which could cause delays of several hours. Moreover, when something broke, the "black box" system offered very little visibility into the root of the problem, so our engineers spent literally hours parsing log files trying to track down and fix each issue -- resulting in further delays.
Our software production problems were not unique, however. The vast majority of our worldwide customer base is grappling with similar issues -- software production systems that become increasingly problematic as software development grows and becomes more complex. Through our work with developers across a number of industries, in addition to our own internal experience, the problem became crystal clear: building and testing early and often is critical to development productivity. We needed to address these tasks from both a best practice and tool perspective.
To address the widespread customer need for an automated, enterprise-class build management solution -- and at the same time, improve our own production system -- Electric Cloud developed and deployed ElectricCommander.
Solution
In 2006, we deployed an early version of ElectricCommander to make our own software build and test tasks more efficient, visible, and repeatable. ElectricCommander uses a three-tier architecture and AJAX-powered Web interface, along with build-and-release analytics for reporting and compliance.
In the four years since we started using our initial production scripts, we have grown from a one-product to a three-product company -- each with multiple releases to support. We had always employed an iterative, agile approach to development, but as our development grew larger and more complex, it became clear that we needed a new approach that included automated CI.
We developed an integration between ElectricCommander and the Perforce SCM system to enable CI with no need for manual scripting. In addition, we utilized a Perforce integration with the JIRA defect tracking tool, along with ElectricAccelerator, to facilitate fast, efficient CI and verification. In essence, this integration allows our team to associate issues with Perforce changelists or link them directly with Perforce jobs. When code is checked into a specified depot within Perforce, ElectricCommander invokes a build/test procedure. Perforce then updates the issue as appropriate ("in progress," "complete," etc...).
The combination of these tools provides an end-to-end build and change management system. In this way, our developers are able to get rapid feedback, find and fix errors early in the process, and deliver software in an agile fashion.
Results
Since deploying this system along with additional best practices, our software production process is more automated, scalable, visible, and has a measurably shorter production cycle times.
Production Time Reduced by 30 Percent
"What used to take six months, now takes us less than four months," said Wallgren. "This time reduction alone has saved us an estimated $700,000 per year, notwithstanding the 'soft' benefits of predictably meeting deadlines and having the confidence that we are consistently producing highly reliable, top quality software."
A major contributing factor to our efficient system is the notion of parallel builds. With the script-driven approach, our development team only had two "lanes" for builds (where only two builds or procedures could be executed at a time). With the new system, the team can easily manage four lanes, meaning it can do four simultaneous builds at one time. This has allowed us to develop versions faster and support our expanding business. This infrastructure also lets the team take the next step beyond CI, providing the bandwidth and automation needed for "pre-flight" developer builds.
CI has gained favor among developers in part because they gain rapid, frequent feedback on their work and thus reduce the likelihood that they will introduce errors during an integration build. Automated pre-flight builds, where developers are able to run their own builds and a reasonable set of tests prior to code check-in, takes that benefit a step further. Integration errors are identified as quickly as possible without affecting production builds, thus bringing validation even earlier in the lifecycle. Our developers are responsible for writing their own unit tests and some system tests, and they run those along with the build prior to checking code into Perforce. If the build breaks or these tests fail, they have not impacted the rest of the team and can rapidly fix the problem. Developers can run pre-flight builds and tests on more than just their local machine, thus ensuring they have covered all necessary targets/platforms.
Pre-flight builds and tests are now a best practice at Electric Cloud and among many of our customers. We've seen that this approach is directly related to better software quality and ensuring we meet product delivery dates.
Improved Visibility and Reporting
Automating the build and change management process has provided powerful development intelligence that allows us to keep a pulse on our entire build and release system -- often considered the "heartbeat" of development. We have new visibility into our software production, allowing us to quickly answer questions such as, "How many tests were run today?", "Which teams are building successfully?" and "How many clean builds have run this month?" Before, it could take weeks to get simple reports on trends over time. Now, just a few clicks and it's done.
Moreover, we now have a shared platform for disseminating best practices and reusing common procedures, so all the company's development processes are documented, repeatable, and no longer a mystery to newly hired engineers. As a result, we have been able to integrate new engineers into the build and tests system in a matter of hours rather than days or weeks, which further supports our growth.
Conclusion
Electric Cloud has successfully replaced brittle, unscalable software build scripts with an enterprise-class, end-to-end change and release management system. The benefits are quantifiable. By using ElectricCommander and adopting best practices including CI and pre-flight builds, we have reduced the our software production time by over 30 percent. We are saving an estimated $700,000 per year in addition to improvements in productivity, reliability, and visibility.
About the Author
Martin Van Ryswyk is the vice president of engineering of Electric Cloud, a leading provider of Software Production Management (SPM) solutions. Prior to joining Electric Cloud, Martin was a senior director at EMC Corporation, where he managed a number of software development teams. He was also the vice president of engineering at Luminate (acquired by EMC in 2001), a privately held software company that pioneered the delivery of systems management software as service. Previously, he held management and technical roles at Tidal Software.
