Jesse Dowdle sat down with us recently to discuss the benefits that embracing "massive" continuous integration and deployment brings to testing. Jesse is presenting the session "Massive Continuous Integration and Light Speed Iterations" at the 2012 Better Software Conference East.
Senior manager of development for AtTask Inc., Jesse Dowdle directs the engineering efforts of nearly a dozen agile teams in the U.S. and overseas. Successful engineering in a fast-paced industry requires constant research in emerging technologies. Jesse drives thought leadership on continuous delivery, development best practices, and process evolution to help AtTask ship software at lightning speed.
Noel: I read where you said, "AtTask decided to rapidly rethink their development model." What brought on that decision, and how can other companies benefit from doing the same thing?
Jesse: AtTask is software as a service, a model that has exploded over the last few years. In the early days it was typical for a company to deploy monthly, and our infrastructure and application stack were built to that expectation. As the industry was progressing, our company was growing as well, so we were experiencing the pain of multiple code branches, less than stellar automation, etc. Ultimately, we were having trouble hitting our monthly releases.
AtTask never does anything small: it's kind of a cultural thing for us. Instead of just fixing the monthly deployments, we decided to shoot for a big hairy goal… what if we deployed every day? That goal is what led us down this path.
Noel: Continuous integration appears to rely heavily on directed, constant communication. What advantages does this create, and what are companies lacking without it?
Jesse: Continuously integrated builds, particularly at scale, work best when the results of the system have high visibility in your organization. This visibility creates a terrific rallying point. We think there's a lot of value in knowing, at any given moment, what the quality of your build looks like.
When you couple that with automated deployment scripts (which we also have), you create a level of flexibility within your development group to respond to changes in production within minutes. That level of agility has lots of power. We use it to split-test features, hot-fix high visibility defects, measure performance impact, and a host of other things. Ultimately, we can move more code around faster than our competitors.
Noel: For those wonder if they can afford "massive" continuous integration, what are the costs involved, and how can companies keep those costs down?
Jesse: Building out an automated system is all about trading fixed cost for elastic cost. We used to take our team off-line for a week to do the QA required for a release. Doing this monthly meant we were spending 8% of our payroll annually just certifying builds. Once we went to a fully automated cloud-based system, we were able to reduce the time to certify a build to about an hour, and required the attention of just a couple engineers during that time.
The primary cost (after investment in automation) is the cloud infrastructure to run your tests. We use reserved instances and small machine sizes to keep it affordable. Additionally, all of our automation is designed to tear itself down when its job is done, that way the meter is never running on unused compute. The whole system is fully elastic.
Noel: With available tools and technologies constantly being introduced to the software developing world, how do you see continuous integration becoming even more effective in the future?
Jesse: It's becoming easier to stand up a complex application stack with all of its dependencies in a virtualized way. Being able to replicate your stack (or parts of it) is critical making something like this work. With companies like Amazon investing in tools to simplify the process, I think it's going to become a standard practice, just like version control systems a decade ago.
Right now some of the most popular open-source build systems (like Jenkins) have absolutely horrible user experiences. I would expect to see more iteration in the area of look and feel on these systems as people realize how critical the visibility is. This will in turn make CI much more effective for organizations that deploy it.
Noel: What's the best way for those who want to incorporate massive continuous integration into their own companies to start the process?
Jesse: You will need a fairly high degree of test automation right up front. You'll need to invest in a build system (although most engineering shops these days have one already). If you're trying to do this at scale, you're probably going to need a cloud provider or a lot of hardware lying around.
Forming a small team to pull together the different threads and build a prototype can be a great way to begin the process of building a scalable CI system.
Ultimately, keeping continuously integrated is a way to push more customer value into production. This is possible because you always know the state of your build, and you free yourself to retire technical debt, as well as keep lots of engineering teams all working on the same codebase with minimal down-time for reintegration.
If you are struggling to hit your delivery dates because you have uncertainty in the QA phase of your software development lifecycle, going with a continuously integrated system is a fantastic way to close the gap.
