Why Testers Need to Code: An Interview with Simon Stewart

Josiah Renaudin: First off, please tell us a bit about yourself and your role at Facebook.

Simon Stewart: I’m a software engineer in Facebook’s London office. My original team worked on developing the APIs and infrastructure we needed for automated testing of our mobile apps on Android and iOS. We ended up using and contributing to the open-source projects “Selendroid” and “iOS-driver,” and we now run a stable subset of our end-to-end tests for every change we make to our Android and iOS apps. It’s a great team to work with and, thanks to their work, we’re in a position to use the same set of abstractions we use for our “correctness” tests for other purposes, such as performance testing.

I’m now focusing on our build tool --- “Buck” --- which, again, is open source. Buck started life as being an amazingly fast way to build Android applications, and we’ve been working hard to extend it to support not just Java and Android but also native code development for our back-end services and iOS.

I’m sure you’ve seen the XKCD cartoon with two developers fencing on chairs explaining that they’re not being lazy, they’re waiting for the code to compile. The Buck team’s goal is to make sure that Facebook’s developers never get a chance to use that excuse!

Larry Wall once wrote that there are three attributes shared by all great programmers: laziness, impatience, and hubris. Taking each of those, in this case “hubris” refers to wanting to write programs that no one wants to say anything bad about. “Impatience” is about the desire to have the computer react to your needs as close to instantly as possible. “Laziness” refers to wanting to reduce energy to complete a task: You’re willing to work really hard to solve the problem in an automated way so that you never need to expend effort to solve it again. I guess you could say that the theme of the work is to allow developers to be lazy, impatient and hubristic.

I am also the lead of the Selenium project, and created WebDriver, which came out of the tribulations building and verifying software that the teams I worked with went through. There’s nothing like first-hand experience to reinforce the lessons about what’s important and what’s not. Right now, the W3C are working on “WebDriver” specification, supported by Google, Microsoft, and Opera amongst others. Although we’re working hard to get to Last Call it’s a ways off, but there are already implementations for Chrome, Firefox, and Internet Explorer. It’s been absolutely wonderful seeing the browser vendors add support for easier automation to their browsers; I feel it’ll make the lives of many developers and testers a lot easier.

Josiah Renaudin: Can you explain just how big a piece of the Facebook pie the mobile department is? What percentage of Facebook users access your app, compared to its web companion?

Simon Stewart: My impression is that Facebook does things very differently from other companies, and I think that makes it an interesting company to work for. We don’t have a “mobile department” since we found that hard to scale appropriately. Instead, teams working on features, such as photos or the News Feed, own that feature on every platform we support, from the mobile web, “traditional” desktop browsers, through to mobile platforms such as iOS and Android.

This approach means that when a team releases a new feature it happens at about the same time on every platform. As an Android user, I used to find the lag in features after an iOS a source of bemusement, so I think that the current approach works out better for most people.

Having said that there’s no mobile team, there are, of course, individuals who are particularly familiar with the individual platforms, and there are small teams dedicated to providing common infrastructure on mobile that all the feature teams can use, just as there are teams working on providing infrastructure for use by our web developers.

I’m not sure of the exact numbers of people using the app as compared to the web, but I do know that mobile traffic is increasing and in some countries surpasses desktop usage. As the shift from desktop to mobile has been happening, we’ve been working to make the apps perform better under the constraints of mobile. The first big move was our shift from using web views (which allowed us to iterate really quickly) to writing native apps (which, certainly at the time we made that choice, gives people a smoother way of using Facebook). The improvements we’re making now may be less obvious, but we have automated tests which track things like power consumption, memory and CPU usage, and how we use bandwidth, the goal being to improve (or, at least, hold steady) all of those metrics with each release.

Josiah Renaudin: How do you determine what you want to focus on with each Facebook update? What’s that process like?

Simon Stewart: There are a number of different models for how to do software releases, no matter whether it’s on the web or an app to mobile, but they all play with three factors: time, features, and quality. Naturally, quality should always be pegged to “as excellent as possible,” so that leaves a choice between choosing to release when a suite of features is ready, or doing time-based releases.

The feature-based releases seem appealing on the surface, but prove problematic to deliver consistently. After all, when was the last time you saw every software project at a company meet its planned ship date with everything working as expected? So releases get held up as some features are finished before others, and sometimes features need to be bumped as priorities change.

All that means that we do time-based releases. Our release cycle has the app ready to ship every four weeks, though it might take longer than that to get into people’s hands because we also need to get into the app stores. During those four weeks, every day we push a new build of the app to “dogfooders” within the company (a charming phrase, coined by Netscape, which describes the process of “eating your own dogfood” --- you naturally want it to be as tasty as possible). Every two days on Android, we push a version to the “alpha” channel, and once a week to the “beta” channel, too. Finally, all Facebook staff are encouraged to try out the “release candidate” builds. That means that by the time our app lands on your phone, you can be sure that it’s been given a thorough test drive.

As well as all these builds, we also run a battery of automated tests against every build we do. In fact, we also run most of those tests on every single commit to our code base. We started from a position of not really having a culture of testing, but that’s changing over time as people see the value in the existing tests we have.

Ultimately, the automated tests are taking more and more of the strain out of development, because regressions are being caught sooner, and therefore being fixed faster, sometimes before the code has been committed.

So, an answer to your question about what we decide to focus on with each release, “all” we do is to continue to focus on what we think people will love to use. That might mean bug fixes, it might mean new features or it might be something as “simple” as slimming down the size of the app so it loads faster. Within the company, each feature team has a roadmap for what they’d like to achieve in the short, medium and long term, and actually doing that is their day-to-day job.

Without explicitly following any particular process, I think that this bears many of the hallmarks of agile delivery. If you take a look at the Agile Manifesto, our development process leans heavily towards the values on the left (favoring “responding to change,” and “working software” in particular). I very much like that.

Josiah Renaudin: Are development cycles like those seen at Facebook viable in the long run? With such compressed windows, is there a concern that developers can burn out?

Simon Stewart: It’s interesting to note that as we shorten development cycles, our developers are actually getting less stressed. With a four week release cycle, you have to wait almost two months to see your feature in the wild if you missed the day we cut the release branch, but if we could drag that time down, the wait would be shorter and the stress (surprisingly!) lower. So, yes, I absolutely believe that release cycles as compressed as Facebook’s are viable in the long run. In fact, I’d love to see us be able to release even faster --- Facebook on the web gets updated twice a day. It’d be lovely if we could do the same thing on mobile.

The key to this speed is automation. There’s just no time to do a full manual run through of every feature before a release. A traditional QA department, following scripts to verify that everything worked as it should, would dwarf our development team. Instead, we’ve placed layers of automated tests to ensure that regressions are as infrequent as possible.

This frees people to do the things that people are really good at: thinking of strange and unusual ways to use the application. As an example, our extremely talented (can you tell that I’m proud to work here?) security team gets to spend more time ensuring that people using Facebook are safe rather than desperately scrambling to keep up with the deluge of changes. Of course, they also automate as much as they can.

The second half of this interview is now available.

A software engineer at Facebook, Simon Stewart helps build the tooling for testing their mobile applications. Simon is the current lead of the Selenium project, the creator of Selenium WebDriver, co-editor of the W3C WebDriver specification, Facebook's W3C AC representative, and contributor to Selendroid, a mobile implementation of the WebDriver protocol targeting Android. As his experience and current work suggest, Simon is keen on automated testing and views it as instrumental to allowing software development with today's compressed release cycles. The way Simon sees it, testers can spend time doing things more creative than walking through tedious checklists of details. Simon tweets at @shs96c.