The Neglected Practice of Iteration

Scribbled on a whiteboard in the Chicago office of my company is the phrase "It's not iteration if you only do it once." That sounds obvious, right? Well, my company develops software using agile processes, and we plan and build software in two-week iterations. Although we call them iterations, we often forget to iterate. That's why we need the reminder. Let me start an explanation with a digression.

Fred & the Werewolf
I recently saw Fred Brooks deliver a keynote speech at a conference. Fred is famous for writing the 1986 paper titled "No Silver Bullet," in which he breaks down software development into two primary parts: essential and accidental. "Essential" refers to the difficult part of understanding the domain for which we're building software and the determination of what software to build to make life easier in that domain. "Accidental" is the necessary programming we do and process we follow to implement the solution we devise. Fred argues that, although we've made great strides in improving the accidental part of software development, "the hardest single part of software development [remains] deciding precisely what to build."

Because of this problem, Fred explains that there's "no silver bullet" that will kill the werewolf found in that complex essence. He points out that we continue to look for improvements in technology and tools, but much of the emphasis focuses on the accidental and not the essential. He asserts that, because of this, we'll not find an approach that's likely to provide an order-of-magnitude improvement in software development productivity.

Fred's Solution
In 1986, the remedies Fred observed that offered the most promise were:

• Iterative refinement of software requirements
• Incrementally growing systems versus building them all in one whack from a specification

Today we call approaches like that "agile development," which is further proof that agile development isn't anything new, but rather the stuff we knew we should have been doing since computers were programmed with punch cards. (Note that there's more to agile development than iteration, but that's a big part of it.)

Iterating and Incrementing
If I were to iterate, I might start with a rough idea of a solution to a problem, then build a prototype (i.e., something that would validate my rough idea). A software prototype may not be beautiful to look at; it may lack robust data validation and error handling, but it would serve its purpose of validating my best guesses at software requirements. And, since I'm working iteratively, I'd expect to be wrong-or, if not wrong, then darn lucky. I'd validate the prototype by simulating the usage of the software for its intended purpose. In response to inevitable problems with the design, I'd adjust the prototype and try again. I'd do this over and over until the design seemed good enough to proceed to building real, robust software.

Notice that when we iterate, we start with a vague idea and what we choose to build gradually emerges.

I find that a lot of people mistake working incrementally for working iteratively. By incrementally I mean building a little bit at a time-sort of like adding bricks to a wall. If I were to build a house incrementally, I might build the first room completely, including the paint, furniture, and interior decoration. Then I'd build the next room in the same fashion. We don't build houses incrementally, but we do build neighborhoods incrementally-one house at a time.

Notice that in the figure above, incrementing requires a more refined understanding of what's being built. We then add pieces incrementally.

In practice, we generally mix iterating and incrementing. That is, we build things up by both revising a little and adding a little. The area where I see software development practitioners fall down is forgetting to iterate deliberately. When problems or errors in the requirements of the software are identified, we often consider that a genuine problem. We believe that if we'd tried harder and had been more thorough, we wouldn't have to fix problems and we wouldn't have to iterate. It's that misguided optimism that gets us into trouble. When we don't plan to iterate, we get caught either delivering poor quality software or delaying schedules in order to make time to iterate. We kick ourselves, or others, for not "getting it right up front" when we all know that Fred is right: The hardest part of software development is figuring out what to build. Fred's werewolf seems to get us every time.

Apply Iteration to Software Requirements and Development
When working with software requirements, we can iterate using models such as use cases
(Alistair Cockburn--Writing Effective Use Cases), user scenarios (Jeff Patton--Write a Blockbuster Using User Scenarios), or any other means you can devise to validate your assumptions and the assumptions of users and business stakeholders. Remember, it's not iteration if you do it only once. Write a user scenario, and review it multiple times with users and stakeholders. Build a paper prototype and simulate its use multiple times with prospective users. Plan on at least three iterations per artifact: the original, plus two revisions.

When building software iteratively, a first iteration may be a bit of a hybrid between prototype and finished code. It may lack the refinement necessary for release, but it should contain enough sophistication to allow a user to validate that the software satisfies its intended purpose. Chances are you'll need more development iterations to change and refine features. Remember, it's not iteration if you do it only once. Once you're confident you're building the right thing, take steps to make sure you're building the thing right by adding necessary validation, security, performance capability, etc. Plan on at least three iterations of development for each releasable pieces of software. At each development iteration, evaluate and change the software in response to that evaluation.

When constructing project plans, make time for iteration. Don't overstuff plans by scheduling exactly as much work as can be done in a particular time period. A full project plan is a late project. Erase the idea from your mind that projects are late because of changing requirements and poor estimation. Requirements are always suspect. Estimation is always difficult. Projects are usually late because of failing to plan in iteration.

Further Reading

• Fred Brooks, No Silver Bullet: Essence and Accidents of Software Engineering
• Alistair Cockburn, Iterative Versus Incremental Development