Emergent Design: Leveraging Agile Retrospectives to Evolve Your Architecture

Technological debt is mistakenly thought of as a technical problem, but when system design cannot change according to the needs of the business, it becomes a business problem. Big Design Up-Front leads to technological debt. Architecture must be allowed to emerge according to the needs of the product and the business. We know iterative, emergent development works; iterative, emergent design is no different. Agile teams should use Retrospectives as a tool to determine current needs and enable emergent design.

The Case for Emergent Design: Understanding Technological Debt

Technological debt is mistakenly thought of as a software problem, but when system design cannot change according to the needs of the business, it becomes a business problem. Just like financial debt, technological debt is a business liability - it affects the liquidity of the business (see Figure 1). Think of good design in a system as a line of credit: just as purchasing new equipment for the business without generating income creates a debt against the credit line, adding features to a system without improving its design is "borrowing" against the capabilities of the system. There can be good reason to do this; after all, credit is used to improve cash flow and solvency.[i] The danger is in thinking of this temporary loan as a permanent increase in funds. The debt must eventually be repaid.

Failure to reduce the debt (improve the design) in favor of making new purchases (adding new features) eventually results in a large portion of effort going toward interest payments (patches and bug-fixes), and reduced buying power (ability to add new features). Eventually, this can lead to a situation where all the effort is going toward interest payments (struggling to keep the system alive), and only trivial features can be added without major rework.[ii]

If the business depends on new features for competitive advantage, this technological debt has now directly affected its ability to thrive in the marketplace. Thus are the seeds sown for the opening line in a long-standing argument between the business and its technologists: "why does it take you software people so confounded long to add a simple capability?"

So why not design the system well up-front? A well-designed system should have complete requirements, a thorough design document, and analysis that shows it will meet the needs of its users. Where is the risk of technological debt?

In fact, the presumption that we can predict the future in terms of software components is often quite inaccurate. We note architect Louis Sullivan's decree that that functional use should take precedence over aesthetics: we can build beautiful design documents, have perfectly defined requirements, but in the end if the software released does not generate value, then the effort was wasted.[iii] This paints a grim picture indeed: technological debt is actually created at the point that the system is designed, because it has not yet created value.

This is the premise of Agile product development: if we design and release in short iterations, we can predict and realize value at a much more rapid pace. In this same manner, technological debt can be determined much sooner. The design becomes less relevant than the value that the system delivers. If we think in terms of value generated rather than in terms of technical documentation, we can than assert good design manifests as a well-valued system.

Retrospectives Enable Emergence, Emergent Design Enables Low Debt

As Agilists, we know that iterative, emergent development works; iterative, emergent design is no different. One tool we can use to enable this sort of emergent design, and keep our technological debt low, is the Scrum Retrospective.

Retrospectives are enablers of emergent behavior via the three pressures of innovation, selection, and conservation. Biological life adapts to its environment over time through these same three pressures: mutation creates new traits in an organism, and its survival (or not) within its environment selects for the chance to conserve its own traits by passing them on to its offspring (or not). Given enough iterations of this process, characteristics emerge - in ways which no human could imagine ahead of time - that enable the organism's perfect adaptation to its environment.[iv]

Retrospectives enable this same effect for the development team, their process, and their product, with three traditional questions. Asked as a matter of rote behavior with the answers then filed away never to be seen again, these questions at best produce an improvement for one Sprint, and at worst are little more than a chance for the Team to vent.

Applied correctly, these simple questions evolve an Agile process that works perfectly for the team and enables it to thrive amidst the complexity of its environment. By "correctly," we mean that the Retrospective enables selection, conservation, and innovation not just for one Sprint, but repeatedly across multiple "generations" of the Team:

1. Ask "what went well." The ScrumMaster should ask the Team what it needs to repeat such good effects; obtain formal licenses for the tools that were used on a trial basis and worked well; share successful behavior with other teams... in short, work to conserve these good team traits.
2. Ask "what didn't go well," as a way to select out traits that hurt the Team, and for each "didn't go well" item, be sure to ask:
3. "What could we change to improve this issue?" For each "didn't go well" item, the Team innovates to identify actions it can own, or that the organization must own. Team-owned changes either become Backlog Items, or behaviors of which the ScrumMaster must keep the team aware. Organization-owned changes become Organizational Impediments which the ScrumMaster must work to resolve.

We can incorporate these same selective pressures when it comes to architectural decisions. The following design questions can be asked as part of the usual Retrospective, or they can be conducted during a separate Design Retrospective:

1. Ask "what part of the design (code) delivered value?" The best designs are often those that are birthed out of refactoring which, by its nature, implies design should be iterative. These portions of the code should be held up as "adding value."
2. Ask "what part of the design (code) should not survive?" The Team targets those parts of the system which made it difficult for team members to add value.
3. For each item that made it hard for the Team to work, ask "what can we change in the design, or add to the design, to achieve the current needs of the business within good design standards?" The Team applies its innovative powers toward the problem of adding value beyond that of system features, and generates work to do in future Sprints.

As for output from these design questions, it should be up to the Team to determine if having a Design Story or having a Design Task for each Story works better. Teams will differ on how they like to work. The important part is not the mode of capture for the design, but how much value in the end that the design has delivered.

The team should also be allowed to choose how to apply the design questions at the retrospective: all questions together or a separate Design Retrospective. However, to those teams who fear that not all team members are capable of understanding architectural issues during, we point out that asking all six questions together is an opportunity for the Team to help those who may not be up to speed (see Figure 2).

Note that the six questions at the Sprint Retrospective, or three questions plus three more at the Design Retrospective are not quite enough on their own to evolve process and design. Some kind of "heredity" or Team memory must be maintained. Whether on a wiki or on the Team planning board, each of the Team decisions must be kept visible. There should be opportunity at the next Retrospective to review the changes that were made and see if they helped, harmed, or had no effect. And the process repeats, Sprint after Sprint, perfectly adapting the Team's process - and the system design - to its environment.

The Architect as Technical Product Owner

Agile principles encourage us to deliver demonstrable product. If reducing technological debt is important, the entire product must include not just system features, but also the capability to produce those features according to shifting business needs (see Figure 3). Briefly, this means considering "the product" beyond the usual definition of the features of the software. Instead, the "entire product" must be viewed as the composite of the software's features, the software's design (which helps the Team make new features), and other capabilities.[v]

This view implies a shift of roles for the Architect away from designing the overall solution and leaving the coders to fill in the blanks. The Architect must think beyond assembling pieces developed by "mere coders" into a unified whole. Nor should the Architect be content to use UML diagrams, patterns, or high-level component design to constrain how the coders will work. All these behaviors smack of Big Design Up-Front, and will lead to the unhappy situation of technological debt being created before the software even exists.

Rather, the Architect should be thought of as the Technical Product Owner, with ongoing responsibility for limiting technological debt to the business. Just as a Product Owner is responsible for answering questions on what the system's capabilities should be, and sets priorities based incremental estimates of ROI, the Technical Product Owner should be responsible for specifying what the capabilities of the system design should be, and setting priorities on work which will improve the Team's ability to add new features.

We have actually seen a user story written in this spirit: "as Bob the Frustrated Operations Guy, I want Engineering to be able to swap in different file output components, so that the system can generate different file formats without waiting for a months-long software development cycle."

A caveat about this new Technical Product Owner role for the Architect: the ScrumMaster must pay attention to the dynamic between the Product Owner and the Technical Product Owner. When we use Agile methods, the Team must move as a unit and make decisions together. Just as QA staff are the part of the Team focusing on quality, the Architects are the part of the Team focusing on good design. But they are still part of the Team.

Two failure modes to watch for are the Subordinate Technical Owner and Dueling Product Owners. In the case of the Subordinate Technical Owner, design work is continually deferred in order to get "just one more feature" into the system. We described the effects of this situation at the start of this article.

In Dueling Product Owners, both the Product Owner and the Architect have equal ranking power. This is good in that it ensures that feature-oriented work and design work are both taken into consideration, but something is wrong if the Team continually finds themselves facing a work backlog containing "feature stories" and "design stories" of equal priority. The Team may then either prioritize these stories on their own (in effect, becoming another Product Owner), or accept the risk of delivering both at once.

The need for ongoing coordination between the Product Owner and Architect should be readily apparent. Whether setting explicit goals during Sprint Planning for "ability to react" and "eliminate technological debt", or working to eliminate conflicts in the work backlog before they reach the Team, both parties must maintain ongoing conversations.

If the Product Owner, Architect, and Team all strive to be mindful of business value as discussed here, the conversations will be fruitful ones. In all things: balance.

About the Authors

Derek W.Wade is a Partner and Senior Consultant at 3Back LLC, a CSM/CSP (Certified ScrumMaster / Certified Scrum Practitioner), and Agile team coach. He has over 13 years experience across all roles of the software lifecycle in several industries including Aerospace, Distribution, Internet Commerce, Digital Mapping, and most recently Loan Financing and Employment Testing. One of Derek's distinctions is bridging the gaps between developers and business users, and his current focus is the "hard problems" of the psycho-social aspects of team product development in corporate cultures. He has led discussions on software testing and transitioning to Agile, and is currently focused on tools and behaviors which improve agile development for non-collocated teams.

Scott M. Barnes is an Executive Vice President at Persis Consulting Company, a CSM (Certified ScrumMaster) and mentor. He has more than 10 years management experience, working in a myriad of market spaces. Bringing to bear over 13 years of technical expertise in various roles in industries such as Vehicle Telematics, Location Based Services, Digital Mapping and Navigation, and Enterprise Content Management, he has been a catalyst for change in business at all levels. Currently focused on evangelizing Agile at the Executive level, he has been successful at engaging businesses and encouraging growth based on solid results. He has proven success in managing projects, consistently meeting or exceeding expectations. Scott is working within the business community at the C-level to bring to bear the power of Agile practices within businesses that are ready for change.

[i] Shore, James. "Voluntary Technical Debt." http://jamesshore.com/Blog/CardMeeting/Voluntary-Technical-Debt.html. Accessed February 4.

[ii] Foote, Brian. and Yoder, Joseph. "Keep it Working." http://www.laputan.org/mud/mud.html#KeepItWorking. Accessed February 11, 2008.

[iii] "Louis Sullivan." http://en.wikipedia.org/wiki/Louis_Sullivan. Accessed March 4, 2008.

[iv] Darwin, Charles. "The Origin of the Species." Gramercy, 1995.

[v] Rawsthorne, Dan. PhD. "Managing the Work on an Agile Project" Accessed February 22, 2008.