Despite our best efforts we need to know what we are going to code before we write the code. And as much as we might like to test before we write the code we can't really run tests until we have some code. Agile overlaps requirements discovery and implementation so coding can start with minimal or outline requirements but there is still a sequence.
Yet when it comes to fixing a broken development process or organization starting with requirements can make the situation worse. Although counter-intuitive, the first priority should be making development more effective then focusing attention on the requirements process.
Overwhelming demand
One of the problems I frequently see in an organization is a stream of business requests for features that the development groups are unable fulfil simply because there are so many requests. The result is the equivalent of trying to put a quart into a pint jug: it overflows.
Work overflows and doesn't get done; managers spend their time arguing about what should be done and why things haven't been done. Developers cut corners so the work that is done doesn't satisfy the need. Quality is sacrificed in the name of getting something - sometimes anything - shipped.
Part of this is because of the way features and requirements are requested. It is very easy to ask for a new feature. A request might be as simple as an e-mail to the development leader, in which case the cost of making the request is minimal.
Development groups often respond by increasing the price of requests. Before any work will be considered it must be formally documented and signed-off. Increasing the cost may reduce the flow of requests but it also becomes more difficult to withdraw requests that have been made.
However the request it made it is normal that satisfying the request takes more time, energy and resources than making it. Consequently there is an imbalance supply and demand. Demand is almost unlimited, the cost of asking is low and cost does not increase as more requests are made. However supply is strictly limited and very expensive. Increasing supply will actually increase costs because more co-ordination (management and technical) is required.
It is easy to see how demand builds up. Some requests are simply "good ideas" which escape into the requirements process. Others are the result of previous poor performance form IT groups, burnt by previous failures internal customers see a development project preparing to start.
Knowing projects typically arrive late and without all the promised functionality customers make more and more requests in the hope that they will get some of what they want. In response development groups respond by padding schedules, demanding more resources and implementing more gating criteria.
At the same time as making more requests customers ask for more details - project plans and schedules - and even inject themselves into the process to monitor what development does. Too frequently both sides hide behind plans and the illusion of control.
As a result it is perhaps unsurprising that Mary and Tom Popendieck[1] say: "Only about 20% of features amp; functions in typical custom software are used regularly." Any organization that is serious about following the Agile or Lean agenda need to throttle the requests coming in to ensure only features which are needed are implemented.
Fortunately this thinking fits with the way a lot of managers think. During management training many managers are taught the importance of doing the right thing over doing things right. This is good advice but in the case of problematic IT groups could make the situation worse.
Requirements trap
Research from Bain amp; Company[2] shows that a staggering 76% of all IT organizations are neither effective in what they do nor aligned with the business needs. We can think of these two criteria as "doing things right" and "doing the right thing."
Using these two criteria - alignment and effectiveness - Bain identified four types of organization - management consultants just love 2x2 matrix! The first group they labelled "maintenance zone" - these companies are less effective and less aligned. Unsurprisingly they had average IT spending and as a result of poor IT sales were falling at the rate of 2% annually over 3 years.
Unsurprisingly those companies that were more effective and aligned to the business - doing the right thing and doing it right - demonstrated 6% below average IT spending and sales increasing by a staggering 35% annually. However this group constituted just 7% of all companies.
Assuming an organization is in the "maintenance zone" - and after all 76% of companies are - they must decide which to focus on first: to be more effective or more aligned, do the right thing or do things right.
Traditionally Agile improves the effectiveness of development teams. While Agile doesn't ignore requirements the thrust of most Agile processes is to improve effectiveness. Successfully improving effectiveness would allow a team to move from "maintenance zone" to Bain's "well oiled" quadrant. By being more effective companies do things right but don't necessarily do the right thing.
Still, the returns for "well oiled" IT are impressive. The 8% of companies in this category demonstrate below IT spending 15% below average while sales grew at 11% per annum.
These figures show that being an effective development group is actually a pretty good place to be and a vast improvement for many. Simply being effective at what a group does saves significant amounts of money. For some companies just getting to this position will be enough to stay competitive.
But what about requirements? Moving to "well oiled" doesn't address the need to produce what the business wants. What if, instead, we followed the managers' instinct? What if we improved alignment and focused on doing the right thing before doing things right? Wouldn't this be better still?
The answer is No. Bain label this group of companies "alignment trap". While they are more aligned to business need their effectiveness is still poor. These companies see higher IT spending, up by 13%, and more worryingly sales falling faster, 14% per annum than the "maintenance zone" companies. It turns out that doing the right thing poorly is a terrible place to be. This 11% of companies are in the worst place possible.
Yet it gets worse. Bain go on to say that once in the alignment trap it is harder to move to the high performing - aligned and effective quadrant - than it is from "well oiled".
Why is it that doing the wrong thing well is so much better than doing the right thing badly? I think the explanation lies in our old friend the waterfall.
Organizations which attempt to do the right thing naturally focus on requirements. This leads them to take a requirements-first approach. This starts the waterfall, and since 80% of these requirements maybe unnecessary these organization create the demand that pushes up costs.
Explaining Agile
This model explains why Agile has had significant success to date. Agile has its roots in what developers do. This is most obvious when one looks at one of the first Agile methods to gain popularity: Extreme Programming (XP).
XP provides for an onsite customer to produce requirements but on the whole XP is concerned with what developers do - TDD, refactoring, simple design, etc. By focusing on the development process XP improves effectiveness.
As a result XP - and other Agile methods - are very good at making development teams more effective and moving them from Bain's "maintenance zone" to the "well oiled" quadrant. This move itself is a significant benefit to an organization and may be enough to justify the Agile change alone.
This shift was possible because many of the practices and processes advocated by the early Agile methods can be adopted by developers without significant management involvement. Coders can simply start writing unit tests first, they can start refactoring they can set up a continuous integration server. Little management involvement or approval is needed for these changes.
As a result the early Agile successes have been bottom-up successes. Developers just do it. Conversely there is little managers can do to impose many of these activities: only developers known when refactoring is appropriate, forcing TDD can lead to meaningless tests producing evergreen test bars, and how is a manager to know when a design is the simplest possible and when it is not?
Neither is it possible for an organization to transition to Scrum by edict. No manager can force a team to be self-organizing simply issuing an instruction that a team will henceforth be organizing themselves.
So teams can adopt Agile practices and improve their performance, even move themselves to the "well oiled" quadrant. But addressing the requirements flow requires management authority and support.
Resolution
For an organization transitioning to Agile development requirements present a paradox. To be truly Agile, and to learn from Lean, requirements must be arranged so that those with the greatest value, those which will actually be used, are developed. Yet addressing this issue at the start of the change to Agile risks making the overall position worse.
Resolving this conundrum is a matter of sequencing. At the start of any transition the focus needs to be on the development teams and practices, initially the requirements process should be left as it.
This means adopting engineering practices, routines and organizational structures which allow requirements requests to flow smoothly through the system. Requirements go in, working software comes out.
The first benefit of improving the effectiveness of development first is to build trust. Seeing working software delivered they will involve themselves less with how IT works.
Once trust is established more benefits flow. When customers can discuss requirements not just in terms of what they want but relative priorities. Conversations can move on from "When will it be delivered?" and "Why isn't it working?" to those that matter, about what is required and when it is needed by.
From here it becomes possible to fix the alignment issue and move from "well oiled" to "IT-enabled growth." As Bain says: "Contrary to conventional wisdom, the path to IT-powered growth lies first in building high effectiveness and only then ensuring that IT projects are highly aligned with the business."
What next?
So the message is clear: The first step in Agile adoption must be the fix the delivery machine. Improve the effectiveness of the development group.
As effectiveness improves focus can switch from how to what. The changes required here are if anything more far reaching. They are also more long-lasting.
To date Agile methods have had relatively little to say about requirements and business alignment. The current crop of Agile methods largely focuses on development and project management practices. As more companies adopt Agile - and become well oiled - we should expect to see a renewed focus on requirements discovery and business alignment.
[1] Implementing Lean Software Development, Poppendieck amp; Poppendieck, 2007
[2] Avoiding the Alignment Trap in Information Technology, Shpilberg, D. et al, MIT Sloan Management Review, Fall 2007.
About the Author
Allan Kelly is a London-based consultant and interim manager specializing in Agile adoption. His first book, "Changing Software Development: Learning to be Agile" was recently published by John Wiley amp; Sons. He is a qualified Product Manager and Project Manager, and holds a BSc in computing and an MBA in management.
