2011 Prediction: Organizations will Continue Applying Agile Strategies at Scale

When agile software development went mainstream several years ago many agilists declared success. It clearly was a success, but only one of many. Recently I’ve been seeing more and more organizations attempt to apply agile strategies in what I would consider scaling situations for large teams, on geographically distributed teams, or in regulatory environments for example, and I predict that this will continue for several years to come. Why? First, agile strategies not only work better in practice than traditional strategies they also appear to work as well or better in scaling situations as well. Second, as organizations experiment with applying agile at scale they will observe that they’re succeeding and choose to continue with tailoring agile to meet their needs.

This begs the question how do you go about scaling agile in practice, and this is where the Agile Scaling Model (ASM) comes in to play.  The ASM is a contextual framework which defines a roadmap to effectively adopt and tailor agile strategies to meet the unique challenges faced by a system delivery team.  Mainstream agile development processes and practices, of which there are many, have certainly garnered a lot of attention in recent years.  They’ve motivated the IT community to pause and consider new ways of working, and many organizations have adopted and been successful with them.  However, these mainstream strategies, such as Scrum or Agile Modeling (AM), are never sufficient on their own – as a result organizations must combine and tailor them to address the full delivery lifecycle.  When doing so the smarter organizations bring a bit more discipline to the table, even more so than what is required by core agile processes themselves, to address governance and risk.

The first step in scaling agile approaches is to move from partial methods to a full-fledged delivery process, examples of such include Disciplined Agile Delivery (DAD) and Harmony/ESW – the DAD process framework focuses on information technology (IT) projects whereas Harmony/ESW focuses on systems engineering projects.  These methods provide end-to-end advice, from project initiation to releasing your solution into the marketplace, for applying agile strategies effectively.  Disciplined agile methods explicitly include agile/lean governance strategies which enable them to work within the overall IT organizational ecosystem.  Finally, disciplined methods explicitly go beyond the narrow focus of software to address solution delivery, which includes software, hardware, business process improvement, organizational improvement, and other important issues.  Agile applies to far more than just software development.

To get a feel for what I’m getting at, compare the Scrum construction lifecycle of Figure 1 with the DAD lifecycle of Figure 2.  You can see how the DAD lifecycle extends the Scrum lifecycle to address the full solution delivery effort.  I’ve run into numerous organizations around the world that have developed something very similar to Figure 2, yet it’s been very rare to find organizations that thought to go looking around on the Web for this sort of thing.  Regardless, I think it’s fairly safe to predict that organizations will continue to apply agile strategies across the full delivery lifecycle in 2011 and beyond, and not just to construction.

Figure 1.  The Scrum construction lifecycle.

Figure 2. The Disciplined Agile Delivery (DAD) lifecycle.

 Once you have experience with applying agile across the full delivery lifecycle, the second step to scaling agile is to address the degree of complexity faced by your solution delivery teams. A lot of the mainstream agile advice is oriented towards small, co-located teams developing relatively straightforward systems.  But once your team grows, or becomes distributed, or you find yourself working on a system that isn’t so straightforward, you find that the mainstream agile advice doesn’t work quite so well – at least not without modification.  Let’s explore each of the ASM scaling categories one at a time:

• Team size.  Mainstream agile processes work very well for small teams of up to 10 – 15  people, but what if the team is much larger?  What if it’s 50 people?  100 people?  1,000 people?  Organizations, are in fact, successfully applying agile strategies on programs of hundreds of people, but they tailor their practices and tooling approach to reflect the increased communication and coordination risks.  In mid-2010 I ran a survey, which explored the success rate by delivery paradigm and found that agile strategies worked as well or better than traditional strategies regardless of team size.  This leads me to predict that we’ll see more and more organizations reporting that they’re succeeding at large-team agile in 2011.

• Geographical distribution.  The more geographically distributed you are the more challenging it becomes for team members to collaborate.  Collaboration is easy when you’re all in the same room, a bit harder if you’re in different cubicles in the same building, harder yet if you’re in different buildings in the same city, and harder yet if you’re internationally distributed.  Surveys have shown, once again, that agile teams are as successful if not more so than traditional teams regardless of the level of distribution.  They’ve also shown that the majority of agile teams are geographically distributed in some manner, so I’ll go out on a limb to predict that organizations will continue to find ways to support geographically distributed agile teams in 2011.  As an aside, if you are doing geographically distributed development, I highly suggest looking at the Jazz platform tools.

• Regulatory compliance.  Many agile teams face regulatory compliance issues, such as the requirement to conform to Sarbanes Oxley, ISO or FDA regulations.   As a result agile teams often must increase the formality of the work that they do and the artifacts that they create to conform to the regulations, and luckily surveys are starting to show that organizations are successfully doing so.  Although agile regulatory compliance projects seem to be small in number now, I predict that 2011 will experience a noticeable upsurge in such efforts as organizations roll out agile strategies to a broader number of project teams.

• Domain complexity.  Some project teams find themselves addressing a very straightforward problem, such as developing a data entry application or an informational Web site.  Sometimes the problem domain is more intricate, such as the need to monitor a bio-chemical process or air traffic control. Or perhaps the situation is changing quickly, such as financial derivatives trading or electronic security assurance.  The greater the domain complexity the greater the need for techniques to deal with the complexity.  I predict that due to the continuing success of agile, we’ll see more organizations push their boundaries and apply agile in more complex situations in 2011, which in turn will drive a greater focus on agile modeling practices to help address such complexities.

• Organizational distribution.  Sometimes a project team includes members from different divisions, different partner companies, or from external services firms.  The more organizationally distributed teams are the more likely the relationship will be contractual in nature instead of collaborative.  Over the past few years I’ve seen service providers apply agile techniques on IT delivery projects, sometimes of their own accord and sometimes at the request of their customers, and I predict that this trend will accelerate in 2011 due to the desire to reduce risk and time to delivery on outsourcing projects.

• Technical complexity.  Some applications are more complex than others.  It’s fairly straightforward to achieve high-levels of quality if you’re building a new single-platform system from scratch, but not so easy if you’re working with multiple platforms, existing legacy systems, and existing legacy data sources – all of which are typically less than perfect.  The complexity only seems to grow from year to year, so as agile is applied to more situations we’re going to see more teams running into harder technical situations.  As a result I predict a greater focus on agile architecture strategies and agile testing and quality strategies in 2011.

• Organizational complexity.  Your existing organization structure and culture may reflect waterfall values, increasing the complexity of adopting and scaling agile strategies within your organization.  To make matters worse, different subgroups within your organization may have different visions for how they should work.  Sadly, I predict that this will continue to be a challenge for many years to come.

• Enterprise discipline.  Most organizations want to leverage common infrastructure platforms to lower cost, reduce time to market, improve consistency and promote a sustainable pace.  This can be very difficult if your project teams focus only on their immediate needs.  To leverage common infrastructure, project teams need to take advantage of effective enterprise architecture, enterprise business modeling, strategic reuse, and portfolio management disciplines.  These disciplines must work in concert with, and better yet enhance, your disciplined agile delivery processes.  Surveys show that we are seeing successes with agile teams working well with enterprise teams.  However, I still see too much project focus in the agile community and far too much skepticism regarding agile from enterprise professionals, and I predict we’ll see little progress on this scaling factor in 2011.  Hope I’m proved wrong.

In general, I predict that existing trends within the industry surrounding applying agile strategies at scale will continue, and in some cases accelerate.  One of the reasons why I run my various IT surveys is to find out what is actually happening out there.  As a result, I’ve seen the adoption rate of agile techniques grow over the years, the superior success rate of agile strategies compared to traditional ones, and the successful application of agility at scale.  I’m starting to see clear evidence that in scaling situations agile strategies are superior to traditional strategies, which bolsters my confidence in the predictions I’ve made in this article.