Many software engineering teams have adopted Agile. However, teams delivering process change tend to use traditional ‘lifecycle' approaches and have yet to embrace Agile. Why is that? Can the Agile approach be used to implement or facilitate process change?
Achieving buy-in and support is much easier if the process change project mirrors the client organisation. Increasingly, these project teams are confronted with an Agile client environment utilising Scrum. This is prompting process change teams to ask the question, "Should we adopt the Scrum framework for our delivery?" The first port of call when considering this question is the four fundamental values of the Agile Manifesto (http://agilemanifesto.org/). These values appear to complement the implementation of process change.
The first of these values is 'Individuals and interactions over processes and tools‘. At first glance this statement may seem anathema to process improvement, but actually it is very much in-synch with how projects should be run. Change is about people, not process. In good process change projects the documents, processes and procedures you develop are merely tools to institutionalise best practice behaviours and smooth interactions between people.
One of the common themes in an improving engineering organisation is the prevalence of tacit knowledge and working practices treated as policy or procedure. Teams well down the process improvement path operate using explicitly defined procedures. As Michael Polanyi famously said, "We know more than we can tell". Engaging the individuals and understanding the interactions are fundamental to turning Polanyi's "Tacit Knowing" into explicit knowledge.
The value 'Working software over comprehensive documentation‘ is straightforward to match. We are not writing code; however, we do develop a wide variety of documents, processes and procedures to support the implementation of the process changes. The aim is to develop effective processes which can be tailored to the situation, but still ensure the necessary rigour to meet the goals of the organisation.
The risk with putting processes in place is that the process becomes the goal in itself rather than the process being a tool to support a specific organisational objective. Therefore, this Manifesto statement could be rewritten a number of ways, but "Working processes over bureaucracy" should resonate with most readers.
The third value is 'Customer collaboration over contract negotiation‘. If we accept the purpose of process improvement is to institutionalise desirable behaviours, then the key to success is getting your customers to change how they behave. Some process change teams use policy statements as a mechanism to mandate the use of the defined processes. Often, the mandated processes are developed in isolation of the end user. Whilst most practitioners agree policy has a place, used in isolation it will never be as effective as a collaborative approach.
By performing the process modelling and implementation steps iteratively in collaboration with the customer, you get a much better level of buy-in. The result of working in this way is that the client achieves a sense of ownership. With ownership comes accountability and responsibility, which in turn sees quicker adoption, higher quality feedback and understanding.
Process improvement projects are about implementing process change within the client's target organisation. In order to achieve this outward facing change you often have to deal with many internal project changes. Therefore, the fourth value of the Agile Manifesto resonates well, 'Responding to change over following a plan‘.
It is difficult to predict all the impediments to any type of change. No organisation has the same combination of politics, personalities or delivery pressures. As a result, the level of requirements churn we see can be significant; the subsequent planning and requirement re-factoring is enough to make Agile very attractive.
Given that we have established - albeit at a high level - that change projects appear to be compatible with the Agile values, how do we implement this in practice? The Scrum framework is very common in Agile environments and is an easy technique to mirror. It certainly appears to be a useful weapon in the emerging Agile process change arsenal. From a process improvement perspective Scrum gives five primary benefits, which are:
- responsiveness to project change
- support of customer intimacy through the Scrum teams
- frequent planning and delivery cycles through Sprints
- goal orientated roles and team structure
- flexible support of process improvement approaches.
The first benefit listed is the way Scrum supports project change. Ideally, process improvement is a journey not a destination. Scrum allows the introduction of change at any point making this journey easier to manage.
It is not uncommon for an important piece of tacit knowledge to emerge mid-project and affect the plan. In addition, a common and positive side effect of implementing improved processes is the identification of additional process refinement opportunities. Thanks to the flexible nature of Scrum these new items can easily be added to the product backlog and prioritised by the product owner.
Unmanaged change is never desirable, but by following the Scrum process diligently, change becomes a positive sign of process take up. Also, by providing a flexible 'pull‘ of requirements into the internal project planning we ensure that the project team reacts quickly, gives the process adopters what they need when needed and the project team does not lose any early stage opportunity to institutionalise process. This ‘pull' manifests through the iteration planning, in practice it is the reprioritisation of the Backlog by product owner to discoveries made in the previous sprint. For example, if a defined process had not accounted for a SOX compliance requirement then its rework can be reprioritised easily by the product owner and handled in the next sprint.
The second benefit is that Scrum teams are a powerful mechanism for building intimacy with customers, with the customers being the process adopters. Process change projects are notoriously difficult, not because process is difficult to define, but because in order to achieve traction you have to win the hearts and minds of the people implementing and using the processes developed by the project One of the simplest and most effective ways to affect change is to have the process-users own the processes they are implementing. Therefore, how you populate your Scrum teams is important. Ideally, it should be a good mix of subject matter experts from the target organisation supported by process specialists.
The role of the product owner is crucial to an effective project. The product owner for a process change project is usually a reasonably senior team manager, often a delivery manager or QA manager, who is responsible for some aspect of the organisation who will be using the defined processes. This role sits at the level of the adoption team with a view across the entire delivery domain, if multiple work streams exist the product owner will be responsible across the whole piece. This can be challenging and requires a strong individual who can balance conflicting priorities and politics. A good product owner needs to understand that a Scrum project is goal based, so they should focus on what is critical to success and prioritise accordingly. This is a difficult thing to achieve, as most organisations are cost based and focused on providing proof that resources are utilised per the plan. Achieving this goal based focus is important as it allows the process users to clearly see and engage with the project drivers making it easier to achieve process adoption.
Another benefit is the frequency of planning and delivery. Sprints are designed to deliver working code in short sharp bursts. Process definition and implementation do not cope well with big bang delivery strategies. Dumping a large process infrastructure onto a user base in a short period will result in delivery impacts, inefficiencies, mistakes and frustration. What takes time in most projects is a combination of sign-off and the subsequent implementation, the actual development of process is often very brief. An effective sprint strategy is to develop processes in sprints of two weeks, within this Sprint the Processes selected from the Backlog are defined, socialised and agreed so they are ready for adoption. This can then be followed by a two to four week adoption sprint. During this sprint the Requirements on the Backlog take the form of training exercises, coaching sessions, implementation tasks (such as a PM using the new risk log or story template). Running definition and implementation in sequence is especially effective as it allows the process definers to identify defects, changes and issues for inclusion back onto the Backlog.
By delivering targeted bursts of process you are able to de-risk the implementation and aid end user buy-in. The sprints make it easier to trial and refine iteratively, ensuring the processes meet the needs of the client. The end users find this approach much easier to cope with and it minimises the impact on their daily workload.
It can be difficult to know when a process improvement project has met its objectives. An advantage of Scrum is that the frequent iteration planning forces you to take stock more often, helping to ensure work stops at the most appropriate point. Project success is measured by the level of adoption of the processes and then by the impact of new processes on target organisation. From a Sprint perspective this translates to simple adoption measures. The refinement and benchmarking of the processes themselves is usually a separate body of work which may feedback changes into the Product Backlog. Because process is implemented frequently using Scrum, this allows more opportunity than traditionally available for these changes to be fed back to the project team.
The breakdown of roles and responsibilities, as defined in the Scrum framework, is a benefit. Ownership of the processes by the client is much easier to achieve in a Scrum team where you realise a self-organising and self-managing structure. The ability to focus on the requirements in a reactive way allows people to feel their changes are a sensible manifestation of the team's needs.
There is a common recognition that self-organising teams take work. They simply don't happen in the tight timeframes of a normal project without some leadership. This is not about managing the Scrum team, but providing mentoring, guidance and steering to help the team achieve its potential. However, this is comparable with the effort required for leading any form of process change team and should not put you off the task.
Finally, the fifth benefit of Scrum is that it does not dictate the engineering processes to deliver working software. This makes it adaptable to common process improvement approaches, frameworks and methodologies. It shouldn't matter whether you are using Six Sigma, Lean or some other variant as Scrum allows for the best-fit elements of definition and implementation to be catered for whilst still managing the requirements, change and planning in a reactive Agile manner.
When integrating your common toolset into Scrum for the first time it is a good idea to take guidance and advice from experienced 'Scrummers‘. Most of all ensure that you stick to the rules, they exist to minimise the risks.
Is Agile a Good Choice for Change Projects?
The simple answer is yes, in any form of change project, there are six stages of commitment. They represent increasing levels of acceptance to change over time. These are: contact, awareness, understanding, engagement, involvement and commitment. Agile techniques have proven to help to ease the path through these stages.
The benefits of Scrum translate into a collaborative and reactive team working to common goals. The journey from awareness to commitment can be much smoother helping to accelerate progress against the commitment curve. In part, this is due to the self-organising team helping to avoid and identify early on the impediments to change. The iterative planning of the Sprint further accelerates the journey by keeping the project reactive to change.
In an Agile aware and educated client environment process improvement projects benefit from using Scrum. As with any approach, there are challenges, but the impact it has on client process ownership and take-up makes it an attractive option.
About the Author
Justin Glasgow is an experienced consultant who has run professional service and delivery teams for organisations such as Capgemini, Fair Isaac and IP Industries. http://www.linkedin.com/in/justinglasgow
