Is "Agile Methodology" an Oxymoron?

In this article, I’ll explore the differences between a process and methodology. Then discuss why, in my opinion, the word process or methodology is a wrong word to use to label and encapsulate agile-lean product (system-software) development. I will also provide guidance how a team can move forward inspired and motivated to uphold the team’s agile-lean product (system-software) development approach, continuously improve and confident in the security such guidance provides.

First, let me clarify my understanding of a process and methodology. According to Merriamm-Webster.com, the primary sense of the noun process is:

a : progress, advance lt;in the process of timegt; b : something going on : proceeding

2 a (1) : a natural phenomenon marked by gradual changes that lead toward a particular result lt;the process of growthgt; (2) : a continuing natural or biological activity or function lt;such life processes as breathinggt; b : a series of actions or operations conducing to an end; especially : a continuous operation or treatment especially in manufacture

3 a : the whole course of proceedings in a legal action b : the summons, mandate, or writ used by a court to compel the appearance of the defendant in a legal action or compliance with its orders

Thus, a process is holistic in nature and is devised with a specific goal in mind. While different organizations apply system-software development processes that are similar in many respects, their processes also differ in some ways. A system-software development process must be flexible enough to meet the needs of both a particular organization and a particular project.

A methodology is a much broader concept than a process. The Random House Dictionary defines a methodology as “a set or system of methods, principles, and rules for regulating a given discipline….” The American Heritage® Dictionary of the English Language defines the two main senses of the term methodology as follows:

“A body of practices, procedures, and rules used by those who work in a discipline or engage in an inquiry; a set of working methods….“The study or theoretical analysis of such working methods.” Dictionary.com gives this definition of the suffix -ology: a “branch of knowledge, science.”

The Random House Dictionary says the word “method refers to a settled kind of procedure, usually according to a definite, established, logical, or systematic plan.” The American Heritage® Dictionary defines a method as a “procedure, especially a regular and systematic way of accomplishing something” or “the procedures and techniques characteristic of a particular discipline or field of knowledge….”

I have found the codification of system-software development past and present to be comprehensive and extensive bodies of knowledge that attempt to codify/capture the structure and substance of roles, tasks, artifacts, principles, guidelines, examples, templates, checklists, practices, activities and methods, targeting many different system-software development situations and contexts.

“One size fits size all”, software development processes or methodologies don’t work.

It isn't so much a problem with what is contained in a methodology but how it is packaged and its practicality of use. Methodologies end up being a large software development process repository, which is like a large pantry containing all the ingredients you would need to make any number of meals. Unfortunately, people don’t seem to realize or come to understand they don't need all of the ingredients for every project. What usually results is many projects using way too much process, which in turn results in more documentation, more design and less software. Also, trying to quickly find a specific answer to what you are looking for ends up to be an ineffective, time consuming and frustrating experience.

"The six most important words: I admit I made a mistake. The five most important words: You did a good job. The four most important words: What is YOUR opinion? The three most important words: If you please. The two most important words: Thank You. The one most important word: We. The least important word: I." - Author Unknown

The what, why, and how of agile-lean product (system-software) development and delivery is not one persons vision alone; to become reality it needs to be a "shared" vision through negotiation and compromise between individuals, the team and the organization.

System-software development needs to be adapted to each specific situation and within context. Individuals and teams have different skill sets, levels of experience and levels of capability. Projects have different budgets, schedules, scope and risk profiles. Organizations have different value chains and target markets.

Fundamentally, agile-lean product development is an empirical and adaptive system-software development approach to be guided by a general set of values, principles and practices – a philosophy and set of norms rather than a step-by-step process or methodology.

Norms are the behavioral expectations and cues within a team or organization. This sociological term has been defined as "the rules that a group uses for appropriate and inappropriate values, beliefs, attitudes and behaviors. These rules may be explicit or implicit.”

The following set of norms lays a pragmatic and adaptive foundation for the agile-lean product (system-software) development craftsperson to rally around and evolve.

• Adaptive rather than predictive
• People oriented rather than process oriented
• The Business and ITcollaborate and make decisions together quickly and effectively, reducing waste and increasing feedback loops
• Shared system-software development approach that evolves through negotiation and compromise between individuals, the team and the organization
• Focus on quality[2]
• Reduce work-in-process²
• Deliver often²
• Balance demand against throughput²
• Prioritize²
• Attack sources of variability to improve predictability²

The following values are from the 2009 Manifesto for Software Craftsmanship: 

That is, in pursuit of the items on the left we have found the items on the right to be indispensable

• Not only working software, but also well-crafted software
• Not only responding to change, but also steadily adding value
• Not only individuals and interactions, but also a community of professionals
• Not only customer collaboration, but also productive partnerships

The following values are from the 2001 Manifesto for Agile Software Development:

While there is value in the items on the right, we value the items on the left more

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

The following principles are from the 2001 Manifesto for Agile Software Development

• Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
• Welcome changing requirements, even late in development. Being agile-lean harnesses change for the customer's competitive advantage.
• Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
• Business people and developers must work together daily throughout the project.
• Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
• The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
• Working software is the primary measure of progress.
• Being agile-lean promotes sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
• Continuous attention to technical excellence and good design enhances agility.
• Simplicity--the art of maximizing the amount of work not done--is essential.
• The best architectures, requirements, and designs emerge from self-organizing teams.
• At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Practices
A practice, as depicted in Figure-1, is a common approach for doing something with a specific purpose in mind. There are no best practices—only adequate practices in context.

Figure-1 – Candidate Practices

 Since so-called best practices are ‘best,’ they also inhibit a “challenge everything” culture and continuous improvement—a pillar of lean thinking. Why would people challenge ‘best’? Mary Poppendieck, coauthor of Lean Software Development, reiterates this point and draws the historical connection from best practices to Taylorism:

Frederick Winslow Taylor wrote “The Principles of Scientific Management” in 1911. In it, he proposed that manufacturing should be broken down into very small steps, and then industrial engineers should determine the ‘one best way’ to do each step. This ushered in the era of mass production, with ‘experts’ telling workers the ‘one best way’ to do their jobs. The Toyota Production System is founded on the principles of the Scientific Method, instead of Scientific Management.

The idea is that no matter how good a process is, it can always be improved, and that the workers doing the job are the best people to figure out how to do it better… Moreover, even where a practice does apply, it can and should always be improved upon. There are certainly underlying principles that do not change. These principles will develop into different practices in different domains...”^[3]

In Conclusion
Look at a set of agile-lean product (system-software) development norms as musicians do sheet music. Recognizing, the more familiar the musicians are with the musical score and the more experience they have playing together, the less dependent the musicians are on the sheet music; except when introducing new musicians to the musical ensemble. This metaphor is applicable to an agile-lean product (system-software) development team and your set of norms.

We need to though be keenly aware norms should not be rigid or prescriptive and should evolve through collaboration between self-organizing and self-directing cross-functional teams based on reality.

Norm setting can only work if the team is truly able to arrive at consensus. Norms won’t stick if members have reservations about them. However, once consensus is reached, the team is equipped with a guide that can serve to strengthen positive practices. A set of norms can serve as a common reference if contrary behaviors arise. Finally, written norms are handy for potential members and newcomers who want to quickly get a sense of the team’s adoption of being agile.

Starting with a simple, clear and concise set of norms in hand, based on reality, a team can move forward inspired and motivated to uphold the team’s agile-lean product (system-software) development approach, continuously improve and confident in the security such guidance provides.

Given the changing world and the many lessons learned over the past decade we have unofficially refined the meaning of agile software development through trials, tribulations and countless successful (and unsuccessful) agile system-software development projects.

Just like in 2001, when the seventeen like-minded individuals collaborated on creating the four values and twelve principles of the Manifesto for Agile Software Development, and defined the foundational philosophy of present day agile, perhaps the time has come for an official update to the Agile Software Development Manifesto, “The New Agile Product (System-Software) Development Manifesto.” 

We are headed in the right direction by de-emphasizing agile software development processes and methodologies and by elevating the level-of-awareness of system-software development craftsmanship with the likes of the Manifesto for Software Craftsmanship and its Renaissance.

[1] Tacit knowledge is knowledge that is difficult to transfer to another person by means of writing it down.
[2] My indirect influence for these values is David J. Anderson’s book, Kanban – Successful Evolutionary Change for your Technology, pages 22-33, ISBN: 978-0-9845214-0-1
[3] Poppendieck, M., 2004. “An Introduction to Lean Software Development