I have seen many Agile projects, particularly those focused on brand-new product lines, struggle with getting their infrastructure up and running. Much of the reason is the time and effort that is needed to get infrastructure established far exceeds the time it takes to start development using an Agile method, effectively the first iteration. Typically the approach used to establish infrastructure is ad hoc and often not always aligned with the needs of the project. Therefore, a task must be identified to establish infrastructure. The question then is, how to best approach the establishment of infrastructure for a project using Agile methods? We do not want to build excessive infrastructure that may constrain us in the future yet we want to establish enough to keep us stable and productive.
Infrastructure Challenge
Infrastructure refers to the technical structures such as environments, tools, and the processes that are used to support the product. Examples of infrastructure-related components include the servers, network, development, requirements, configuration management (CM), build, defect tracking, and test tools used within the product development context. Infrastructure for a new product typically gets defined and established within an architecture and infrastructure phase of the first release when following a traditional methodology. The challenge is how do you establish infrastructure for a project following Agile methods when you must shift from a phased approach to an iterative approach and when the infrastructure personnel work in a hierarchical organization?
Considerations for Approaching Infrastructure
Let us start with some key considerations that will help frame an approach better suited to Agile methods:
- Applying Agile implies that we move away from big effort up front (BEUF) for establishing the infrastructure and instead evolve it over time and so that it is not an overwhelming effort for the first project and ensures the infrastructure is closer to our needs as the project team uncovers them over time.
- The establishment of the infrastructure is dependent on architecture. Architecture helps define the technology stack that is the building blocks for infrastructure.
- The infrastructure should be constructed to suit Agile. Moving away from phases and hierarchical processes and services where typically much ceremony and permissions are required and instead move toward a more continuous and heterarchy world (i.e., opposite of hierarchy, where the world is flat and people are empowered at the lowest possible level) where knowledge and function and the constant building of such are the mode.
- The project team using Agile methods must continue their development work to deliver functionality and business value per Agile principles as the infrastructure gets incrementally built.
The challenge now focuses on how to build an infrastructure that supports Agile while also not obstructing the progress of the team that must continue to deliver value. One suggestion is to utilize a newly termed approach entitled Infrastructure Envisioning.
With that in mind, consideration should be given to the idea of "envision the future" then "adjust and implement incrementally". This essentially applies an incremental approach to the continuous establishment of an effective infrastructure.
Envision the Future
In order to get an idea of the context of infrastructure, it is important to envision the infrastructure needs of the new produc,t understanding that things are subject to change. Within an Agile framework, this may be best served by implementing an iteration 0 approach that focuses on a high-level infrastructure model. The goal of infrastructure envisioning is to identify the initial scope of the infrastructure, access the feasibility of technologies, and then establish a reasonable go-forward direction. From here, it is important to understand that the while we have a reasonable go-forward direction, that we will continue to revisit the direction and adjust as appropriate.
Infrastructure Dependency on Architecture
Within some Agile frameworks, there are models that describe an iteration 0 or inception phase, where high-level envisioning of requirements, design, and architecture occur. These three engineering fields can help provide content to the infrastructure framework, but none more so than architecture. Architecture helps provide construction guidance for the infrastructure. This is particularly true since infrastructure has become more complex. This is due to integrations among tools, connectivity across systems and domains, and standards being applied across environments.
Leveraging the Agile Model Driven Development (AMDD) framework helps drive the evolutionary establishment of architecture, beginning with envisioning the architecture. AMDD focuses on establishing models (e.g., architecture) before writing source code. However, the key is that a "just barely good enough" model is needed to move the development effort forward, as opposed to an extensive and detailed model. Over time the architecture and design models will evolve in a way that more closely aligns with the actual needs of the product as they become more clearly known. Within the AMDD framework, one of the outcomes of architecture is a technology stack. This output becomes input to infrastructure envisioning. However, the local challenge with architecture is that there are many architecture models and viewpoints that are used so it is important to understand the model and viewpoints within the company you work.
Vision for Infrastructure
As previously mentioned, key input into understanding the infrastructure need is the architecture model. One representation of this stack is the physical model that articulates the layers within the infrastructure such as the platform, programming languages, tools, and more. Some layers are mandatory (e.g., includes the server and desktops). For the sake of this illustration, I will call this the platform layer. Some layers need to be defined early (such as the database, programming, reusable components) since they may be utilized early in the project when using Agile. I will call this the development layer. Some layers support individuals and interactions in their day to day work (e.g., collaboration, configuration management, test, etc.). I will call this the support layer. For the sake of this article, I am keeping the layers simple and have illustrated this simplified architecture framework while highlighting the infrastructure stack in the figure below. Again, the various layers within an infrastructure stack may be different within your company so it is important to identify the infrastructure view within your organization.
Figure 1: Simplified Architecture Framework highlighting the Infrastructure Stack
In some cases, the infrastructure stacks of interdependent legacy systems and IT roadmap standards constrain the choices for envisioning the infrastructure for the new product. However, this is not always a bad thing. If this new application must integrate with legacy products, then this helps assure easier integration. For example, if the existing legacy architecture and infrastructure applies the .Net framework, then introducing a Java Interface Layer (of JIL) framework within a new product may introduce extra work including bringing in extra skills into the company. Typically legacy constraints occur at the platform and development layers.
The key, however, is that you should not let predefined tools, components, or environments constraint the interactions of people, but instead allow the way the team optimally interacts as a means to help drive the better infrastructure choices to the degree that it is feasible. When the Agile Manifesto says, "Individuals and interactions over processes and tools", this is an example of not allowing the tool to drive or constrain the interaction. A suggested approach is to begin projects using Agile and to then see how individuals and interactions meld together into working processes. This information can then be used to help identify tools that may be suited for the work (as appropriate). This is particularly true in the tools found in the support layer.
The end product in the short infrastructure envisioning exercise should include a high-level infrastructure stack list of technologies within their respective layers as it relates to the architecture framework. While you should not expect to fill in all layers, there are some layers you cannot ignore. An example of this is the platform layer where servers and desktop live. However, tools in other layers can be identified over time after it is clear what best agile processes and practices suit the needs of the team. Then a tool may be considered.
Adjust and Implement Incrementally
It is important for infrastructure to emerge based on the needs of the product and the business. Applying an iterative and incremental model with continuous adjustments (as applicable) helps lead us in this direction. This allows us to limit infrastructure debt. Borrowed from the concepts of design debt and technical debt by James Shore, the newly coined term “infrastructure debt” refers to building beyond our need such that when functionality evolves, any incorrect infrastructure becomes a constraint to the functionality the customer wants, ergo a business problem. By applying a more evolutionary approach, it allows the needs to drive the infrastructure while minimizing the constraints.
What are some strategies that can help with establishing infrastructure in an iterative and incremental manner? Some include utilizing risk assessment, applying prioritization, working in parallel between the project team and infrastructure team, and applying retrospectives. Within a project timeframe, you may not be in a position to iteratively build a fully functional and automated infrastructure. Instead, focus on the areas that reduce risk and are perceived to be of high value (priority) to the project team using Agile (the customer of the infrastructure). After each infrastructure piece is delivered and effectively implemented, then acquire feedback on the infrastructure functionality as input for adjustment and prioritization for the next iteration. Let us look at each strategy in more detail.
Assess Risk
Now that you have a high-level view of the infrastructure stack and layers therein, it is time to identify risks that can prevent success. It is important to manage risk, as it provides lead time into potential problems that can slow velocity. By identifying and assessing infrastructure related risks, the more serious risks can be appropriately mitigated. This ensures we do not waste time upfront focusing on lesser risks and also do not waste time later on resolving a problem that could have been prevented if a mitigation of the greater risk had occurred.
Prioritize
Prioritization of infrastructure needs allows the project team using Agile and those involved in establishing the infrastructure with a means of knowing how much value certain tools are to the infrastructure and then implementing them in a just in time (JIT) approach. As an example, the need for a server and desktops typically are high priority or work cannot get done so they should be available when the project team is ready in the first iteration. However, prioritization of other tools in the support layer may be a lower priority or until such time as the effort to implement them is outweighed by the need to have them in place. Another example of this is configuration management (CM) tools. Sometimes on new projects, people avoid a CM tool or use a very simple CM tool. Howeve,r as code starts to change frequently and many people work with the same code modules, there is a potential to start deleting other people's changes. In this case, the effort to recreate lost changes is considered waste and greatly impacts velocity. At this time in the project, implementing a competent CM tool may become a high priority.
By employing prioritization and JIT, we can ensure we do not build out too much infrastructure too soon, therefore limiting our infrastructure debt. Otherwise an "inventory of infrastructure" becomes shelf ware until such time that it is needed. In addition, there may be changes that occur to the direction of the project that makes the shelf ware infrastructure of less value or constrains future infrastructure direction. In both cases, this leads to waste, a muda in the lean world.
Parallel Streams
The next strategy focuses on a model that includes two different groups who need to work together to establish the infrastructure for the Agile method being used. The first group is the project team who is using Agile to deliver working software for the end-user (the ultimate customer who perceives the business value). The second group is the infrastructure team who builds out the infrastructure. Interestingly, the project team becomes the customer for the infrastructure team, just as the end-users are the customers for the project team. Just like the standard practice where the customer should be part of the project team for continuous feedback, the same holds true that a member of the project team should be on the infrastructure team. This is important to understand because the project team is the group that must assess the value of the changes so that the infrastructure team knows which infrastructure changes are perceived to be of most value and which to work on.
Retrospectives
Upon the completion of an iteration of infrastructure work, it is useful for the infrastructure team to apply retrospectives so that the project team can review the infrastructure changes particularly when certain tools are installed. A hands-on review with the project team can highlight the need for adjustments or improvements to the configuration of the tools.
Summary
The challenge is to balance the need for speed while establishing the infrastructure we need. If we build too much up front, it may not be what we need and constrain the team in the future. If we build too little it may result in a decrease in velocity.
The benefit of infrastructure envisioning is that in a short timeframe, it provides a high-level framework of the infrastructure for the new product that is aligned with both the high-level requirements and architecture. The benefit of an incremental approach is that it both utilizes the important aspects of minimizing risk and considering value for prioritization while at the same time spreading out the effort of establishing an infrastructure over time based on the JIT needs of the project team.
References
1. http://www.agilemodeling.com/essays/amdd.htm - Agile Model Driven Development (AMDD): The Key to Scaling Agile Software Development by Scott Ambler
2. "Infrastructure Architecture" by Daniel Jumelet, March 2007
3. http://agilemanifesto.org/ - Agile Manifesto
4. "Voluntary Technical Debt", by James Shore, Sept 2006
5. Heterarchy is a term that Panos Konstantinidis suggests as an antonym for "hierarchy".
