1. Tsunami Development
Moving to agile development (or doing anything new) is not easy. Beyond learning some of the technical stuff, the new terms like SCRUM, stories and so on, it is about changing mindsets. How do you change mindsets? How do you understand mindsets, and to do so quickly? I do this by drawing diagrams, and have project teams identify where they are and where they want to be. Of course, these diagrams should be easily understandable, and so that the project team’s response can be quick. You either win them over in 2 minutes, or you lose them completely. The diagrams should also be easily reproduced such as on a white board, so you are always ready.
Following the tradition of an agile discussion, I begin with a discussion on traditional waterfall development, which is the very first diagram I draw (see Figure 1). The horizontal axis shows a time line with the project start and project end. The vertical axis is some kind of a measure. So, this is a graph, a profile. These axes will be the same for the other graphs I am about to show. In this first graph, I show the case of traditional waterfall development. Actually, I think the waterfall metaphor is incorrect. The more appropriate metaphor is tsunami-development. Ina tsunami, there is agigantic wavethat sucks up all the water and slams against the shore. If the wave does not kill you, the debris will. This is the case in tsunami development. All postponed bugs and issues (debris) slam against the helpless project team.
Figure 1 - Tsunami Development
Anyway, in this first diagram, the green line shows the progress of implemented requirements. In tsunami-development, implementation does not begin on day one. Instead, there is a lot of upfront requirements work – requirements are collected, analyzed, debated, detailed, documented, designed, tracked, and whatever you can dream of that requirements and design people do.
There is this special day when implementation/coding starts. Either there is some agreement on the requirements and design or it is determined enough time has been wasted and if they do not start implementation/coding, they will never start.
The stakeholders celebrate. The requirements and some design people pant a sigh of relief. The developers started working. So, you see the green line moving up. Everybody is rushing to get requirements implemented. At the same time, bugs are inevitably introduced into the system. Of course at this time, there is no real testing being done. The bugs shown in red are there, but nobody acknowledges it. Everyone has faith that their implementation works. Morale is good.
Then the day comes when all required functionality is implemented and the system is handed over to testing. Low and behold, there is a huge bug list, some more severe than others. The development team now rushes to fix the bugs. They perhaps find more bugs and at this time, stakeholders may want to add more functionality – They are the pay masters, so they have the right to do so. Meanwhile, the developers are frantically fixing and the testers frantically verifying. Until one fine day when the bug count lowers to some acceptable level and the team releases the system.
To summarize the situation, tsunami-development emphasizes functionality over quality. With such mentality, the focus is getting to full function (i.e. functionality) before getting to good quality. So, it is functionality first, quality second. The proponents can argue their hearts out that heavy up-front requirement and design work are meant to achieve quality, but that is really far from the truth. Tsunami-development emphasizes functionality over quality.
This is different with agile iterative development, which emphasizes quality over functionality.
2. Continuous Iterative Development
The diagram in Figure 2 depicts an ideal lean and agile iterative development approach. Instead of postponing development until a complete requirements and design sign-off, the project team begins with what they know and what they are sure of. Those requirements that can be developed immediately can start immediately. Those that are still unclear and debate-able can wait, but they should not stop the stable requirements. So, the green line representing requirements implementation starts moving upward from day one.
Figure 2 - Continuous Lean and Agile Development
Starting early is not the only difference. There is a strong emphasis on quality and active bugs are re-solved within each iteration. Testing is a daily affair. Bugs found are fixed immediately. There is no need to wait. The red line representing active bugs is kept low. Quality is non-negotiable. It is a habit of the project team. Everyone is responsible for quality. Everyone emphasizes quality.
Compared to the tsunami development approach, quality comes before functionality. With tsunami development, the project team completes a full function or feature before fixing the outstanding bugs and release when the bug count is low. In the ideal lean and agile development approach, quality is always ensured. On top of this sound and firm foundation, requirements are implemented. The project team is ready to ship the product anytime. If the Product Owner says, “This is enough functionality to make available to the customer,” the team is ready to ship a stable build immediately. Perhaps there is a need for a release sprint, but no more. This makes the project team extremely responsive to market changes.
Development is now continuous. It is lean. It is agile. There is no start and there is really no end. The concept of a project takes second place. Here, the waterfall metaphor is appropriate. Features flow continuously along the stages you would expect for each feature – requirements, design, code and acceptance. The project team takes features from the backlog, finishes them completely before taking the next feature. It is a nice and beautiful waterfall – nobody gets killed, unlike a tsunami.
3. Facing the Reality
Getting to the continuous development model is not easy. It is very difficult indeed. Many do not know how or are not fully committed to achieving it. So, they do what they know and they call it their own brand of lean and agile. But without making an actual change, what they have is a fake. They have iterations, but they are not really iterating. They are puzzled why they are unable to reap the benefits of lean and agile development.
The situation is shown in the Figure 3. They start off development early. They do some kind of testing. Yes, and found bugs. Perhaps they attempt to fix the bugs, but not quite. Bowing to the pressure to deliver features, they frantically develop new features and the green line moves up. When the system they develop is still small, not so bad. Gradually the system grows. They cease to fix the bugs. The bugs remain as shown by the red line. The bugs accumulate.
Figure 3 - The Great Deception or the Great Reality
They have iterations to look at the bug count (perhaps). But since the bug count grows and nobody really bothers to fix them, gradually, they stop tracking the bugs. Eventually, they stop testing altogether because the system is so difficult to test. At the same time, the system (i.e. code) becomes more and more complex (represented by the blue line). Cyclomatic complexity goes beyond 20 or 40 or even a 100. Source files become big – several are 100kB. Much time is spent merging the files before checking in, which is accepted as normal behavior. There is no time for refactoring. But they have all the time to suffer the consequence of bad habits.
The reality remains that the emphasis is functionality before quality. There is no real improvement. They will eventually give up one day and blame it all on iterative development – saying that it does not apply to them. The situation is not very different from tsunami-development. Still, the fact remains that they do have iterations – not the ideal, but it means that there is still hope.
4. eXcuse Programming
Everyone (hopefully) recognizes that the continuous model is an ideal place to be. But the reality is a fake iterative development approach. Here is a very important transition point for most project teams. They have tried something. There is some experience on the thrills and perils of iterative development. There is some recognition of the challenges.
Now there are two groups of people. The first group (which represents the majority) does what is most natural. They complain! It is no wonder that the most popular software development method today is XP. XP does not stand for eXtreme Programming. XP stands instead for eXcuse Programming. Yes, they come up with excuses, all sorts of excuses.
- There is no time to fix the bugs. They need to press for speed. The stakeholders and customers are pushing for a release.
- Testing is too difficult. Too much effort. Let’s reduce waste by doing tests just once later when the system is more ready.
- What you (actually referring to me as a coach/trainer) are saying is too academic, too idealistic, too abstract – it does not apply to them. Their situation is different.
- We have our own way of doing things, or so they say.
Of course, there are those who are so extreme in eXcuse programming that they do nothing. They remain at whatever state they are. To this large majority, Figure 3 is a great deception, a great hoax, which they are pulling off. They would eventually return back to tsunami development.
5. Or Biting the Bullet
The next group faces the reality of the situation. Figure 3 represents the reality of their attempts. When I showed Figure 3 to project teams, they light up. They recognize that either they win the competition or their competitors kill them. They recognize that there are no silver bullets in software development. They (stakeholders and development included) recognize the need to bite the bullet. It hurts. But it is necessary. Better now than later. I like to work with this group of people.
The point which hurts the most is time. Everyone has no time. But somehow they manage time to invest in stocks, talk about real estate, complain about politicians and their bosses. So, they indeed have time. The key is how to use it. Figure 4 shows time reserved to fix the bugs so the bug count goes to zero by the end of the iteration. The allocated time is signified by the flattening of the green curve at the end of an iteration. This flattening means that there is a principled refusal to add new features into the system until the current ones are really completed and accepted. The allocated time is also used to perform code improvements – some refactoring to prevent complexity from exploding. So, the blue line representing complexity flattens across the project. Complexity can be evaluated by the permitted maximum source file size for the system, the permitted maximum Cyclomatic complexity of any function or method and the like.
Figure 4 - Biting the Bullet
At the beginning, stakeholders and team leaders make a deliberate and principled decision to allocate time towards the end of the iteration before accepting the development of new features. The idea is to instill the right values into the team – that quality is not negotiable and everyone is responsible for quality. When this gradually becomes a habit, team members will check and fix their work every week, then daily and then instantly until it is second nature to them. So, the flattening of the green line towards the end of the iteration will become shorter and shorter until it eventually disappears. When this happens – voila! They achieve the continuous development model!
6. Ascending Mount Fuji
Achieving the ideal iterative model is not easy. It is hard work. My coaching engagement requires me to be in Japan and everyone here knows about Mount Fuji. Mount Fuji is a majestic mountain which is visible from the skies as you approach Tokyo. The reason why I bring up Mount Fuji is that there is a series of famous paintings by the Japanese artist Katsushika Hokusai (1760–1849). He created the master piece – “Thirty-six Views of Mount Fuji (??????, Fugaku Sanj?rokkei). The first four views bear an uncanny resemblance to the four diagrams depicted earlier. When I have something more than just a white board, I can use the first four views in my presentation slides. I show these four views below in order.
| The Great Wave off Kanagawa ?????? – Kanagawa oki nami-ura This represents the big wave of tsunami development (see Figure 1), a very risky way of development |
| South Wind, Clear Sky ???? – Gaif? kaisei This clear sunny sky represents the ideal, which we want to be eventually. The ideal is continuous iterative development (see Figure 2). |
| Rainstorm Beneath the Summit ???? – Sanka hak? This represents the stormy reality (see Figure 3), the challenges (the rainstorms) which the team faces as they make their transition to agile development. Very often the team see the storm and lose sight of the ideal. |
| Under Mannen Bridge at Fukagawa ?????? – Fukagawa Mannen-bashi shita The phrase ?? (Mannen) means ten thousand years. Yes, the team needs to allocate time and resources to fix the bugs and issues by the end of each iteration and cultivate the habit of ensuring quality first. This is grueling transition depicted in Figure 4. |
I have used the four diagrams above to help teams big and small understand where they are in iterative development and help them to understand how to get to where they can progress towards the ideal continuous development model. The diagrams appeal to the logical brain, the views of Mount Fuji appeals to the culture. Moving to agile is a change of mindset and we attempt all angles. They instantly get it. They face up to the reality. Some understood the challenges and honestly remain status quo. At least, they are honest. Others understood the challenges and press forward. And we are there to help them
Pan-Wei Ng, Ph.D. is a firm believer in a lean and agile development. He strives to improve quality and reduce waste. Dr Ng helps companies in Asia adopt and scale lean and iterative development and other practices. He believes in practicality and leadership by example. In fact, he has recently lost 20 kg in 3 months as a result of applying a lean lifestyle. As the Asia Pacific CTO of Ivar Jacobson International, he contributes to the innovation of software engineering practices. He co-authored of Aspect Oriented Software Development with Use Cases with Dr Ivar Jacobson and believes that aspects and effective separation of concerns are important enablers to rapid lead and agile development.
