Agile SCM - Review of 2007 and Predictions for 2008

As is traditional for a January column we are going to look back at 2007's review and predictions and look ahead to the coming year. Agile is mainstream these days, which basically means it has crossed the chasm from early adopters. Along with agile crossing the chasm there has been an increased desire to make agile and lean development scale across the enterprise. That is partly responsible for the focus on collaboration, and the working of wiki's into the overall software tracking and the knowledge-management process. But this also has some bad effects as well (in addition to some good ones).

"Agile" everything
All the Agile-named products and services claiming to be Agile (legitimate or otherwise) will be even more mainstream, whether it is claiming to be "Agile CM", "Agile Requirements" or "Agile Waterfall" or "Agile Outsourcing", (or whatever else someone can think of to make a buck)

"Agile 2.0" and "Post-Agile"
Of course the above also leads to the question of the current state of what we've previously seen dubbed as "
Next Generation Agile" or "Agile 2.0". Where is that headed? Lean & Six Sigma have officially been married into "Lean Six Sigma".

• We're seeing more combination of Lean with Theory of Constraints (TOC) these days (and see the recent book "Lean Project Management" by Larry Leach, author of one of the definitive texts on Critical Chain PM). 

• There's been recent talk of a "Post-Agile" movement as something that goes beyond merely Scrum and XP (and even Lean). And there's also been a lot of backlash against that idea as well.

• More recently, David Anderson started a KanBan development for sustainable software engineering movement (and YahooGroup) that claims to do away with iterations when you can have short-enough releases and decoupling release content planning from feature content planning so that the latter can occur more dynamically as feature development progress. The ideas are based primarily on Kan-Ban principles from the Toyota Production System (TPS) together with TOC applied to more the software engineering lifecycles.
• Craig Larman has an upcoming book on scaling & adopting Agile/Lean for successful large, multi-site & offshore using "Large-Scale Scrum"

Global Collaboration and Convergence
We previously mentioned the rise in globalization and the resulting effects on software development. This is an obvious and growing trend with out-sourcing, in-sourcing, off-shoring, near-shoring and all their brethren.

The challenges of this distributed development for SCM are obvious, with various tools and technologies being marketed, developed and adapted to address these needs.

An interesting development is that it is increasingly not just teams within the same company who need to collaborate, but also different companies and fluid networks of partner companies. Within the same company it tends to be easier to mandate a single SCM tool to help address the distributed development requirement. Even in single companies, you get new acquisitions, off-shoring partners etc which make life more complicated.

There are increasing numbers of companies that have fairly fluid requirements to work with different partners remotely. This can be as simple as individual contractors working from home, to third party companies completing a particular project which may take just a few months. Life becomes even more complicated if you have integrations with third party software, or some element of open source software to manage.

One of the key challenges when working with third parties is the likelihood of both sides using different SCM tools. You are thus faced with collaborative working requirements across tools - and traditionally vendors have tended to be less interested in supporting these sorts of scenarios. Any problems and the finger tends to be pointed at the other vendor!

In the past we've seen convergence of software configuration management functionality and software change management functionality, and when you combine that across the lifecycle of different kinds of artifacts (requirements, models, code, tests) the result seems to be ALM. With ITIL (see below), that then becomes part of the service lifecycle. We think more is coming:

• In the last couple of years, we think "Content Management" has begun to enter the picture
• More recently, we think "Collaboration Management" has burst onto the scene, fueled in large part by the Agile movement, and also by the "flattened world" which has increased the desire for geographically distributed development

Tools, Collaborative Frameworks, Mashups and "all that Jazz"
It has been an interesting year within the SCM vendor community, with various product developments, mergers and acquisitions. We looked at some of those developments last time, but there are some new approaches.

We were greatly in favour of the Eclipse ALF (Application Lifecycle Framework) project as a cross vendor approach. Serena deserve recognition for their efforts in driving this project. On the face of it the project rather looks as if it has lost momentum - all milestones and news   pages on the project home page mention nothing after the end of 2006! However, conversations with chief evangelist Kevin Parker indicate that the effort has metamorphosed. Thus Serena is perhaps looking in this direction for their integrations with other tools - reusing work done for ALF. Indeed, according to Kevin:

"There is a remarkable pent up demand for applications to be developed that are not being addressed by "formal IT" but rather by "shadow IT". [...] for some, [perhaps] for many, mashups is the answer." 

So it looks like Serena has instead taken a different tack with the repositioning of their TeamTrack product as "Serena Business Mashups". The concept of "mashups" has certainly leapt to prominence during 2007, and there are many attractions to the idea of being able to quickly and easily combine the functionality of 2 or more applications into a new application. Thus Serena is perhaps replacing the need for ALF.

The SCM challenge of mashups is that your new application now has dependencies on other services. If these are controlled by third parties, you need to keep a close eye on changes and version compatibility. You might also wish to consider what SLAs you have on the dependent applications and balance that against the business benefits obtained.

In our view, mashups are a great way of trying out ideas and producing a proof of concept and a working application. If it is successful, you may have to go back and look at ownership and risk issues, and spend more money on things like SLAs or other controls. Interesting times for Serena in any case, and we look forward to the proof of the pudding for their new direction in next year's review (and being non-partisan wish them well!).

As mentioned last year, IBM has been trying to "Jazz-up" things with its Jazz collaboration management approach. We haven't seen it catch on yet, but if it does, then we expect 2008 to be the year (mostly because we think that is when they planned to release part of it to be open-source). The "open-source process" in 2007  (
or "open commerical" license ) only with invited partners has been interesting. ClearCase has been perhaps suffering recently with a lack of major product enhancements, although the Jazz-related efforts, particularly the recent announcement of IBM opening up access to Jazz ALM, takes it in a new direction by enhancing collaboration.

Meanwhile IBM will be digesting its recent acquisition of Telelogic and we watch with curiosity to see what happens with the Telelogic product line. Just considering relative market share and product strengths, we assume that the DOORS requirements management product seems to have a clear future, although the future of the Synergy product looks perhaps a little less secure longer term given its overlaps with ClearCase (but maybe they are big enough to support both?!).

Another interesting response is AccuRev 4.6 for ClearCase - which gives a new spin on collaboration as a way of helping competition (concurrent engineering), in this case by allowing a competing solution (AccuRev) to co-exist with (in their eyes) a "legacy" solution (ClearCase) for purposes of evaluation/comparison and (ideally) transition & replacement. AccuRev is clearly targeting ClearCase with their marketing, and this approach seems to be their best bet for getting in to existing accounts and encouraging potential customers to try them out.

Vendor competition is generally a good thing for customers!

Among other vendors, CollabNet launched its CollabNet Enterprise Edition 5.0 during 2007 featuring increased collaboration functionality through wiki capabilities among other things. They are also touting the extensible platform and ability to integrate with other tools. As part of a similar trend, Borland launched its Open ALM Platform.

Continuous Integration and Sexy Builds!
Continuous Integration has been catching on like gangbusters, including not just the frequency of integration, but the number of commercial tools claiming to support or implement it in some way. This is a key Agile practice, reinforcing as it does continuous and early feedback.

In addition, as the Dr Dobbs June 2007 article notes: Build is against all logic, sexy - what's with that?! We were very heartened to see this article, and the listing of a variety of (relatively) new vendors who are making a good living solving the problems in this space. We do of course claim prior art in our February 2007 article The Renaissance Builder ! And, in fact, builds have always been central to the SCM Patterns, starting with the patterns from 1999 written up in Software Reconstruction: Patterns for Reproducing Software Builds.

Onwards and Upwards with Open Source
Subversion 1.5 is a big event in the open source world: it marks the presence of the long awaited "smart merge" support in the most popular open-source VC tool. While not yet officially "out of the door", it is very close to final status, and has been imminent for a couple of months. While some of us thought the lack of merge tracking support would hinder the adoptance of the tool, it doesn't seem to have done so, but this is going to make a big difference in our opinion.

Indeed, it was hoped that the release would be announced at the first Subconf in Munich in October 2007. With several hundred attendees, and many industry presentations, the event was proof of the success of Subversion. As well as a couple of excellent keynotes from Brian Behlendorf and Karl Fogel (these address various issues around the difficulties of starting and guiding successful open source developments), Martin Doettling presented some interesting figures:

• Estimated user base now exceeds 2 million
• 10x growth since 1.4

• Large numbers of enterprise users

The release of Bazaar (bzr) 1.0 is also significant in that it shows the vibrancy of the development of distributed tools. Other tools continuing to gain mind share and users include Git, Mercurial and Darcs.

ITIL V3
We mentioned the rise and importance of ITIL last year, and this has only strengthened with the release of ITIL V3 during 2007. In general, V3 makes the link between ITIL's best practice and business benefits both clearer and stronger. The main development is that V3 takes a lifecycle approach to guidance, as opposed to organising according to IT delivery sectors. It is more tightly aligned with business usage, and is more compatible with internation standards such as ISO/IEC 20000.

ITIL has achieved a lot of management mindshare in countries such as the UK, the Netherlands, Scandinavia, Canada and Australia, and is steadily growing in the US.

In some ways ALM can be viewed as a subset of the service management lifecycle, and there are many benefits of taking this approach. Application development is often 30% or less of the whole cost of the application through its life. The advantages of considering the transition into service (release management traditionally) from very early in the lifecycle, and also taking a holistic view of the whole lifecycle through to maintenance and retirement, and considering the integration with other services, tend to be more cost effectiv    e in the long run.

The Backlash
The interesting thing with some of the developments discussed above is that there is in some quarters almost a backlash in that some approaches have been overhyped and are perhaps harmful. We are certainly of the opinion that Agile SCM itself has been accused of throwing out good SCM principles in pursuit of some uncontrolled chaos or "license to hack". This is a failure to communicate the disciplines required for good agile development.

A backlash is also a positive sign in that it fuels debate, and as long as this is kept constructive, the results are only going to be positive for all of us.

In a recent 2-part article in Better Software magazine (Nov & Dec 2007 ), Gertrud Bjørnvig, James O. Coplien and Neil Harrison discuss the problems with TDD (Test Driven Development). This builds on Coplien's blog article: Religion's Newfound Restraint on Progress. The reference to the research by Siniaalto and Abrahamsson is perhaps over stated. This paper (available for purchase via IEEE) actually does two things: summarises 16 studies (in industrial, semi-industrial and academic settings), and does a comparitive case study of the effects of TDD on 2 iterative test-last projects vs. a single TDD project. The summary questions the generalizabilty and significance of some results, but does conclude that TDD may enhance software quality and may not decrease developer productivity!

The results of their comparative study are rather more ambiguous - they indicate that TDD does not always produce highly cohesive code as predicted in the literature, but qualifies this by referring to their single TDD project having less experienced developers than the other 2 projects. It does note that test coverage is improved by TDD. This sort of paper is definitely to be encouraged, and more examples are needed, but we should not perhaps read too much into one example. The rest of the Better Software article contains many good points and points out various dangers that, in our experience, do occur if TDD is not sensibly applied. For now, we think the jury is still out, but as ever a realistic approach, as opposed to naive belief in the hype, is going to bring you the best results.

On the ITIL front, there is anecdotal evidence that some companies and individuals are annoyed that their ITIL V2 qualifications now need to be redone to make them ITIL V3 compliant. In addition, the various changes in terminology and document/process structure can appear more complex than they are in reality, especially to organisations that have made progress in using ITIL V2 to improve their own processes. This in itself is to be expected as part of the change process - homeostasis suggests that changes are well examined before being accepted. Sharon Taylor, ITIL V3 Chief Architect, spent some time refuting the "nay sayers", in an entertaining way, in her keynote at the recent itSMF UK conference.

Conclusion
In our view, the industry continues to move forward quite rapidly which means we all need to keep on our toes! The backlash section may seem negative, but to our mind it implies that most of the principles do in fact remain the same even if the terminology changes. We do need to avoid getting caught up in the hype of new fads and buzzwords, and yet also remain open-minded enough to recognise genuine progress.

Happy New Year and keep reading!