A number of issues that not only affected the SCM group, but the entire software development group arose while I was working at a company as the Director of Process Management. My primary focus was to develop a methodology (or set of methodologies) that would (1) reduce software build time and (2) make the overall software development lifecycle more cost-effective. The job at hand was to automate the company’s many manual business processes using a new application development system for automating e-business change.
The system was not so complex, so I’m told, that it could have been created in one year using Visual Basic and SQL server. However, company executives had made some extremely bold promises to several large and highly reputable companies who funded the development effort with the proviso that the company had to use their business applications and tools. This resulted in an agreement to incorporate a range of technologies, some of which were rather complex and unsuccessful, to first build an application for the company that would automate real estate escrow and title accounting, and second to market (pieces of) the application. There was also a commitment to use the Unified Modeling Language (UML), Rational Unified Process (RUP), and support a J2EE enterprise architecture that would cure all woes and make things easier for all.
Chaos and ad hocism prevailed throughout our organization. Architects did their own thing, which mostly entailed dreaming up ridiculous (not very practical and expensive) system concepts and peddling snake oil to the CIO while business analysts tried unsuccessfully at first to document business requirements and business rules. Developers were expected to interpret the verbose and very complex requirements from the unfinished and dynamic (constantly changing) requirements specification. The results were very predictable. That is, developers really believed that the users were stupid, and the users refused to collaborate with “arrogant” software developers. So the software never met user expectations. Things went from bad to worse when senior management determined that the developers they hired were slow learners, so they hired contractors. At the worst point, a total of 50 contractors who were getting $125 - $250 an hour, with no hourly limitations, were hired to “fix” or resolve the problems.
The CM problem domain was identified when the 150+ people working on the project were given the authority to do whatever necessary to make things right, which by the way, went on for at least 18 months. We started out with a small, inexperienced SCM group who worked their tails off in an attempt to make things work. There was no change control process in place, so new requirements were inserted at will whenever an analyst agreed to add it to the release from an irate user. It took the SCM group a significant amount of time to understand the problem domain (which no one ever did in my estimation) and then apply configuration management to objects within that domain. There was adequate budget, but the SCM group was technically not ready to invoke or even establish basic and essential patterns needed to implement effective CM. Designers were not really designated, so when someone thought up a new or seemingly revolutionary concept, the design was altered to reflect the new feature set or function. Software developers wrote their code without validating any functionality (no test cases were written), and then threw their code “over the wall” (to the SCM group) for integration builds.
Things got real ugly when it was determined that the applications the development group was forced to use (and mandated by senior management) had interfaces that caused more problems than coding defects. Builds broke constantly. More contractors were hired. Still there was no change control.
I developed an iterative development methodology that would save similar projects in the future, but gave little hope to this project when I briefed the executive staff on how it worked. It was embraced and rolled out to the project with the understanding that the transition could not hamper or otherwise impede ongoing project progress. (Yeah, right, like any progress had been achieved so far.) I conducted classes on the new methodology to the entire project group and senior managers off-site so their full attention was maintained. Extensive studies were conducted within the SCM group to determine why builds took so long and why so many of those builds broke. The results were interesting to say the least. Developers were checking in untested code, which meant that there were no peer reviews being held, little-to-no supervision, and a significant amount of gold-bricking done because user expectations were not very clear. Requirements changed faster than developers could consume pizza, and those changes were not communicated throughout the development group. Developers did not always check in their code before leaving for home each night. Parallel development was not allowed. Branching was done late in the development game and just before a required build. The Product Manager wanted change control only after a successful build had been performed, which meant that baselines were nebulous at best.
The change control process was developed in flow chart format first, so I could walk everyone through the process and get approval or sign off. Once the change control process was in effect, things got a little slower, but builds improved. That is, they became more successful – not faster. I suppose the reason was that people became more conscious of their work effort and products before promoting code to SCM.
In spite of all this, the problem’s roots were not in the workflow processes, although some needed significant improvement, but in the technology. The SCM solution and the J2EE application server were at odds. Mostly, there were serious formatting issues and the SCM group had to reconcile (reformat) the code in order to complete the builds. This went on for several months before someone finally wrote some scripts that automated the reformatting process. There are a lot of things that could have or even should have been done before things got real bad and frustrating for the development group – but they didn’t and work products suffered as a result. Millions of dollars were wasted on unnecessary contractual support that could have been solved by better internal and external communications, and a better understanding and appreciation of the values that can SCM bring to any software project.
The major lesson learned in this scenario, was that all the money in the world cannot fix an ineffective process if people don’t work together as a real team. Looking at things retrospectively, SCM planning was never completed. Come to think of it, project planning was non-existent. Someone at the executive level should have done a little more homework and perhaps listened a little more carefully to what their staff recommended. Clearly, not a very good set of circumstances for any CM manager or release manager, but opportunities to excel abound!
Dick Carlson is the founder of Software Engineering Solutions, a training and mentoring consulting firm. Dick has 20+ years of software engineering experience that focused on training and mentoring, development and implementation of software lifecycle methodologies, software configuration management, software quality assurance, and software process improvement. Dick has trained and mentored teams and individuals on efficient SCM activities, project management, requirements development and management, risk management, business modeling, and business re-engineering. He has also been involved in software process improvement initiatives in preparing organizations for SEI CMM Levels 2 and 3 compliance.
Dick can be reached by email at [email protected]
