The Importance of a Directory Structure for the Development Process

I see software configuration and change management coming to the rescue for process control, but before they do, they need to take a broader look around and be aware of more than their traditional domain of source code, because otherwise they could end up being part of the process problem, rather than part of the process solution.

Assumption
This article assumes the use of some of the rational tools for software development process. This does not imply it cannot be done with other tools, provided they use a file saving paradigm and not a database container for all data elements, so that the CM tools can manage each configuration item independently.

Introduction
Having worked on a couple large financial software projects over the last few years (one of 120+ people and another of 150+ people over multiple sites) and been part of trying to implement the rational unified process while the team were learning an array of new technologies and concepts such as requirements gathering, use case modeling, testing, iterative building, change and configuration management, software development tools, etc., I can tell you that

one of the biggest lessons I learned was how important it is to organize and align what I've termed container structures. As in the UML class diagram to the left, for the term container structures, think "directory structures", directories and files within directories and files recursively.

However what I am trying to describe here are concepts not only relating to a file management system, but rather all software development tools, such as a UML modeling tools container structures, test tools, requirements gathering tools, content management tools, etc. such that they all align and co-operate along a similar philosophy. This can include any software that has an explorer type interface, which allows storage in a tree structure. Think of the browser window in rational Rose, or the folders window in Microsoft Outlook, for example. If we organize a Rose UML model, why not align the package structure of this UML model to the directory structure below it?

Problem
The problems I have seen on projects where directory structures philosophies were not thought out carefully enough; or kept being changed through the project life cycle were huge causes of unnecessary problems. Problems that inadvertently generate huge volumes of work in converting a directory structure from one philosophy to another, moving current contents of the old structure to the new structure to keep the artifacts in a place where they could easily be located for builds, audits, etc. Of course in the rush of the project and under deadlines pressure, this task becomes such a low priority that it never gets done.

On one project I saw the fundamental directory structure change three times, on the grounds that it was (1) either too complex and no-one knew where to put or find things or (2) it was too simple and couldn't cope with the varying demands! This just served to confuse everyone even more and, eventually, some people were using one structure while others were using another!

Without getting the directory structure of the software system under development correct and in synchronization with the Configuration Management policy and the process spells disaster before you've even started a project.

Background
Let’s take a look at how the software development process has evolved and still continues to evolve. While this progress has in no way been linear and sequential, it has from my experience tended to start from the writing of code and worked its way outwards (see diagram with red horizontal arrows below), with other practices and disciplines coming towards it from the outside, such as project management and configuration management (see diagram with vertical arrows up and down).

·       First off, we had the raw production of code by thinking it through and then coding and testing (60's and early 70's)

·       This evolved into code unit testing and project management for software development (70's early 80's)

·       Later, the importance of architecture, product design and at the same time configuration management was recognized (mid to late 80's)

·       Design gave way to analysis while unit testing gave way to user acceptance testing and this was added into the picture (also mid to late 80's)

·       People then realized that requirements should be gathered and deployment should be better managed (late 80's into early 90's)

·       As changes continued to undermine projects, change management was brought into the picture (early 90's)

·       As more and more tools were being required for the software development process over multiple projects environment and program management were brought in to the software development process (mid to late 90's)

·       More recently system requirements has started tying into business modeling and knowledge architecture leaving us with a group if disciplines as shown in this diagram (early 90's, but more recently implemented to a greater extent)

The point of the diagram above is to illustrate how more and more configuration items have started turning up in our typical software development projects. What started out as a simply necessity to branch and merge the source code, has evolved into complex branching and merging of designs, requirements, models, tests and other general items. The red horizontal arrows show how the coding process expanded outwards, while the red vertical arrows show how other areas of business specialization have been brought in to manage the complex area of software development.

Areas of Specialization
Different areas of business specialization each have a different emphasis in terms of what they try to achieve and what they consider significant for success, though these boundaries do cross and blur.

Architects and Designers Tend To Be Product and Code Centric

Architects organize their lives around the product they design and build. In an attempt to isolate systems into loosely coupled but highly cohesive components, the container organization tends to express itself in the system components with their constituent sub-systems. The teams also usually align themselves around the components so that certain people are responsible for certain components. Where and what does the delivery comprise of?

Project Management Tend to be Features and Time Centric
Project managers, on the other hand, tend to try and organize themselves around all the teams disciplines and within that focus on a features delivery based set of containers over time. Who is going to deliver what when?

Business, Requirements and Testing Tend to be Feature Centric
The business, requirements and testing people, like to organize their lives around the business and system features that are to be delivered. What and why do the business and the system do for this iteration delivery?

Configuration Management and Legal Requirements Tend to be Standards and Audit Centric
Pressure from the business in terms of adhering to audit, legal and quality fall on the shoulders of the configuration managers or testers (depending upon the organization), which tends to be standards centric and forces implementation of some CMM level or ISO 9001, SPICE, etc. CM have to try and bridge all these different perspectives, and it’s easy to fall into the trap of just satisfying the architects and designers in being product and code centric.

We need to make sure we cater for all these different forces and requirements on the structure we implement so that the software development processes all work harmoniously together. File Directory Organization
There is a very fine line in balancing the understanding of project members about the structure of the file system with the complexity of the file system you deploy. If too complex, they won't understand it or use it properly. If  too simple, each person will begin inventing their own more complex version.

Good guidelines and training early with fairly disciplined rules on directory creation at certain levels should help with this problem. This UML class diagram attempts to explain the container structure for file systems.  I'm sure it could be more elegantly modeled. This suggested directory structure would consist of conceptual levels of decomposition starting at the enterprise level. Each enterprise may have one or many Programs or Systems under development. Each system would be made up of some sub-systems. Each sub-system would be made up of components, which may be shared across sub-systems and so on. At each level which is just a containing directory, one would have either a set of lower level directories or a set of discipline containers. Within the discipline containers, would be artifacts such as models, project plans, Word documents, spreadsheets, XML files, test scripts, deployment information, environment guidelines, etc.

Requirements analysts could use the system level requirements discipline container for all their use case models, for example. Source code would be stored in either the component level or the sub-system level implementation discipline. On one of our projects we chose to call Implementation "Src" instead.

While this might not make immediate sense to someone on the team who is not too knowledgeable on UML modeling or the rational unified process, if it was explained using example directory structures and a set of guidelines and rules, this concept is really quite simple, as it embodies recursion and fits in with the general understanding of system decomposition.

UML Model Directory Organization
The rational unified process suggests that there are various different views one can take of the same system. These are the use case views from a business and requirements perspective: logical, implementation, process, and deployment view from the architecture's perspective. These are called the 4+1 views:  four architectural and one requirements view. Within this framework of views, the next level down is organized by model type, or model more specifically model purpose

·         Under the use case view, we could typically have for example business use case model and system use case model

·         Under the logical view we could typically have an analysis model, a design model an implementation of the design model and a process model

·         We could have the component model under the component view

·         We could have the deployment model under the deployment view

Rational Rose allows the use of one model file with many .cat files, similar to the concept of a subdirectories (.cat) within a directory (.mdl). Each specific model would then live in one or many .cat files which could then be saved in the directory structure as outlined above. The single overall model file could then be stored according to the owner of the overall model, either the architect, configuration manager or the process engineer, whoever ultimately takes or is given ownership of it. This is where the alignment between container structures starts coming in.

Other Configuration Items Organization
Without going into detail on the items below, the following could be considered for configuration management in the same way:

·         Project /program overall and iteration plans

·         Project Web layout

·         Business use case models

·         Non-functional requirements

·         Use case documentation and trace-ability to models

·         Software architecture document

·         Change management structures and processes

·         Template documents

·         Vision documents

·         Risk lists

·         Security models

·         Candidate architectures and prototypes

·         Project metrics

·         Infrastructure models

·         Configuration management plans

·         Deployment plans

Conclusion
With the experience I have had to date of large software engineering projects, the single biggest stumbling block to speedy progress was not having the basics in place early enough. Trying to change the basics in full gallop is like shooting yourself in the foot at full gallop.

Make sure the container structures align across all disciplines. Make sure all tools structures have a place in the directory structure. Make sure that the configuration management process and change management process work within these structures.

Ideally if you can have your CM structure and process in place ready, working and tested from the last project, you are off to a fantastic start on your new project. If not, the next best is to have it ready in the inception phase of the new project, so that when the work begins in earnest, everything will have a place to live.

Appeal
We are building a set of ready-made documents, such as configuration management plans, directory structure guidelines, software development plans, etc. Any contribution on what you have found useful in your unified process project to help others get a kick start on their unified process project would be welcome!

We envisage an internet site which process engineers can visit and download existing documents that can save them time and energy. Either you or we can convert all specific company information into general or anonymous terms to preserve and protect company specifics and integrity. Please submit them to us at Submit a Document to Publish

References
1) Rational Corporation. Rational Unified Process. 2002.

2) Senge, Peter. The Fifth Discipline. The Art & Practise of the Learning Organisation. 1990.

3) Edwards, Charles. http://www.processwave.com/ April 2002