While many people intuitively feel that we should integrate the development tools we use in our software development environment to better manage the development team, the task of actually doing so can sometimes be as big if not bigger than the software project we are trying to develop. It's hard enough trying to implement individual software development tools while you are in the throes of building software, let alone get multiple sets of these tools to start synchronizing and talking to one another.
In this article we explore the type of process and tools that should support the process. We look at the organizational issues you face in implementing tool integration. We look at the challenges we face with the tools in software development, the likely intersection and where the tools would be sharing certain information. We also spell out some of the dangers and issues that are likely to come up; with some suggestions to consider when implementing this integration...
Set the Scene for What We Would Like to See
Before we start my assumption is we are working on a medium to large project, with a medium to large team of people.
So it seems a great idea for a person on a medium to large project to have functionality along the following lines (give or take the different team structures and sizes):
· When a business modeler /requirements analyst /programmer /tester /project manager /other finishes a development activity
· The set of related files for that atomic activity are checked in from the development tool (UML model, Word text, IDE code, test tool)
· The set of files are then integrated to the sub-system /system they are working on either by programmer or integrator
· the functionality and behavior are[unit tested /reviewed against some pre-written test /criteria
· The [unit tested /reviewed] set of files are integrated to the integration stream for the particular [sub-system /system]
· They then fill out relevant comments to the change /create /defect request and sign it off
· They then check out the next task on the centralized , prioritized list of change control board (CCB) approved /changes /additions /defects
· They check out the linked input artifacts for this task integrated to the business strategy /business model /requirements/ analysis and design/ code /test
In the meantime:
· The metrics of tasks being developed is updated for the project manager /architect /team leader /other management
· The change requests done are then notified to the next in the list (business modelers /requirements /designers /programmers /testers) to notify them of more functionality available for (modeling /requirement gathering /designing /coding /building /testing /deploying)
· Once the build / integration person has enough new changes a system is built according to the valid set of files completed at that point, all referenced by change control records, including all other artifacts like (use cases /other requirements /design models /user interface graphics /content /code /unit tests /system tests /project plans /etc.). This could be one change or n changes between two time points, start and end of iteration
· The whole lot is then packaged up into a release and made available centrally for deployment by notification to testers and production. Once or many times per iteration up to the project
· Then anyone on the project can fill out (new requests /defects /change requests) for processing
What this needs is a lot of integration. It needs:
· A new request /defect /change request mechanism to be available to all people on the project
· The SCM system available to all people on the project, so that artifacts (code /use cases /project plans /design models /tests /many other artifacts) can be version controlled, integrated and built
· Single business strategy and models to be captured by many and centrally held and shared to many
· Single requirements to be captured by many and centrally held and shared to many
· Single models to be diagrammed by many and centrally held and shared to many
· Code and code unit tests to be designed from models, created and built (or code generate models backwards)
· User graphics to be designed, created and merged with the code
· Database data to be designed, created and added into code
· Messaging data to be designed, created and added into code
· Security mechanisms to be designed, created and added into code
· Tests to be designed and scripted by many and centrally held, executed and shared
· Deployment of the combined set of systems files to be packaged up from the build ad notified to those receiving them
· Metrics to be shared from raw data output form each tool and analyzed centrally for those in management who require information to make decisions
· Once in production needs storage management, and system administration
· Guidelines on all topics to be shared by many
· Tool mentors on all tools to be shared by many
Do we decide to build something like this? Who says it should be done? How do we build something like this? In what order would we build it? Will the tools we have/decide to buy support this?
Organizational Implementation Issues
The type of organization and projects they undertake (no pun intended) play a part in the decision to integrate tools. Obviously it does not make sense to spend months of effort integrating a set of tools if this development project is a one-off to the organization.
Even if there is more than one project which would use the integrated tools once implemented, then if a project is say six months long and the integration effort is around the same length, that particular project will not see much of a benefit. In fact that project may well be stunted through all the actual implementation, changes and learning curves, etc.
Essentially different approaches would be necessary according to the organizations particular situation and available time, knowledge, willingness to improve and budget. In some cases we may choose only to implement a simple SCM tool as the core product. In other cases we may choose to implement an integrated SCM and change management system. We could have a strategy of integrating tools into the project as we go. It depends on what is already there and how well its integrated already.
The people who are responsible for making this change are varied:
· A champion to sell and direct change effort
· A process engineer and tool smith(s) for the process definition and technical implementation (in some cases the champion and process engineer are one person)
· Senior management for putting in finance, resources and enabling the process engineer enough power to effect the changes
· Project management for helping control the change in the varied teams on and off the project
· When the project team members to buy into the change, then go through the learning curves and work with other teams to effect the change across the whole team, especially if the teams are geographically dispersed
· All of the project members such as PC support, hardware and network staff who control the organizations infrastructure
· External organizations such as tool vendors, development sub-contractors, etc.
· There are probably others and this will vary from situation to situation
Tool Challenges We Face in Software Development and Tool Integration
The software development implies rather a large list of challenges, however within the context of the tools integration, the list below is a more scaled down set of challenges:
Challenge | Explanation |
Tools vendors use to get integration to work: · between their own product offerings same version · between their own product offerings different version · between other vendor offerings | On a typical large project there are probably going to be some tools of different versions because they were bought at different times, which do not quite work together sharing files, having stunted features, etc. There will probably be one camp of people who like modeling tool X and another group who swear by modeling tool Y. |
Implementing integrated tools across different disciplines is non-trivial. | People from different teams generally don't work together naturally, there is a feeling of, "My team’s OK. We have our processes, don't impose your teams extra requirements on me." |
Multiple Tool Databases of information. | Each tool stores data in its own format. Some in simple files, others in Small databases, others in SQL database, some configurable others not. |
Processes embedded in tools that may be ideal for one particular team may not suit other teams. | The vendor of a tool modeled their tool around some pre-conceived way of working. It may work well for the people who work this way. It may not work at all well for other teams who need to work differently because of their own particular situation. |
Tools that don't accommodate externally defined process. | Some tools can be configured for change. They have a meta definition ability, which after setting up certain attributes and properties will allow them to mold themselves around the user’s specific needs. Other tools are far more rigid in their approach and can therefore not be used easily or at all. Try and look for tools that can configure easily. |
Different tool vendors and versions of tools around the organization doing the same job. | On a project, team A is using modeling tool X and team B is using modeling tool B. It now becomes much harder to share information, especially if both tools do not allow sharing of data. Even if they do, there is a whole extra set of processes to transform data from one file structure to another. |
Budget available for buying the tools we really need. | Some tools are very expensive and others are far more affordable. The price / performance ratio is not always the same. The real issue is once you find the ideal tools for your process, does your organization have the budget to not only purchase the tools, but purchase the maintenance and implementation consultancy, etc.? This could be a huge stumbling block. Make senior management aware of just how important a good process can be in producing quality software. |
Multiple tool vendors jockeying for their own position of strength on the project | Often a project will devolve into politics when multiple vendors are involved. It's quite common to discover that the vendors in these situations often start jockeying among themselves to try to grow their business, which can be a major problem. A common strategy is for the vendors to put their own pet architectures or processes forward and then snipe at the solutions presented by the other vendors. The host company will want to actively manage these people. *1 |
Challenges in Implementation Between the Tools, Project and Process
In the business use case diagram below, you can see the goals of each of the individual people in the roles as drawn. In their own right they are each trying to do a valid honorable job of work. Each of the goals have a certain amount of conflict with the other persons goal, though, which inevitably causes problems both for the individuals and for the project as a whole if it is not managed properly.
What are these three main conflicts?
Between Goal 1 | and Goal 2 | we have this conflict |
Implement s/w dev process | Build s/w for this project | The programmers are working against deadlines. People are naturally reluctant to change and more so under pressure. They don't want to be told to do a new process in a different way than what they are used to, it will slow them down, put them off their stride and concentration. The process engineer wants them to improve the process because it will help make the overall integration work better and thus the quality of the produced software more understood, reliable and controlled. |
Implement s/w Dev process | Implement s/w dev tools | The process is defined by the process engineer according to the best overall way that will support the type of people, project, and software under construction. The tools may have been designed for use in a slightly different manner. The tool smith and training material will try and tell the people to work as the tool expects, which may be different from the process that has been defined for the team. Compromises have to be made on both sides, but far too often the tools are rigid and the process has to change to accommodate the hardwired tools. |
Build s/w for this project | Implement s/w dev tools | The tool smiths have to try to understand the process from the process engineer, understand how to implement this in the tools, do trial tests to confirm that the tools and products support the process; if not write scripts or plan work- around options. Once resolved they then plan an implementation and individual tool rollouts. Once done have to help the individual team member learn and use the tools. The programmers are under pressure to deliver working software and are reluctant to give up time on their workstations to have new tools installed and have to learn new software. |
Development Tool Categories
Here is a list of tool categories*2 which are likely to be used on a large project.
· Business process reengineering (BPR) tools
· UpperCASE tools
· LowerCASE tools
· Configuration management tools
· Content management systems
· Database management systems
· Documentation tools
· Management tools
· Metrics tools
· Multimedia tools
· Performance tools
· Process engineering tools
· Requirements management tools
· Security tools
· Storage tools
· System administration tools
· Test tools
· Deployment tools
Development Tool Integration Points
Below is a rather spider web looking relationship between the possible tools on a project. This implies there are relationships between each tool and almost every other tool. I'm sure there are more relationships and tools that could be added to this, but this gets the point across. Look at the likely integration point and what is being integrated at each point. In some cases there are two way integrations.
Suggestions to Consider When Implementing Integration
· Keep it simple silly. KISS principal.
· Understand what you already have in place before you start, and where you want to get to, then incrementally move from the beginning to the end.
· Implement the changes in planned manageable iterations following a predefined strategy.
· Work out the dependencies between tools and start with the most important one everything else is dependent on.
· Try and keep this as a separate project with its own plan and risk list, etc. Run it in parallel with the actual project, but keep the goals and improvement iterations aligned with the actual project.
· Even better, if you can, try to set-up the tooling and process (under its own project) before the real project starts, so that when the project starts they just come in, get trained and use the tools.
· Define who will do what bits of the plan.
· Keep and manage a continuous risk list. Run it like a mini Unified Process development project.
· Warn members of the team what to expect for each iteration give them enough time before you do it.
· Feedback to management and the team regularly as progress is made. This important as it puts people fears to rest and minimizes potential sabotaging.
· Picking a tool vendor becomes a complex issue. You can go for a vendor that is in it for the overall process and for tools that integrate across the board, but the risk is some of the tools may not be as good as dedicated tools.
· You can go for the best of breed for each discipline, but they may each implement their own persistence strategy and integration is difficult / impossible.
· Decide to write your own tools. This can work, but takes time and effort. It also diverts you from the real project at hand.
· Cost could be a consideration factor here too.
· If the vendor’s tools do not integrate as you would like them to, look at their strategy and decide if you can live with the inefficiency until it is fixed.
· If you can't live with this then, try writing some scripts that will function as work a-rounds until the vendor gets it right.
· The three way conflict mentioned above should be understood up front and made aware to the individuals before the implementation begins. They will then understand why they are conflicting with each other later. This helps people realize they are not having personal difficulties with each other, but legitimate business issues which they have to confront and overcome.
Conclusion
We need to improve the integration between the SCM tools and all other development tools, but only when the tool vendors make the tools configurable enough to be able to impose varying processes on the same tool, and when the products from different vendors can interoperate seamlessly together. Alternatively one vendor makes a suite that works together well.
These are high aspirations, but we have come a long way in the last five to ten years, I'm sure we will get a lot better in the next five years. It is a cause to champion though, and it will deliver far better quality software at lot quicker in the future once we get it right.
References
1) Firesmith, Don - Open Process Framework Extensions Web -Tools List. as at May 2002.
2) Ambler, Scott - Comments about multiple vendors taken off the RUP forum. May 2002.
3) Edwards, Charles http://www.processwave.net/ May 2002.
