Performing an SCM product evaluation is important. This allows us to evaluate whether the selected product meets the needs of the application being developed. Typically, there is not as much time spent evaluating SCM products as needed, even though an SCM product will be one of the more highly used tools in the application lifecycle. It is with this in mind that it is important to understand the levels of an SCM product evaluation and the level of risk each assumes.
Communicating risk can help you raise the priority given to this task. It can also lead to the appropriate level of SCM evaluation that is aligned with the level of risk an application team is willing to take. There are several levels of a SCM product evaluation that may be performed, each with a different level of associated risk. They include the research evaluation, the demonstration evaluation and the in-house/full evaluation.
Research Evaluation
This constitutes an evaluation based on reviewing existing documents (e.g., articles, research, etc) that discuss and evaluate SCM products published in technology trade magazines, journals, and evaluation documents such as (and not limited too) Software Development (SD) Times, Crosstalk, and Ovum. This may also include reviewing input from folks with actual working experience from SCM discussion forums like CM Crossroads. This will give you an insight into what you might look for in an SCM product. In specific, at this level of evaluation, ensure the SCM product meets the basic entrance criteria for the environment. Entrance criteria will include and are not limited to development platform(s) you will be working in, basic versioning capabilities for the type of code you will be working with (source, documents, binaries, etc), and cost constraint. After a review of these documents, a decision for selecting the product can be made. If you perform this level of SCM product evaluation, you are placing yourself in the highest risk of incorrectly selecting the best SCM Product to fit your needs since little time is used and no demonstration and hands-on experience is occurring.
Demonstration Evaluation
This level of evaluation would include reviewing the output of the research evaluation that includes focusing on the top two or three SCM products. Then it includes defining clear SCM product requirements for the application’s needs. This would involve forming an SCM product evaluation team, gathering SCM product requirements, weighting the requirements by importance, getting a demonstration of each requirement from SCM product vendors (and asking how their product meet the requirements), and scoring each requirement. After the demos and scoring of the products, a selection may be made. Performing this level of SCM product evaluation, reduces the risk of incorrectly selecting the best tool to fit your needs since you have a good idea of your SCM product requirements and you have seen demonstrations of SCM vendor products, and hopefully determined at a high-level which SCM product best meets your needs.
In-House/Full Evaluation
This level of evaluation would include the research and demonstration evaluation. The top one or two products (or as many as desired) from the demonstration evaluation would be brought in-house for a 3-6 months to exercise and test the SCM vendor product(s) against the requirements specified and in a project scenario. The input from request for proposal (RfP) from each SCM vendor can be reviewed as part of the evaluation. The SCM RfP should ask about the how the vendor delivers releases and patches, how they solicit customer requirements, and how they price their product (amongst other things). Beyond this, you would want to establish an SCM pilot environment and select an application in which you would place under SCM version control. This environment should be similar to the working application environment (e.g., same platform, sample application code, development tools, etc). Tasks will include installing the SCM product, identifying an application (or pieces of code), and importing the code into the SCM product repository. This may entail establishing an SCM version control, build, and packaging process. Then exercise and test the SCM product(s) to determine if it meets the SCM requirements that were defined. This type of evaluation has the lowest level of risk associated with the selection of the SCM product.
Summary
The level of SCM product evaluation that is chosen and conducted may vary according to the available schedule, effort, resources, and level of technical expertise within the company. If an organization only wants to commit the minimal amount of time to evaluate an SCM product (aka, research evaluation), then the application owner, project manager, and/or SCM professional should communicate the risk that is associated with this level of SCM product evaluation. This is why it is important to understand the risks associated with each level of SCM product evaluation. Overall, an in-house/full evaluation is ideal since it minimizes the most risk and validates that the SCM product can actually met the requirements. The minimal evaluation level, IMHO, is the demonstration evaluation since the SCM product will be one of the more highly used products in the application lifecycle. It is important to at least know your SCM product requirements and view the SCM vendor products to compare against your requirements.
