Software Development : How Good It Can Be

This article is dedicated to telling you how good developing software can be. It's a shopping list of sorts: I'm describing a whole lot of things working together, some of which you don't have in your shop. If you like an idea you see here, implement it. After your first implementation steps, you can return here to find yet another idea to implement, thus justifying an even larger budget for next year.

No single shop I've been to has implemented all of these things. On the other hand, I've seen all these things at different client sites. This fantastic voyage is broken up on a calendar basis, following my imaginary development shop through a release cycle. Each month (chapter, if you will) describes a little bit more of the process.

December

Customers are unreasonable. They want things, so they ask for them, while paying no heed to production schedules or project plans. This is true no matter how sophisticated or flexible your development team gets. In fact, the better a team becomes at delivering features, the less likely customers are to ask for things that are already on the project plan. It's a trust thing: when customers don't trust a vendor, they ask for obvious things just in case. When customers do trust a vendor they relax about the small stuff and their horizon becomes a lot wider.

Not surprisingly, these customers are asking for new features - the kinds of features that take a lot of work. To get any of them delivered will take massive changes to the existing software.

Being clever, the product management team has assembled a product steering committee with representatives from some of the most important customers. Their input is solicited before every major and minor release. The major releases focus on new function, while the minors help find the important shortcomings and most critical bugs.

Product management prioritizes the bugs, enhancements, and new features by involving sales, development, and the steering committee. The steering committee sorts the new functions and defects by their importance to existing customers. The sales department sorts the new functions and defects by their importance to new customers and competitive analysis. Development reports the estimated delivery cost of the most important features and fixes, and the likely sequence of implementation. The result is a single unified schedule of changes for use by all of the planning groups.

The product management team establishes their priorities and begins meeting with members of the project management office (PMO). The PMO allocates project managers to work on all projects throughout the organization. In fact, there is a pool of project managers that always work with the software group.

The PMO is using a set of project estimation tools to generate schedules, resource allocation tables, and other predictive data. The tools use data derived from industry as well as project data from prior projects in the organization to generate their projections. The more they are used, the more refined their predictions become. Product management emphasizes the need for a quick release to generate some marketing buzz and keep the customer base happy with the responsiveness of the organization.

The PMO announces that a three month mini-release can deliver the top 30% of bug fixes, but the new features would require at least a year. However, dropping some of the more esoteric features will allow a seven month release, with potentially two more releases at four month intervals.

Realistically, the eleven and fifteen month time frames are not as clear as they could be, due to changes taking place in the industry. The PMO feels it would be better to allocate all of the long term development to a single release, and re-evaluate the product plan and feature list at a later date.

Thus, just before the holiday break everyone knows that the new year will see three separate streams of development: a Bugfix release, an early feature release, and a catch-all late feature release.

January

Development has begun, albeit with a hangover. The build management team creates three parallel branches using the software CM (SCM) tool, one to support each scheduled release. This maps the high-level "strategic" branching strategy of the product management team down to the "tactical" branching strategy used by the development team internally. Automated nightly builds are activated for each branch, with flow-forward propagation of changes between the Bugfix branch and the early feature branch. The late feature branch will require large-scale changes, so propagation to that branch will occur on an intermittent, negotiated basis.

The Bugfix Release

The Bugfix release is drawing most of its changes from the defect tracking system, with a few low-impact enhancements thrown in. These low-hanging fruit are a scattering of severity 3 items: enhancing consistency across the user interface, or providing a shortcut in a dialog box.

Once the initial release gate has been passed, the PMO generates change requests (CRs) for each proposed change. The CRs derived from customer reported items were linked to the defect reports in the defect tracking system. Naturally, the defect reports are already linked to the call tracking system: in general, multiple customer call tickets may link to a single defect, with the status of all tickets being updated when the defect status changes.

The defect tracking system is a homegrown legacy system called PT, because it's always called PT. Developed decades ago, PT is linked to all kinds of bizarre places, including the product shipping system and accounts payable. Attempts to replace it have been unsuccessful to date, so instead of trying to cost-justify the replacement effort (almost impossible at this late stage), the engineering management team has chosen to integrate the defect tracking system with a more modern CR system.

The PT system tracks some rather strange fields. Some of these fields (usually five to ten of them) are concerned with precisely specifying the root cause of the defect, including the product lifecycle phases in which the defect was introduced and corrected. This data is used for phase containment metrics - a way of understanding the impact and cost of defects. As part of the root cause analysis, the defects are individually linked to the requirements in the requirements management tool and to the CR that introduced the defect, if it can be determined. Note that a change made to implement one requirement can cause a deviation in an unrelated requirement. Part of the root cause analysis is to look for unknown linkage between requirements.

Because it can take a varying amount of time to resolve the defects, the project manager has elected to assign earned value to defect bundles. The defects and minor enhancements are broken into nine groups of similar estimated work duration (estimation of work duration was performed in December during the project initiation phase). Each group is assigned 11% of earned value, corresponding to one week of scheduled development time (nine weeks of development, three weeks of software freeze).

The Early Feature Release

The early feature release is composed of a page or so of lower-priority customer requested changes (defects or low impact enhancements) plus a collection of planned new features. Because these features have been planned for some time, the requirements analysis and high level design work are already complete. The development and project management teams have voluntarily shortened the early stages of the development process.

When the initial release gate is passed, the change review board (CRB) marks the proposed requirements changes (already entered and with status proposed) as scheduled for implementation in the requirements management (RQM) tool.

The integration between the RQM tool and the CR system automatically generates CRs when the requirements changes move forward. CRs are generated for the software (it is automatically assumed to be out of conformance until shown otherwise) as well as for the development of validation test plans and test cases for the new requirements.

The newly-generated CRs, along with the change proposals from the RQM tool, are batched into a report that is provided to the PMO. The PMO manually links the CRs to individual tasks in the project tracking tool. The step is manual because the project plan is managed at a finer level of granularity than that of a single CR, although many CRs will correspond directly to tasks on the project plan. Note that some SCM systems include CR components that support finer-grained tracking of work: it may be possible to link directly from the CR system to the PM system.

Once the CRs have been created they are independently reviewed, to ensure the CR correctly reflects the needed change. Then coding can begin. The development team assigned to this release begins by ensuring that every defect has an automated test case to demonstrate it. These test cases will serve two purposes: they will be used to confirm that the defect has been resolved, and they will be added to the regression test suite to ensure that the defect stays resolved in the future. For this reason, test cases that are coded to demonstrate defects tend to grow, either into multiple test cases or an entire test plan that looks for similar defects across the board.

The Late Feature Release

The late feature release is composed of mostly new or unusual features, lacking the advanced design work that has been done for the early feature release. A lot of the late feature release is blue-sky requests from customers, sales, and marketing. Obviously, this is going to require a full treatment, including all the steps in the software development lifecycle.

The first step is to convert the desired features from marketing-speak into a concrete set of requirements. The systems engineering/architecture team begins to work knowing they have a very short window to produce their documents.

The first artifact is a release requirements document (RRD). This is a small document detailing the requirements for the release. This document is the translation document from the marketing and product management team's product vision document. Entries in the vision document will be linked to the RRD. The RRD will be linked to the system requirements Specification (SRS) and other technical documents as needed. The linkage from the RRD to the SRS, and the changes to the SRS to support the new RRD, will be supported using the RQM tool (which stores all these documents) and the integrated requirements change process. Like all other change requests, requirements changes are reviewed, approved, and implemented in a formal manner.

Using the requirements extensions to UML, the requirements work is linked directly to the analysis modeling in the CASE tool. Included in the UML requirement object attributes is the origination of the requirement. In this case, the origin is an entry in the RRD.

When individual subsystems are completed, the corresponding requirements are exported from the CASE tool and imported into the RQM tool as proposed changes. The exported data, of course, includes links back to the CASE tool.

Once all the requirement change lists have been imported to the RQM tool, requirements change requests (RCRs) are created for each entry in the RRD. The next step in the requirements change cycle is to ensure that the CRB has enough information to make a decision.

The RCRs are assigned back to the systems engineering team to provide estimates for the impact and effort required to implement each change. Since the proposed changes are still on the drawing board in the systems engineering group, the impact estimation is a simple matter.

The scheduling and required effort is another story, but the project management team already has some of this data in a tentative project plan that they used to provide the original schedule estimate. The old plan is used as a skeleton for the late feature release plan. The first iteration of development is kicked off with the objective of providing accurate scheduling data for some of the more esoteric plan entries. The remainder of the plan is massaged into a workable format, and then used as initial data for the project estimation tool.

The estimating tool has already been used to generate the schedule requirements for the release. Now, it will be used to generate more detailed estimates predicting earned value and defect flux over time.

February

With the success of a series of overnight builds, the configuration management office creates dedicated pages on the release dashboard for each of the three branches. The dashboard is an intranet application that combines data from several locations: the nightly builds, the automated regression test engine, the defect tracking system, the CR system, the project scheduling tool, the estimating tool, the RQM tool, and the test case management tool.

The reported data on the dashboard includes the following for each branch:

·         status of the nightly builds

·         current status of the automated regression tests

·         status of the allocated defects

·         status of allocated CRs

·         current earned value

·         predicted earned value

·         any tasks deviating from the predicted schedule

·         present defect flux

·         defect flux charted against estimate

·         size of delivered changes

·         size charted against estimate

·         complete set of validation test plans

·         wait/try/pass/fail status of all validation plans

n Fridays, the PMO reviews the overall project status with engineering management. The earned value is monitored, and the status of all change requests associated with the current week is reviewed. Additionally, note is made of any additional CRs that may have been resolved early. Finally, the CRs to be resolved in the next week are reviewed and assignments finalized.

The same meeting (usually referred to as the “weekly beating”) sees the QA organization reporting on the status of the testing and validation elements. Normally, this sees QA confirming the successful closure of issues from the current or previous week. However, when more testing is required, or when a defect is found in the testing process itself, the meeting includes estimates for the work required to resolve the testing problems, as well as any costs associated with revisiting CRs that were already believed to be resolved.

The real value of these meetings is that they allow the technical team leads, QA, the PMO, and engineering management to get on the same page. All the participants can discuss the ramifications of any issues, and all understand the reasons behind the assignment of priorities.

The Bugfix Release

Development on the Bugfix release is now in full swing. The project schedule is in place, the software is rebuilding each night, and the automated regression tests are running.

The PMO has scheduled a series of propagation meetings. Meetings are held each Monday and Thursday to coordinate the propagation of changes from the Bugfix release forward to the early feature release. Representatives of both branches, along with the PMO, review all of the delivered changes and the estimated impact.

When the changes are incompatible, or when they are not morally correct (as when a feature is disabled to eliminate a defect) the propagation is prevented and a new item is added to the Early Feature release schedule. The propagation of changes has been configured to be automatic from the Bugfix to the early feature release. Preventing propagation is a manual step that must be coordinated by the PMO and confirmed by the CMO.

As the CRs are being worked, most of them are causing updates in the PT system as well, since the integration between defect tracking and CRs is bi-directional. In this case, the status of the defects changes from acknowledged to work in progress when the CR is assigned. All customer-reported defects (as opposed to defects discovered internally) have an associated trouble ticket from the customer call tracking system. These call tickets will be updated as well, reflecting the status change of the defect entry.

In addition to automatically updating the PT system, the CR system will update the project-tracking tool, crediting partial completion when code is delivered. The test case management tool will credit full completion when the associated test cases succeed.

Some of the development team working on the Bugfix release will be new to the code base. A short, well-contained release is a great training ground for developers. Accordingly, interns, fresh-outs, new hires and contractors just coming up to speed will make up as much as seventy percent of the development effort on this branch.

There are four important things for the new developers to absorb while working on this release. First and foremost is the culture of the organization: coding standards, policies, and styles of work. Second is familiarization with the procedures to be followed for working: meetings, code reviews, check-in and checkout procedures, the mechanics of the various systems. Third is familiarization with the codebase itself. Finally, the new developers will be involved in the detail work of the process, providing their mentors with an opportunity to explain the rationale behind some of the procedures. Most standards documents have a separate rationale document associated with them, but corporate culture won't.

An excellent example is the root cause analysis process. The obvious starting point for any defect is a review of the code: is there a coding problem that causes the defect? If so, then why didn't an earlier code review catch this problem? If not, then from where does the defect spring? A combination of factors? A design defect? A requirements problem? Walking through the root cause analysis can introduce a new staff member to a lot of the development artifacts. Following the root cause analysis with the rework needed to correct illustrates most of the software development procedures.

The Early Feature Release

The development ramp-up has gone smoothly for the early feature release. This isn't surprising since so many people have been thinking for so long about how to implement these features. In order to solve the one outstanding technical problem that the development staff had not solved in advance, project management has elected to contract with an outside development shop.

The outside contractor selection process focuses on the ability to deliver the needed technology, compatibility with the organization's processes, and the stability to ensure they chosen contractor will be around for maintenance tasks for the lifetime of the components. Much of this will be handled at the management level, using methodologies not normally associated with the software development arena (such as schmoozing and kickbacks).

As the paperwork is being signed, the CMO and PMO spring into action. The CMO has to make sure that some paperwork of its own is taken care of, as well as integrating the contractor into the overall project. The PMO now has another set of project tasks to monitor, some additional resources to allocate, and an interface headache trying to stay on top of the contractor's progress.

The first step for the CMO will be to decide the level at which the contractor is integrated into the ongoing development operations. If the contractor were more of a body shop, then the technical experts would be working on-site. This simplifies things immensely - the experts act like other contractors or employees, everything is contained in one location, all is well.

In this case, the other firm is doing the work off-site, managing the work internally, and generally acting like a good contract development shop. The CMO has to update the software CM plan for the project to specify the interface to the contractor, as well as detailing the vendor CM plan, list of deliverables, and delivery schedule. Of course the contractor tries the old "Don't worry: we'll deliver what is required on the last day" dodge, but CMO is having none of it. Instead, CMO and PMO combine to define a set of milestones, each corresponding to a deliverable baseline from the contractor.

As a final measure, CMO arranges access for the contractor to the internal CR system. This will require some extra work for them, since the contractor isn't using the same version control and build software. However, it's worth doing so that the PMO and upper management can stay on top of project status.

The Late Feature Release

Things aren't going well for the late feature release. In fact, the demands made by the two near term branches have consumed almost all of the resources needed by this branch. Knowing a problem when they hear one, the PMO begins negotiating with upper management for a slightly broader resource base.

The result is the go-ahead for both the early and late feature releases to use offshore resources from the company's development and testing affiliate in India. The company has been working with these developers for two years now, so the wide area network is already established, the development environment has been exported, and the developers are easy to roll on to the projects.

The CMO is familiar with the needs of offshore contributors and enables repository replication between the Bangalore development center and the local office. The first few nightly builds are made without any changes, since the PMO is still trying to reallocate the work breakdown to utilize the new resources. In particular, they ensure that all offshore work is tested and validated locally, while assigning offshore test resources to work completed locally.

Near the end of the month, the assessment iterations for scheduling some of the features is wrapped up, with enough data to enable the project estimating tool to provide a finer estimate of the whole project. Some of the requirements documents will have to change, with corresponding changes needed to all the downstream elements: test plans, validation plans, UML design documents, etc.

March Bugfix Release

The first week of the month marks the final round of pure development. After the first week ends, the Bugfix release enters soft freeze.  This is a period when active work may be concluded, but nothing new may start. This is the beginning of the final push to get the release out the door. The PMO has allocated a one week soft freeze because the individual items in the project plan don't take much time to complete.

At the end of the soft freeze comes hard freeze.  No changes at all are permitted to be delivered, except as individually approved by the QA group in response to severity 1 or severity 2 defects. This period last two weeks and allows the QA group to run all their testing one more time against a stable target.

The developers are reassigned to the other branches as their tasks are completed, except the two or three luckless individuals always stuck dealing with the last severity 2 issues before release.

The CMO attempts to get ahead of the game a little bit by doing some of their physical configuration audit early. In this case, they are running all of the necessary reports except for the release content. This audit reports list defects and requirements changes, the associated software change requests, and the corresponding test plans and test cases. The purpose here is to ensure that all of the scheduled changes from the project plan do in fact appear in the product.

The PMO, satisfied by daily status reports from the QA team, issues a warning to the software publishing team to expect a new release on schedule. Once the QA group has passed the final release candidate, the pre-release process kicks off.

The QA group and the CRB update the resolution data of all the defects resolved by this release. The CMO updates all of the CRs for this release to reflect their released status.

The CMO performs the last step in their physical configuration audit, confirming that the changes previously reported are completely represented in the delivered product, and demonstrating that no changes are in the release that were not approved by the CRB and included in the release plan.

Additionally, the CMO performs its functional configuration audit - reviewing the requirements, the test plans, and the test case results. All requirements have associated test plans or specific test cases identified that verify the requirement is satisfied. The RQM tool, which maintains links between the requirements and the test documents, is used to generate a traceability matrix that is merged with the test case results to confirm that all the relevant requirements have been verified, or that deviations have been written up (in the form of defect reports) for any defects. In the unlikely event of a severity 2 or worse defect, no release can occur without slipping until the defect is resolved, or without obtaining a waiver. Either of these cases requires explicit sign-off from engineering management.

Finally, the CMO generates the release content letter, sometimes called a version description document, which is a document skeleton automatically generated using the reporting capability of the CR system. This document describes all of the defects and enhancements present in the new release, listing them by their external reference numbers. The documentation team develops the boilerplate sections of this report: the corporate contact information, website address, and troubleshooting tips. Product management provides the custom introductory material and performs the final document review.

Once all of the pre-release steps have been performed, the final release image is generated. This image is archived by the CMO and forwarded to the software publishing team for distribution.

The Early Feature Release

With the looming release of the Bugfix release, the focus is placed on getting the release ready to go out the door. The automatic propagation of changes from the Bugfix to the early feature release is disabled. One last round of propagation meetings is performed before the Bugfix hard freeze. No further changes will be propagated until after the Bugfix release goes out the door.

The CMO and the build management team are kept busy setting up synchronization of software repositories between the headquarters and the external contractor hired in February. The contractor is using an open source software repository, so establishing bi-directional synchronization every night is a relatively straightforward process. Now that the changes made by the contractor are being built and the approved updates coming from the nightly build are being exchanged between the sites, everyone can be much more confident in the ability of the contractor to deliver.

Unlike the Bugfix release, the early feature release will definitely need updates to the installation program. The new features being added will depend on the installation hardware. The installation development group already has some of the necessary work completed, since the target operating environment (TOE) document has been modified, including required features and revision levels. The remaining work (installation script code to update the target system for the new features) is handled as regular development in the architecture-specific installation components. Most of the routine scripting is handled automatically, since the TOE data is processed mechanically to extract minimum required characteristics of the target system and generate the installation script.

When the installation package updates become available, the development team will receive most of them automatically. The development network is using a systems management package that tracks the latest versions of software approved for the standard operating environment (SOE). When developers register as members of the team for the early feature release, the software automatically keeps their system configuration up to date. This has the added benefit of ensuring that the last successful nightly build (the last good build, or LGB) and the last stable build (the known good build, or KGB) are automatically deployed to each developer's desktop.

One of the major hurdles when rapidly deploying updates to a complex application is updating the database schema. Fortunately, the architecture group has designed in a live update capability and provided versioning data for the database records. This enables the developers to perform development and testing while database upgrades are in progress, with relatively little slowdown. Major schema changes, obviously, may prevent this from functioning correctly. The ability to roll back minor upgrades means that a copy of a testing database can be successively upgraded to a new nightly or stable build, tested for a few days, and then rolled back to the original data when a new test run is about to commence.

The Late Feature Release

With the availability of developers from the offshore development center, the PMO has reassigned local developers to the late feature release. Specifically, some of the more blue sky features being considered for this release are going to require a lot of research and the teams to work those features need to be firmed up early. In the two months since the original project schedule was set up, some of those developers have become deeply involved in the early feature release: transitioning them will be a challenge.

The simplest parts are the automated systems. CRs assigned to these developers are bulk-transitioned backwards to pending assignment status, to enable the development managers to allocate them anew. Some of these changes may very well be allocated overseas. Since the overseas development center uses the same CR database, this is simple. In order to make the hand-off as painless as possible, interim design and code reviews are scheduled for each CR. As the reviews are completed, the CRs are assigned to new developers who are now up to speed on the status of each item. Notes from the reviews are entered into the CR records as well, to minimize the questions that may come from some of the overwhelmed developers taking over all these changes.

While development is reorganizing to use its new resources, the testing groups are coming up for air after the rush to get the Bugfix release out the door. The big issue for testing will be the development of test plans for the Late Feature release (the early feature release test plans have been ready for some time, as they knew the work was coming, just not when). The requirements for the late feature release have finally settled down, and the architecture changes are pretty well understood (if not totally documented) so the overall test planning process can get pretty far before needing to sync up with the engineering teams again.

Once the test plans are laid out, they are linked in the test case management (TCM) tool to the corresponding requirements, architecture, and design elements in the RQM and CASE tools. Defects reported by the testing group will be automatically logged against the corresponding design elements. As test plans are finalized, some of the test engineers begin work on updating the testing harness; some of the more esoteric features require that the testing process change a little. Once all the test frame changes are themselves tested, the test frame undergoes an internal release and is made available to all testers. The development of the test cases themselves can begin.

The build manager for the late feature release has come to the realization that some of the changes are going to require massive changes to the SOE and TOE. The new web services framework, for example, will require that all the developers working on the release have a new operating system version on their desktop, as well as new development tools. Similarly, the new deployment and automatic update features will require a new OS and new third party software on the target installation platform.

Many of these changes are relatively simple to implement: the new operating systems can still be updated using the systems management tools, and the new operating systems themselves can be licensed and acquired easily. But the need for developers to be able to work in two or more radically different environments when supporting both releases may pose some problems.

Some of these problems can be ameliorated by dual-booting machines for various release configurations. Other solutions involve using KVM switches to connect several machines to a single developer's desktop, using soft switching technology to connect a single keyboard/mouse to multiple machines, using removable hard drives to archive good configurations and development support environments for very old releases, using environment replication software (ghost images) to archive entire configurations and using machine emulation software to run several configurations simultaneously on the same desktop.

April Bugfix Release

In April, the sun is shining, the birds are singing, and the Bugfix release is finally out the door. All that remains is for the last post-release steps to be taken care of. Since this is purely a Bugfix release, there are no feature updates required. This means that no training material, sales brochures, etc. need to be changed.

The software production group has already archived the golden disk containing the official released image. The distribution centers have received the update, confirmed it, and superseded the old release in production. The RCL has been burned onto the release image as well as being printed and inserted into all new packages. The sustaining engineering team has written an upgrade/update pamphlet for existing customers wanting to deploy the new software. In point of fact, the upgrade/update letter looks suspiciously like the last one: one team's consistency of style is another team's plagiarism.

The CMO will finalize its audits and hand the results off to the PMO. The PMO will organize a release postmortem to look for any problems with the process that could have been handled better. Once the audits are completed, the traceability from defect reports and requirements to release code is verified. The requirements and software CRs, and the open defect reports, can be closed out. Those defect reports associated with customer trouble tickets will be updated and the customer trouble tickets will be closed. The customer support organization will contact the individual customers to notify them that their issues are resolved and to make arrangements to deliver the software updates.

The Early Feature Release

With the wrap-up of the Bugfix release, the testing team and a few developers have become available for assignment. The PMO and the test organization will move them around as needed. Since they are coming in to an ongoing project in the middle, mostly they will receive "new work" instead of trying to take over an existing task.

Changes made while change propagation was frozen in March will be examined by the PMO and the development team. These meetings will allocate the propagation (or reengineering) of the changes for the Early Feature release among the development teams, and update the development schedules as a result.

The first baseline for the local contract-coding shop comes and the contractor is already behind schedule. This surprises just about no-one, since the CMO reported in their vendor CM audit that the contractor looked to be at CMM level one, with some hints that they could see the path to level two but were not on it. The PMO drags one of the senior developers to a meeting with representatives from the contractor. After pointing out the clause in the contract that provides for significant monetary penalties for non-delivery, the PMO suggests that the senior developer provide a schedule and estimate for the deliverables assuming that the developer were using internal technology. This can be merged with the contractor's own estimates (which turn out, sadly, to be a series of guesses for entire systems: "two months to get that done, and another month for this" with no documentation, work breakdown, or task list).

When the PMO and engineering management realize what is going on, a set of emergency meetings is scheduled and the PMO assigns one of their own people to monitor the internal situation at the contractor. Weekly status meetings begin. According to one project manager, "They're a month into the contract, and already two months late. But we'll get ‘em fixed, no sweat." The synchronization of the contractor's CM system with the internal build system means that development can monitor all the contractor deliverables, and keep the PMO informed when things go amiss. This costs resources, but the schedule remains roughly intact.

In an attempt to keep relations friendly with the contractor, the CMO sets up a build system at the contractor's location. This server is updated with the latest build of the project, including the contractor's part, on a nightly basis. The CMO and development management hope that having the entire system available will enable the contractor to test and debug a little quicker, thereby getting back some of the lost time. There are some intellectual property issues here, so it takes quite a while to get the whole thing approved.

The build team, meanwhile, has heard from the developers that some of the features are "just about done". Staying ahead of the curve, a good nightly build is copied out and designated as the first numbered release candidate. This gives the build and test teams a chance to practice the handoff, and provides a straw man for the audit and trace functions. All of the automated content verification, delivery, and regression scripts can be run (to failure, mostly) on this first candidate. We knew the scripts would work, of course: "Trust, but verify."

The Late Feature Release

The updates to the requirements and architecture are complete, the high-level design documents for the new features have been substantially wrapped up, and the test plans (not test cases) for the new features are completed. Since this is a logical documentation milestone, the CMO has scheduled an allocated baseline for this month. All documents will be reviewed and archived, and a baseline audit will be conducted to check the traceability and verifiability of the design artifacts.

With the conclusion of the Bugfix release the PMO elects to propagate all of the Bugfix changes, and whatever Early Feature release changes have made it into testing, into the Late Feature code base. This bulk merge of changes risks interrupting the development teams, but the extent and impact of the changes is already pretty well understood because the Early Feature release did the same sort of analysis for their own merge earlier in the month.

The changes to the test framework have been finished. After some "test testing" (in this case using the new framework to test an old release) the testing organization performs an internal release to make the new framework available to the developers and other testing teams. Documentation is available, of course, but because of the changes the API document has to be updated. The testing group's intranet web site provides all these documents once they are reviewed and approved.

The highest impact document is the required tests database. This database (mapping the language, purpose, and design of modules to the testing elements that are required or recommended to be present) must be updated for the new technologies being introduced with the Late Feature release. This involves the local experts (including contractors) working with the testing and architecture groups to determine the right amount of mandatory testing.

Aftershocks from the changes to the SOE and TOE are still being felt throughout the development organization. Some of the development and (especially) testing hardware is incompatible with the updated software requirements. Engineering management is working with the Licensing group to determine how many customers will be impacted by the incompatibility, and how severe the impact will be. It is not anticipated that the impact will be so high that the upgrade must be abandoned. But if the impact were sufficiently high, a non-upgrade release might be considered as a support path of last resort for legacy customers.

Another problem has arisen for the CMO and build management team: the existing metrics tools don't support the new technology used in this release. Part of the problem is legacy data - the project estimating tool, for example, needs legacy data to use when generating estimates. The other part of the problem is technical; the code-counting and analysis tool doesn't understand the new languages.

Both of these situations can be remedied by applying to the respective vendors. In the case of the code analysis tool, it is likely that an update is already on the way. If not, some customer-funded engineering may be required to get the desired results. Alternately, a different tool may be used, entirely or for just the new parts of the project, but integrating a new tool is often more expensive than paying to have the existing tool brought up to date.

The project estimation tool will take more work. Since the technology is totally new, there is no data available. Engineering management contracts with the tool vendor to provide internal scheduling and metric data to the vendor in exchange for access to the updated models when they become available. This is relatively common, but it does tend to skew the vendors' models early on, since the data are from shops that rigorously collect data: usually better organized than most.

May the Early Feature Release

The problems with the local contractor have begun to have real effects on the project schedule. The PMO decides to handle this by moving one late-scheduled feature from the early feature release to the late feature release. The development group, with this pressure removed, has the resources to spend to manage the contractor more closely.

With the creation of the first numbered release candidate, the testing team has begun to execute their test plans. Some test cases remain to be written, of course, but the regression suite and the performance evaluation tests from the prior release can still run, as well as certain basic unit test cases for the new features. As the month progresses, all of the existing functionality tests will be run, and as many of the new feature tests as possible.

The CMO and the build team have an automatically maintained impact analysis database that maps source code and third party modules to affected deliverable products. This database is used to automatically select regression tests to run against the nightly build, based on the changes delivered to the build.

In order to schedule tests effectively, the testing organization polls the developers to predict the modules impacted by changes that are assigned or pending but which have not yet been completed. This data, combined with the project schedule, gives the testing group a roadmap of which tests they will be expected to run in the future. This, in turn, is used to build the overall test plan - running the right tests in the right order streamlines the entire process.

The Late Feature Release

Both development and the build team are having some problems caused by the SOE and TOE changes. The offshore development group in India and local contractor both have to keep up with the changes manually; he systems management software used internally cannot update either site. This is not a problem of record keeping, since the CMO is tracking the changes to both documents, so much as it is a problem of making sure the changes get applied in a timely fashion. The India site, especially, is cursed by the vast time zone difference as well as not having access to the same software market as the local office. Frequently, the build management team has to purchase software locally and then FedEx it overseas.

The build team is working around the issue of metrics for the new language code by cobbling together a set of work-around scripts to provide rudimentary values for the database. The new language looks enough like C that with some basic text manipulation the existing tools can be run. The team is less sanguine about the complexity metrics, but whatever data can be had will suffice until the tool vendor provides an update.

June, the Early Feature Release

With two months to go until ship date, the early feature release is about to go into soft freeze. With the exception of changes coming from the local contractor, all the feature work that is going to be delivered has already started. The PMO requests a process deviation to allow the local contractor to catch up. The development organization is working smoothly, and expects to have all of the features implemented within the next four to five weeks.

The testing team is working at full capacity. The organic testing groups who are directly attached to development teams - have worked with the developers to code the unit and module test cases. Once those tests are ready to run, and the developers can run them automatically as part of their desktop build process. The testers, released from this work, help out the formal testing teams and help with support activities like code reviews.

The PMO monitors the results of the testing process, including the unit testing, via the project dashboard. The numbers of defects found and fixed is one of the metrics that the project estimating tool tracks (this is a good indicator of where the project is in its development/test cycle).

The Late Feature Release

The update to the code metric software is delivered in very rough form. A significant part of the functionality is missing for the new language, but the basic metrics (size, organization, and complexity) are available. The CMO begins work to update the metrics data in the database with the correct values reported by the tool. Having the data stored in the repository simplifies the generation of many of the metrics reports.

The product management team contacts the PMO with the news that: the customers are asking for new features (the kinds of features that take a lot of work). To get any of them delivered will take massive changes to the existing software.

July

July sees the early feature release go through hard freeze and then ship. The late feature release is developing smoothly, although the shipment of the early release is a good time to perform a propagation sweep, thus getting all the changes moved forward in one fell swoop. Since this is an iterative process, and we've now seen two iterations, we can safely fade away before the whole thing starts up once more:  déjà vu, all over again.