A Motivating Story
A new developer, Jim, is joining your team today. You have been extremely busy, and you want to get Jim productive as soon as possible. The first thing you need to do is to help Jim get a development workspace set up. You’ve been meaning to automate the process for a while, but since setting up a workspace is something that you do only once in a while, you’ve been putting it off.
You point Jim to a Wiki page that has instructions for setting up a developer workspace environment. Jim opens looks at the wiki page and follows the steps for downloading and installing an assortment of tools, including the compiler, version control client, etc. Hours later, after installing the tools, if Jim remembers that the purpose of this activity was to develop software, Jim gets to the steps that tell him what command to execute to get the source code. He checks out the code and tries to build it. The build fails.
Jim re-checks the list on the wiki and realizes that he made a mistake on step 3. He fixes the problem and tries to build again. It compiles! But the tests fail part way through. After checking to see if he made a mistake, he grabs one of his colleagues who walks over with a USB Drive and says, “You need a copy of this configuration file to run tests on a development box.” Right before Jim leaves for the day, he can build the application. Maybe tomorrow he will be able to run the application on his machine and get to work.
The Repository Pattern
Having a well-defined process for setting up a development environment, the workspace, is the point of the repository pattern, which advises you to have a single point of access, or repository, for your code and related artifacts. Make creating a developer workspace as simple and transparent as possible.
The repository advises you to create consistent workspaces for development, integration, etc. This enables:
- Reproducibility, by having the recipe for reproducing an environment under revision control, and reducing the “works for me” phenomenon.
- Productivity, by making it simple for developers to get started or create new workspaces for different projects.
- Efficiency, by automating the process, and also making it less error prone.
- Maintainability, by allowing you to create multiple workspaces for release lines.
In principle, you can attain the goal of reproducibility by having a defined manual process. Having a documented process made it possible for Jim to get started rather quickly compared to an ad-hoc one. Having a documented process puts Jim’s organization ahead of some.
The risks with having only documented processes are that manual processes are slow to execute, error prone, and difficult to maintain. And once a document shows signs of being out of date people will start ignoring it, thus leading to discrepancies between workspaces, negating the value of having documentation. The value of an automated workspace creation process is best expressed by the Agile Manifesto value working software over comprehensive documentation. The simplest way to ensure that a workspace is consistent is to make it very easy to set one up.
An automated process for doing this sort of set up has all the benefits of the documentation only process, and also has these benefits:
- It will be faster by eliminating idle time: Most set up processes involve starting a process (a copy or a checkout), and then after the step is finished, doing the next one. During the set up the developer is either sitting idle, waiting for a process to complete, which is wasted time, or doing something else, which means that there will be down time unless the developer is looking at the screen when the process stops.
- It is repeatable: To change the set up, you change the set up process. By minimizing manual steps, you minimize the chance for error.
- It can be validated automatically: An automated set up process makes it possible to have low cost tests run periodically, so that you can detect errors that may creep into the set up process automatically.
To simplify workspace set up you will need to use some sort of tooling. This leads to a bootstrapping problem that can be addressed.
Bootstrapping
A good workspace set up process has very few, if any, manual steps. In practice, it’s difficult to automate every step of the process, so you should strive to have as few manual steps as possible.
To begin, you will need some tool in place to bootstrap the creation process. For example, if you start with a checkout from the source code repository, you will need a version control client. If you choose to start things off with a script, you need to be sure that the right version of the scripting tool is available on the target machine,
It is also important to make as few assumptions as possible about the environment, such as assuming that certain tools live in a certain place. This makes the build process fragile and not robust to change.
Rather than making a simple workspace creation process seem like an unreachable goal, you can get most of the benefits of automation by ensuring:
- The number of prerequisites is small. Ideally only one or two tools which are easy to install.
- The documentation for the setup changes rarely; any changes that affect existing workspaces should be reflected in the tool or build process.
- The workspace setup process verifies that the tools all have the correct versions, and if not, provide the user with some information about how to upgrade.
With these principles in mind, you can create a Getting Started page for your project that looks something like this:
- Install some basic tools that need to be installed once.
- Run a command to get a bootstrap build script.
- Update the source code
- The build should configure any user customizable configuration files automatically.
In the interest of getting a process working, we’re allowing that some tools can be installed manually it’s important that the build step verify that these tools are the correct versions. For example, your build script can have a “check-prerequisites” step that verifies that the build tool and complier are at an acceptable version, and fail with a message indicating that the developer needs to upgrade. While this is not as good as automatically upgrading, it is acceptable if the upgrades are infrequent, and developers are warned in advance. Treat core tool upgrades like changes to a published API.
We will work through an example using the java build tool maven. While this may not be something that suits your environment it will give you a sense of what is possible.
Starting a New Project
As it happens there is an open source project that already demonstrates the tooling we need to make it simple to create a new project from nothing: the Apache Wicket project. Wicket is a framework for building java web applications. Wicket claims to make building web applications simple and enjoyable. Its quick start page exemplifies this ideal. Figure 1 shows the dialog from the Wicket project QuickStart page. By completing the form, the web page generates a command line to create a template project. By copying the command line and pasting it into a shell window you can have project with everything you need to create a simple Wicket application.
The way that this works is that they web page does the details work of helping you build the maven command line that downloads the archetype project from a repository. The archetype is basically a template project. This process assumes that you have installed:
- A web browser
- Maven
- A Java compiler
This example relies on the Maven archetype mechanism to do most of the work, and the code to generate the command line simply saves you from copying and pasting.
And the archetype is just a template project with the default files in the default directory structure. But it is exactly this kind of automation that makes starting a new project simple and foolproof.
With the help of tooling that Maven and the Wicket Quick start provides we can now create a wicket project.
The good:
- The quick start page minimized the risk of errors of mistyping the maven commands.
- The template project structure saved us a lot of typing.
Not as good:
- The process relies on us having maven installed, so it’s not foolproof
- The process only starts a new project. The more common scenario is to jump in on an existing project.
The Wicket Quick Start example showed us that it doesn’t have to be difficult to provide directions on how to start a project. You need only tell a developer that they need to go to the QuickStart page and type in the name of their components. The web page can enforce naming conventions. Or you can select the components from a menu.
Ongoing Development
To extend this approach for an existing source tree, you could have the archetype include only the files that specify the source repository. You could then execute mvn scm:update to update your source tree. If you are doing work on a branch, you can incorporate the release labels in the drop down.
Deployment
One more thing that the Wicket set up gives you is the ability to run your application using the jetty application server. What you have is the ability to build and test consistently in your development workspace.
Recap
To review, the essentials of the repository pattern are:
- A single, somewhat static set of documentation in a well know place that describes the bootstrap process (install Maven, for example)
- A step to identify what files to retrieve from version management
- A step to update to the latest source version and obtain all the dependencies
- A build and configuration process that allows you to run all of the pre-commit tests in your workspace
If the getting started web page seems intimidating, simply add the maven command on the Getting Started page, and have the developer edit the command by hand. Not as foolproof as the generation approach, but should a developer mistype a label or the like, the process will fail quickly.
One you have this basic process in place you have enabled your team to create workspace easily. This will allow you to:
- Build more reliably, as it is easier to create workspace from scratch, and thus eliminate drift between developer workspace and the integration workspace.
- Do integration builds consistently because all of your dependencies are defined.
- Perform support tasks, since creating a workspace for a Release Branch is also simple.
Once you have the basics in place, you can work towards automating more steps. It’s possible to automate tool selection for example.
Getting Started Again
After Jim had been on the team a few weeks he realized that he had wasted a lot of time getting set up. He knew that the team would be growing so he worked to improve the setup process.
When Fred started, Jim pointed him to the getting started page:
- Install subversion
- Checkout out the source using the command "<>"
- CD to the project directory and type "build.sh"
The build shell script executes the correct version of maven that is installed in the project tools directory, and Fred is ready to go after an hour or so.
Conclusions
The Wicket Project Quick Start page provides one example for setting up a project and provides an inspiration for how you set up a project in a more general case.
References
[1] [SCM] SCM Patterns Book
[2] http://www.agilemanifesto.org/
