Open source development projects typically include a testing phase, just like their commercial counterparts. But how do open source projects manage the testing process? What tools and resources do these projects use? This article discusses the ways that open source projects accomplish testing goals, and how open source testing compares to testing in a commercial project environment.
Open source development projects typically include a testing phase, just xxlike their commercial counterparts. But how do open source projects manage the testing process? What tools and resources do these projects use? This article discusses the ways that open source projects accomplish testing goals, and how open source testing compares to testing in a commercial project environment.
Imagine that you've become the test lead for Project X; however, your testing team has no official members. You have to rely on the contributions of volunteers from around the globe to accomplish testing tasks. You'll have almost no face-to-face contact with your testers. You have to provide all of your volunteer testers with access to Project X's software. You have to create a bug tracking system that can scale to 1,000+ users. You and your team have to accomplish testing tasks and meet the goals of the testing effort.
What would you do? How would you manage testing for Project X?
Open source projects have to tackle many of these issues when managing the testing process. There are challenges unique to the open source arena, so innovative, resourceful tools and methodologies must be called upon to accomplish testing tasks.
What Is Open Source Software?
There are many opinions regarding the definition of "open source" software. In fact, a large sector of what I'm calling the open source community rejects the open source label and prefers the term "free software" (as in free speech). This article is not a forum for such a debate. For the purposes of this discussion, open source software is considered any software whose source is freely available, with no limitations on source modification and redistribution. There are exceptions to this rule, and many different software licenses are considered open source (e.g., the GNU General Public License, the BSD license, and the Mozilla Public License), but such a broad definition will suffice for this discussion.
The GNU/Linux operating system is perhaps one of the most recognized examples of open source software. For an excellent introduction to the concepts behind open source development, read Eric S. Raymond's essay collection The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary (published by O'Reilly).
For open source projects, development occurs in the global community and proceeds in an evolutionary manner. Many people use the software while it's still in development. Users file bug reports. Bugs are fixed. Community debate and discussion steers project development. The open source development model resembles a community bazaar. As Raymond writes, "The Linux community seemed to resemble a great babbling bazaar of differing agendas and approaches…out of which a coherent and stable system could seemingly emerge only by a succession of miracles."
What Challenges Do Open Source Projects Face when Testing?
Open source projects introduce unique challenges when testing a software application, especially when compared to commercial development practices. Some of the major challenges are as follows:
No full-time testers. When working on a commercial project, you expect to have other test team members to work with. If you're a test lead, you expect to have a test team whose efforts you direct. However, in the open source community there are no full-time, paid testing teams. Volunteers, typically working in their spare time, contribute most of an open source project's testing work. Open source endeavors must constantly promote awareness and recruit volunteers from the community. Testing productivity is directly proportional to the number of active volunteer testers.
Global team coordination and communication challenges. Volunteer testing teams contain members scattered across the globe. Testing team efforts must be coordinated on a global scale. Questions and issues can't be resolved in the weekly staff meeting. Typically, there are no meetings at all. Electronic communication becomes the lifeblood of testing teams. With little face-to-face contact, testing team leaders must possess exceptional team leadership skills.
This team environment also creates unique management challenges. Team leaders manage volunteers, not employees. Volunteers aren't given specific tasks with completion deadlines. Volunteers are guided to available tasks and encouraged to contribute, but no clear results expectations exist. Furthermore, there are no tangible performance incentives for volunteers, and no performance reviews. Often the only motivation for volunteers is notoriety within the community, but testers don't gain notoriety as quickly as core developers. Testers who contribute to the project in ways other than bug reports (by submitting bug fixes, working on other project areas outside of testing, etc.) may gain notoriety, but testers' work goes unrecognized unless individual testers contribute in ways that attract attention.
Public, global access to software under test is required. Testing volunteers need constant access to the latest software source and builds. They must be able to download the software under test at any time from anywhere via the Internet. This introduces a new set of challenges for build creation, automation, and distribution.
Need for a public bug tracking system. Consider your current project's bug reporting and tracking system: How many people use it? If everyone who has access to your project software filed bug reports, would your bug tracking system effectively manage them?
With hundreds of community members working with an open source software project, a publicly accessible bug reporting and tracking system must be available. The system must scale effectively; many bug reports come in constantly from test team members and other users and developers. Furthermore, bug reporters must be taught how to file effective reports. In short, open source projects must provide open, scalable bug reporting and tracking systems that effectively manage bug reports from the time of initial reporting to the time of resolution.
What Resources Do Open Source Projects Utilize to Accomplish Testing Tasks?
Community volunteers. The collective power of community volunteers drives open source development. Open source test leads develop testing communities around their projects. This is primarily accomplished through the use of project testing Web sites and "bug days."
A project's testing Web site explains the testing team's goals and needs, empowers volunteers to contribute, and provides resources for team members to get started testing, such as installation instructions, bug reporting guidelines, and general testing instructions.
Test plans are often provided on the testing Web site. The test lead and other team members contribute test plans as they are developed. Although test plans provide critical test cases to execute, a large portion of their value is lost in this environment since test execution results can't be recorded and analyzed in an efficient manner. Testers can reference test cases in their bug reports, but there isn't a collection of historic test execution data. Test leads can't objectively determine which project areas need more testing or which areas are getting too much attention.
Bug days are scheduled days (typically once a month) dedicated to testing. Typically the testing lead and experienced volunteers provide tips and tricks to get new volunteers up and running, which provides a good forum for ramping up testing teams and exposing new users to project software-a great community-building activity, conducted via online chat.
Electronic communication. Testing volunteers are spread across the globe, so face-to-face contact rarely occurs. Electronic communication is the primary method of project discussion. Testing-specific mailing lists are maintained that help facilitate the discussion, as well as one-on-one email between team members. A second tool greatly used is Internet Relay Chat (IRC). Most projects maintain an IRC server for team communication. This allows for immediate, real-time communication-the closest thing to face-to-face contact a project can have.
Online access to the latest software source and builds. Testers need access to the latest project software-often both the source code and binary executables. To facilitate this, most projects provide build servers connected to the Internet from which anyone can grab the latest and greatest software. Build processes and source code maintenance are automated as much as possible. Build servers are often configured to provide software in the form easiest for testers and end users to get installed and running. This helps more volunteers get started testing project software quickly.
The Bugzilla defect tracking system. Open source projects need bug reporting and bug tracking systems. For smaller projects, bug reporting and tracking is typically managed via email and is the lead developer's responsibility. However, sizable projects must use a sophisticated system. The Bugzilla defect tracking system is used by many large open source projects (e.g., Mozilla, GNOME, and Nautilus). Bugzilla is a database for defect tracking coupled with an HTML front-end. This interface works well over the Internet. Testers can file bug reports quickly and effectively, and perform sophisticated queries. Furthermore, Bugzilla can be customized and tailored to an individual project's needs, since Bugzilla itself is open source software.
What Are Some Advantages and Disadvantages to Open Source Testing Methods?
Open source testing communities possess both advantages and disadvantages when compared to closed source, commercial testing activities. Two advantages are
Community involvement with few limits. When testing communities are built successfully, the open source project benefits. An able group of testers can expose bugs efficiently, write effective bug reports, and help deliver a better, polished project as development marches forward. Much of this is true for commercial development also, but open source testing communities are only limited by the number of volunteers they can attract and the team leader's effectiveness.
Early and constant exposure. The open source development model exposes project software to many users early in the development cycle. The software is available via the Internet to any and all who request it. With this limitless exposure, bugs are diagnosed and exposed quickly, design limitations are observed and corrected, and the project evolves quickly, making rapid gains well before its initial "release." Constant exposure of the software to testers facilitates bug detection and resolution.
However, these advantages can be a double-edged sword. Two disadvantages of open source testing are
Community development efforts require time and energy. While a testing community's strength lies in its volunteers and its leadership, community development efforts can take away from much-needed testing activities. For example, testing team leads have to constantly work to develop project awareness, recruit new volunteers, and steer team efforts. For a team spread across the globe, these activities can bog down a testing group. This is a disadvantage when compared to a commercial developer's core testing team with budgeted time and resources.
Goal determination can fluctuate. When a testing team's strength and effectiveness are in a constant flux, dependent upon the ebb and flow of volunteers, precise goal planning is difficult, if not impossible. The volunteer community environment makes it difficult to plan goals and schedules for reaching those goals, with no clear measure of productivity or team effectiveness.
Summary
For open source projects, test team building is an exercise in community development. Without the traditional resources that commercial projects possess, open source test teams must overcome unique challenges to become productive. First, projects must develop awareness within the general development community and begin to recruit testing volunteers. This is typically accomplished via project mailing lists and an effective testing project Web site. Second, projects must provide the basic tools for a testing team to function-communication resources (e.g., mailing lists and IRC), access to the project software, and an effective defect tracking system. Once testing activities commence and a team forms, developing the community is a constant activity: test leads must guide testers, recruit more testers as other volunteers leave, and generally breathe life into the testing team and ensure its continued effectiveness.
Once given the basic tools, open source volunteer testing teams can be incredibly effective, on par with (or sometimes even greater than) their commercial counterparts. However, there are limitations to this testing environment that community development won't cure. First, community development can become an all-encompassing activity in and of itself, and testing activity will suffer as a result. Second, in the community volunteer environment, there is no accurate gauge of productivity for a team that constantly fluctuates. This limitation makes goal determination nearly impossible. Commercial testing teams have the edge here-goals can be set and timelines established based on given productivity levels.
Related Interest Items
The Open Source Initiative
The Free Software Foundation
Bugzilla Bug Tracking System
The GNOME Desktop Environment
