Get User Feedback

Agile methodology to software development emphasizes collaboration with the end users, therefore first and foremost you should get user feedback on the latest existing software version, and find what could be improved. User feedback can be obtained through request forms, user forums, and as direct feedback from existing, established users. When getting user feedback, it is important to obtain information about the existing software version from the user’s standpoint, as the user may not be aware of new features already suggested for the next version. Obtain user feedback about improvements that should be made to existing features, new features to add, and bugs to fix. Also get user feedback on compatible software to support. As an example, a user may request for a new software version to support a newer version of Java, or Apache Hadoop on which the software depends.  

Have Scrum Meetings  

Scrum is an Agile framework that helps teams explore issues in an incremental and collaborative way. A Scrum Team consists of a Scrum Master, a Product Owner, and Developers. The Master organizes and implements the Scrum process and manages the Scrum Team. The Product Owner establishes the Product Goal, which is the long term objective for a product. The product owner also creates a Product Backlog, which is a list of tasks to implement to reach the product goal. The Developers are the Scrum Team members who actually perform the tasks established in a Product Backlog. You should have Scrum meetings that include both end users and developers. 

Discover Features to Improve

From user feedback and Scrum meetings you can discover the existing features that could be improved in the software. Typically such features may reduce the code that is needed to implement a task to fewer lines, add support for new configuration settings, add new support to existing API, add new feature options, and improve performance. Such improvements are not directed at specific users, and should be added only if they affect a wider user base. When making improvements to existing features, identify any inconsistencies the improvement may introduce, and whether the improvements would break existing code making use of those features. Whether improving or adding features, compatibility with existing code is a priority.

Discover New Features to Add

Similarly, Scrum meetings and user feedback can also aid in the discovery of new features to add. It is important to distinguish between making an improvement to an existing feature, and adding a completely new feature. Also worth noting is the features to add or enhance should be prioritized, because when developers implement changes, the top features are implemented first. Keeping with schedule is also a factor, so not all feature enhancements and additions that are initially listed for a new version make it to the final software version release. New features are typically not introduced as finalized features in a new software version. A new version may introduce a new feature as an experimental feature, or a preview feature. 

Discover Features to Deprecate

As software evolves, some existing features may need to be deprecated and eventually removed. As an example, OpenJDK 17 deprecates the Applet API for Removal. A deprecated feature can still be used, but typically an alternative to a deprecated feature is recommended. The deprecated features are not recommended for use in other software production as any software that depends on a deprecated feature won’t run after the feature is removed in a later version.

Identify Bugs to Fix

Most software has bugs that need to be fixed, and naturally as users use the software, they discover them. Some software even provides a web page to submit bugs. For example, Java has a special web page to report bugs. Developers identify which of the reported bugs are actually bugs, and which are due to some other issue such as software not being used as needed, such as an incorrect configuration setting. Also, it needs to be identified which of the bugs should actually be fixed, as some bugs may be related to some feature improvement, or deprecation. If a feature improvement removes a bug, or a feature deprecation and removal makes fixing a bug unnecessary, the bug is indirectly fixed.

Set Target Compatibility Matrix

Most software has dependencies on other software. Developing a new software version also involves upgrading the dependencies. For this, create a compatibility matrix for the new software in relation to its dependencies. New software versions should retain binary compatibility such that existing applications continue to run. As an example, Apache Hadoop sets the compatibility guidelines.

Create Jira Issues 

After performing an initial review of features to improve, the next step is to add, fix, and create Jira issues. Jira is an Agile project management framework used by software development teams to create, discuss, and track issues during software development. Create a separate Jira issue for each task to be performed, whether it is a new feature, a user request, or a bug to be fixed. Jira categorizes issues into different types, so make sure to select the issue type when creating a new issue. Jira provides templates to support the commonly used Agile development frameworks Scrum and Kanban. Jira also provides templates for bug tracking and DevOps. Alternatives such as Asana, or Trello may also be used.

Create a List of Tasks

After going through Scrum meetings, getting user feedback, and getting developer feedback on Jira, the features to improve, add, and fix should have been established. Different software follows a different process to list the software changes to be made. As an example, Apache Kafka makes use of Kafka Improvement Proposals, and OpeJDK for Java makes use of Java Enhancement Proposals. Any such list of tasks identifies: the component of the software that a task affects, the scope of the improvement, the target version, and committer/s assigned to the task. 

Create a Schedule

A timetable, or schedule should be created for a new software version. Such a timetable should list what is to be completed in each phase of development. As an example the schedule for OpenJDK is as follows:

2022/12/08Rampdown Phase One (fork from main line)

2023/01/19Rampdown Phase Two

2023/02/09Initial Release Candidate

2023/02/23Final Release Candidate

2023/03/21General Availability

Assign Tasks to Committers

A software project typically has several committers; as an example, Apache Hadoop has more than 240 committers. Find suitable committers for each task, and assign all the tasks with at least one committer per task. Not all committers may need to be involved in a software version development.

Create a Git Branch

Create a new Git branch for the new software version and fork existing software source code to the new branch. Create sub-branches within the new branch and assign the sub-branches to different committers. As an example, Apache Hadoop has more than 300 branches on Git. 

Review Changes

For each task and branch assigned to the committers, review the changes made before merging the develop branch with the main branch. The task of merging sub-branches with the main branch is typically performed by a lead committer.

Test Software  

Test the software incrementally as changes are made. The different stages of a new software version typically include Alpha, Beta, and Release Candidate (RC) versions before the final General Availability (GA) version. Each intermediate stage is tested for functionality, and bug fixes.

Add Documentation to Source Code

An Agile approach includes adding documentation to the source code itself so that anyone referring to the source code in the future should be able to find what changes were made.  

Develop Release Notes

The Release notes for a new software version typically list the new features added, the feature improvements, and bug fixes. Release notes typically are linked with the actual tasks that perform the changes. An example is the Release Notes for Apache Kafka 3.4.0, which lists the features added and/or improved. Release notes also mention if a change is minor or major as in the Apache Hadoop 3.3.4 release notes. A changelog for each change made is also created. For example, the Changelog for Apache Hadoop 3.3.4  lists the changes categorized by type.

Develop Upgrade Documentation

While the Agile Manifesto emphasizes, “working software over comprehensive documentation,” the importance of documentation should not be undermined. Without an upgrade documentation, an end user won’t be able to upgrade to a new software version. An example of an upgrade documentation for Apache Kafka describes the upgrade process in detail.