The Testing Eight (Not Coming to a Movie Screen Near You)

If you’re a movie buff, you’ve probably heard of the Magnificent Seven or the Dirty Dozen. Well, I want to talk to you about the slightly less catchy sounding “The Testing Eight.” The Testing Eight is not the name of the latest Hollywood blockbuster but instead refers to eight testing fundamentals whose principles can be applied to any development methodology to ensure the quality of its deliverables.
 
Development methodologies seem to be everchanging nowadays with new buzz words and definitions springing up all the time. The traditions of Waterfall and V-Model have been joined by DevOps and the Agile and Lean methodologies of Scrum and Kanban. With this shift toward development agility and people’s misunderstanding of these methodologies, there has been a tendency to focus on speed of delivery—sometimes to the detriment of quality. Obviously, with any software development, it is essential to deliver efficiently and to meet deadlines, but we shouldn’t neglect the basic principles that help ensure that what is being delivered is fit for purpose.

Contrary to popular belief, quality assurance doesn’t have to be onerous or time-intensive. As the old adage goes, “A stitch in time saves nine.” Employing basic QA techniques early in the development lifecycle will save a lot of time, pain, and effort in the long run; after all, it’s far cheaper in project terms to weed out issues as early as possible rather than having to fix the same issues later in the lifecycle. This sounds like a no-brainer, but given the current appetite to be Lean and Agile, how can this be achieved? If this were a cowboy film, this is where The Testing Eight would come riding over the horizon.

1. Know Your Scope
This seems like a no brainer! Knowing what you’re planning to test should be a given, but this is not always as clear as you’d expect as certain areas of functionality may move into and out of development scope for various reasons outside the control of the testing team. Clarity on scope is critical when trying to ensure deliverable quality. If it’s not clear what is being delivered it’s impossible to measure success, so a summary of testing scope needs to be agreed upon within the project team and documented for reference.

2. Share and Review
Remember when you used to ask your mom to read through your homework before you handed it in? The same practice provides benefits when applied in IT development. Whatever your development methodology looks like, reviewing of requirements, designs, test requirements, and test cases should be standard within the project team. Identifying and resolving a requirement issue at the requirement gathering stage is much cheaper than letting that issue permeate into the design and become a defect in the software that is later identified by a test. Reviewing also promotes collaboration and knowledge transfer within the project team, with different teams having better awareness and appreciation of each other’s deliverables.
 
3. Show Where Your Tests Hail From
Once scope has been agreed upon and test requirements have been extracted, it’s essential to demonstrate testing traceability. This is achieved by simply providing traceability between each test and the source from which it has been derived. This could be requirements, designs, or wireframe documents. Testing traceability is important as it demonstrates test requirement coverage against scope. A review of the documented traceability by the project team can highlight any areas that have been missed or shouldn’t be tested.

4. All Tests Are Not Created Equal
Any experienced test professional will tell you that test windows ALWAYS get squeezed. Even if this wasn’t the case, it makes sense to employ a risk-based approach to prioritize testing of the most important tests, which will give you greater confidence over those of lesser importance. Once test cases have been extracted, it is important to grade them from a risk of failure and impact of failure perspective—with a high, medium, or low test priority. The risk of failure can be determined through discussion with the development team, while the impact of failure requires input from business or technical support teams. Testing of high-risk, high-impact areas should be prioritized to identify any potential issues and deliver confidence in these areas as early as possible in the test execution window.

5. Plan Your Execution Effort
Once test cases have been assigned priorities and all test cases have been prepared and reviewed, the next step is to plan the test execution effort over the execution window. This will take into consideration the assigned test priorities by planning to execute the high, medium, and low priority tests in that order. Planning should include a daily schedule of what tests are expected to be conducted on each given day of the testing window and should include a degree of contingency to accommodate any potential issues and the time taken to raise and retest any defects. This plan will also provide a measure against which to report progress once test execution begins.

6. Know Your Entry Gate
Entry criteria detail the conditions that need to be satisfied for test execution to be able to start. These criteria will include external dependencies, such as approved requirement/design documents, scope definition, test environments, test data, test users, and test team activities such as test prep, completion, and review. Entry criteria should be used as a quality gate into test execution, ensuring that all the required conditions are in place prior to the start of execution. An entry gate meeting is usually held with the relevant delivery stakeholders to review these criteria and make an informed decision whether all entry criteria have been satisfied and all parties are happy for test execution to begin.

7. Know Your Exit Gate
Establishing and agreeing on your exit criteria is critical for any phase of testing. The exit criteria set out what testing is aiming to achieve and what success looks like. As part of planning, you need to identify what you need to achieve before your testing can be deemed complete. These criteria are usually a measure of test coverage and defect numbers and types. For example, your criteria may specify that:

• 100% of all high and medium priority tests must have been executed

• No severity/priority level 1 or 2 defects are outstanding

• Severity/priority level 3 defects may be outstanding, as long as an acceptable resolution process and timescale has been defined


8. Communicate Your Results
Reporting can be separated into progress and completion reporting:

• Progress Reporting: Accurate and efficient reporting of progress during test execution is essential as it provides a barometer for the quality of the deliverable under test. Many test tools now facilitate progress reporting, but in simple terms, reporting should provide a measure of tests executed against tests planned and of the executed tests, those passed, failed, blocked, etc., and defect stats in terms of numbers and priority/severity. Reports should be granular enough to highlight any potential problem areas; therefore, it makes sense to provide granularity on individual functional areas and the status of the high, medium, and low priority tests for each. Reports are usually produced on a daily basis in order to provide updates to interested stakeholders.

• Completion Reporting: A brief completion report summarizes the testing completed during test execution. This should again be granular enough to provide test coverage and results achieved for the individual functional areas under test and summarize defects discovered, resolved, and outstanding at the end of execution. Ideally, this report will also provide root cause analysis of defects identified, so that this can be used to address problem areas going forward. This document will inform the development team and stakeholders as to the quality of the deliverables that were tested and allows them to decide whether this is sufficient.


So, there you have them—The Testing Eight—not coming to a movie screen near you but instead being used to improve the quality of an app or IT system near you! As you can see, some of these fundamentals are dependent on collaboration within the project team and lend themselves to interactive meetings and workshop sessions. Discussions and agreements are good, but without recording the outputs of discussions, important project decisions can be lost, so quality assurance is dependent on a certain degree of documentation.

 
 
Now, while the word “documentation” can strike fear in many IT professionals who have cut their teeth in Agile or Lean development environments, it needn’t. A common-sense approach can be adopted to ensure that the detail supporting these fundamentals are captured and recorded in a way that complement and don’t inhibit the development methodology in use, however Lean it may be. This detail may include key requirements, decisions, project assets, and test cases to name a few examples. The recording of these can be easily achieved with brief documents or spreadsheets—they just need to be clear and easy to understand when distributed for stakeholder review and approval—but it is essential to have a record of the detail that has been agreed upon.
 
The important thing to remember with quality assurance is to strike the right balance between governance and development speed. The right level of governance can ensure the quality of the systems that your organizations depend on without being onerous, giving you the confidence that they can deliver the functionality that you and your customers need and expect.