A company used crowdsourced testing as part of the testing process when redesigning its website. This testing employed internal resources to achieve the benefits of crowdsourced testing at a greatly reduced cost and provided the added benefit of getting company employees used to the new site. Read on for a review of the process.
My company’s website, PNM.com, was getting a total redesign, which included adding many new features and self-service functions. The site was designed as a blend of adaptive and responsive design to provide the best possible experience for customers using PCs, tablets, and mobile devices.
We had written and executed more than a thousand test cases, but they were performed by professional testers using desktop PCs and a limited pool of mobile devices. We needed to get feedback about field use, where customers might use any device. As always, time was short.
Old website
New website
Crowdsourced testing is the process where testers test on their own terms, using their own devices and on their own time. This testing is used by all major web-based companies. Our solution was to do crowdsourced testing but to use our employees as the testers. The goal was to find bugs and risks to product adoption, not complete defined test scripts. This turned out to be a very effective way to get much larger test coverage of the universe of possible customer experiences. There were many benefits we experienced in using this testing method:
- As the local utility, most of our employees are also customers and had accounts set up in the test database, which was a copy from production.
- Our employees understand the business model of the company, so they could move forward with very brief, limited training.
- Employees could use their own personal devices and home networks located in our service area.
- Our employees understand what is important to our customers and could focus on those things.
- The testing process allowed our employees to get familiar with the new website so they could promote it to their friends and neighbors in conjunction with our marketing efforts.
We called our project the PNM.com Bug Hunt and marketed it to the employees in several ways. Test leadership sent postcards to our target audience in customer service and BTS, the internal technical group. These groups have a high level of technical skills and also direct experience related to what our customers will want most. We also placed posters throughout the company to attract a large and diverse testing pool and wrote articles for the company’s internal website.
Traditional crowdsourced testing is driven by paying for bugs found. One of our goals was also to get a large pool of participating testers, so we turned the exercise into a game, a process called “gamification,” and provided rewards to players, including branded prizes and cash.
Our testers reported bugs on a SharePoint page; they could also visit the page to see what the other hunters were reporting. By keeping the SharePoint site easy to use and available to everyone, we increased the chances people would make the effort to repot bugs, even at home or on mobile devices.
So, What Happened?
We found so many bugs in the first cycle that we ran a second cycle in order to build confidence for our site team—plus, our bug hunters were interested in doing more testing. We had:
- 56 bug hunters (the majority of user-acceptance testing cycles here have ten users at most)
- 561 bugs found and reported
- More than two hundred hours of testing time reported
- More than a hundred different devices used for testing (PCs, laptops, cellphones, tablets, etc.)
- 80 percent of the bug hunters reported that they tested to help the company
- 93 percent said they would do this testing again if asked
The cost of the crowdsourced bug hunt was less than $3,000, which included $1,300 in cash rewards.
We also were able to build an excited and experienced employee user group on the website prior to its going live publicly.
Conclusion
The Bug Hunt worked well for us in a number of ways. First, our testers were from all over the state, just like our customers. They also used many different personal devices, and we gained real confidence our site would not have major device or network issues when it went live.
The testers reported many bugs we were able to fix before the public rollout, which gave us more capacity to deal with the ones that slipped by after. And our customer support team was ready on day one for calls and questions about the new website because they had had time to get used to it.
The bug hunt was a cost-effective method to test our new website, and it had the extra benefit of educating our employees on the new features and functions of the site. And most importantly, our employees were involved and excited about the project and felt like an important part of the team.
User Comments
I wonder how did internal testing team reacted to such crushing result?
It was a very good experience as most of the items were concerning undocumented features and where the offshore testing did not fully understand how the site was going to be used by US customers. We learned a lot from the process and other projects here are interested in doing the concept again as it was much better to find this stuff before the full release
A site's ease of use or its usability, is an integral part of its success, especially with websites becoming more and more interactive. website testing is the single most important parameter that defines the success of application in the market. Using crowdsourcing to test website is proven to yield much better results than traditional outsourcing services. I have come across http://qualitrix.com/ which is empowered by crowd sourcing model, is designed to ensure that websites gets tested thoroughly from all angles so that its behavior is completely predictable in the real world.
Cheers Jack!