In this interview, Rajini Padmanaban shares some of the innovation being created through crowdsourced testing. Its immense versatility helps provide a fantastic return on investment for projects of all sizes; have you begun implementing crowdsourcing yet? Here's why you should consider it.
Noel: The abstract for your upcoming STAREAST session "Implementing a Crowd Sourced Test Effort" states that "achieving quality in the current day scenario calls for newer and innovative test techniques." What is it about today's times that require such innovation?
Rajini: Software development processes and methodologies are evolving lately to accommodate changing user needs and requirements. These are core drivers for success in the market place. These core drivers are necessitating several radical changes in how software is developed including:
a. Faster time to market
b. A collective ownership of quality
c. The need to focus on product domain knowledge, understand competing products, creatively emulate end user scenarios and mimic end user environments in test labs
Traditional testing techniques involving a dedicated test team will continue to be used to enhance product quality amidst these changing scenarios. The question is, will they alone suffice to ship a quality product within these constraints that teams operate within? This is why newer and innovative techniques such as crowd sourced testing are becoming an additional need in the current day.
Noel: What specific qualities about crowd sourced testing made you a believer in it, and then a proponent for it to be used on a much wider scale than just your own projects?
Rajini: Crowd sourced testing is a very versatile technique. It has a very simple definition of bringing in the community at large to test for your product, yet is very powerful and scalable spanning across companies (whether product or services, small or large), technologies, domains. It helps bring in diversity into the testing effort, enhance team productivity at very low costs, and strengthen test coverage especially across multiple geographies and devices which might otherwise not be possible to mobilize within an in-house test effort.
I’ve practically seen examples where it has been leveraged in large ISVs such as Microsoft as well as mid-sized services companies such as ours (QA InfoTech). This versatility combined with amazing return on investment, once the effort has been well planned, are what excite me about crowd sourced testing.
Noel: When you're defining crowd sourced testing for groups or companies that you speak with, do you see any variations in the definition, or more specifically, misconceptions that people may have about what crowd sourced testing really is?
Rajini: Yes, in fact one of the core things I address in my sessions / discussions on this topic is to help people understand the varied manifestations of crowd sourced testing.
Often there is a misconception that crowd sourced testing is all about engaging / working with a company whose business model is to pool in a crowd of testers, who get paid for valid bugs they report. Well, this is definitely one form of crowd sourced testing, but there are several other more important and creative manifestations to look at, to leverage the crowd’s wisdom including:
a. Sourcing relevant people from within one’s company although they might not be directly working on the said project
b. Building a pool of end users gradually over time to provide feedback across various phases of product development (be it public beta users, dog-food programs, private betas covering MVPs – Most Valuable Players etc.)
c. Partnering or working with organizations and universities who have domain knowledge and subject matter expertise or are capable of representing the end users in testing your product
The important thing to keep in mind in all these definitions is that money is not what always motivates the crowd. There are other factors such as brand loyalty, transparency and a sense of inclusion in your product before it is released, community involvement etc. that might really excite the crowd in testing your product. Understanding what motivates your crowd and what form of implementation will benefit your project are critical factors in building a successful crowd sourced test effort.
Noel: With the ability to customize or leverage crowd sourced testing to a particular development project's needs, what still holds it back sometimes from perhaps not being the best fit for certain projects?
Rajini: Great question. While we tout the benefits of this model and the need for it in the current day development world, acknowledging that this technique may not suit all projects or at all times is equally important. Typically crowd sourced testing is not recommended in areas where
a. Projects or certain components require a lot of internal collaboration and have a lot of dynamic moving pieces
b. Projects operate within very tight privacy and IP policies
c. Tasks require very prompt and responsive communication such as executing BVTs, regression tests
d. Core testing tasks need to be retained in-house such as test automation, first level of performance or security testing where base-lining results are important
Crowd sourced testing for any of these areas might make the execution effort more challenging, adversely impact the company’s competitive positioning in the marketplace, de-motivate the internal testing team. So it is important to understand and acknowledge not just “what to crowd test” but also “what not to crowd test”.
Noel: For those that are looking into incorporating crowd sourced testing for the first time, or even if they've attempted it in the past but maybe did not have success, where would you suggest they begin to perhaps give them a better chance for success in the future?
Rajini: Before looking at this internally within a project, let’s look externally and understand when a crowd succeeds. When a crowd succeeds, you automatically succeed making it a WIN:WIN scenario. The crowd typically succeeds when:
a. Diversity is brought in. The more diverse the crowd, the feedback that comes in is invaluable
b. Careful thought is given whether communication amongst the crowd should be allowed or not. There is no right or wrong answer here and this really depends on each individual product and project.
c. The product has a global user base, needs specific domain knowledge that is difficult to source internally, needs specific user environments that are difficult and complex to set up in the test lab
Now, let us look internally within the project team. To be successful in this space, start small. Start with an area where you want to run a pilot and learn from the experience. Carefully think through what, when and how to crowd test specific to your project’s constraints. Work with your stakeholders to get their approval and finally build a very selective and relevant crowd test team over time.
Although it is scalable do not expect to put together a crowd team overnight. For example, we recently setup a crowd sourced testing registration page on our website, with the hope of building a representative crowd for testing our client’s products. So, make this an educated and informed effort and you will be well on your path to success.
As director of engagement at QA InfoTech, Rajini Padmanaban leads the engagement and relationship management for some of QA InfoTech's largest and most strategic accounts. Rajini has more than eleven years of professional experience, primarily in the software quality assurance area. She actively advocates software quality assurance through evangelistic activities; writes on test trends, technologies, and best practices; speaks at conferences including STAR EAST 2012 and STAR WEST 2012; and provides insights on software testing to analyst firms such as Gartner and IDC.
