Quality assurance (QA) professionals often are squeezed into the Web application development process too late in the game, and thus have no chance for success. Common QA frustrations include lack of planning and processes, unrealistic schedules, and lean staffing and financial resources allotted for the QA and testing phase of a project. RadView's Michal Blumenstyk offers guidelines to ensure QA can deliver what it promised.
While in the past QA may have been less of a focus due to time and money, today's Web applications are strategic components of corporate success and simply cannot fall short.
Successful organizations rely on Web applications as effective points of contact with customers, partners, and employees, and as a means to help reduce costs internally. Web applications are at work, for example, when a customer fills out a response form on a Web site or when a salesperson files a purchase order via a wireless Internet device. It is no longer an option of whether or not to perform a battery of tests on this Web application before they go live; testing and analysis are critical where Web applications are concerned.
Today's Web administrator understands that these applications cannot shut down or post incorrect data for any user-that the assurance of quality is a must. Today's QA professional also understands the Web administrator's challenge, and would be better positioned to help if functional and load testing methods were implemented earlier in the development cycle. When will this collaboration truly take shape so QA is more closely united with development teams in the planning stages of a Web development project? What are the best ways to go about testing to assure organizations that today's business-critical Web applications won't fall down on the job? How can small-to mid-size companies prepare their Web applications to compete with the applications of larger organizations?
Costs of Not Testing
Most people are aware of the high-profile sites that could not handle load, such as the Victoria's Secret fashion show, the dot-coms that advertised at Super Bowl 2000, and more recently the FIFA World Cup Soccer site for ticket sales. While most Web site failures don't make headlines, even partial downtime of business-critical Web sites is not cheap. Overall, the financial losses and perception of a poorly functioning Web site can be long-term. In the financial services industry, companies could lose a staggering $6.5 million per hour in brokerage operations, and $2.4 million per hour in credit card sales applications. Media companies could lose $150,000 per hour in lost pay-per-view applications, and an estimated $113,000 per hour in home shopping TV. Companies in the travel industry stand to lose as much as $89,500 per hour in lost airline reservations from down and poorly functioning Web sites.
Industry analyst firm Zona Research published reports that have become almost gospel in the industry by citing the risks: "Page load times of eight seconds or more, combined with normal ISP page download and connection failures, could cause as many as 30 percent of online buyers to "bail out" of a site before buying anything. The combined losses of these site problems could be as high as $362 million a month."
But how real are the risks of Web site failure? "Of the e-commerce sites we have tested, 99 percent have failed to meet desired performance levels" on a first test, according to J.D. Brisk, regional vice president of Exodus Performance Labs, now KeyLabs. In fact, Exodus Performance Labs has actually tested and found sites that do not function correctly or even at all with as few as two simultaneous users-even though the sites theoretically were designed for larger numbers.
A Systematic Approach to Scalability and Functional Testing
One sizeable financial services company has done an exemplary job in global planning for its large-scale Web testing. They understand that a Web site is an important tool in providing a broad range of funds and services to institutional and individual investors. Before embarking on the testing process, they outlined a process for the testing phase to make sure it was linked with early-stage development and met business objectives.
- Develop business metrics. This is where the performance criteria for the site are defined. Good performance criteria describe an end user's experience in relation to a certain load. For example: "While under the load of 5,000 concurrent users each page on our site will load within eight seconds of being requested."
- Establish requirements and expectations. Clearly define the test objectives. Specify what will be reported.
- Document performance methodology. Describe exactly how the test defined below should be run.
- Get management buy-in. This part is up to you, but the earlier steps provide a ogical basis for a solid presentation to management.
- Choose whether to outsource or bring dedicated resources, such as hardware, software, and personnel, in-house; either way, one must
- make sure testing activities are automated, that is, able to run without manual influence
- establish dedicated test data
- document results
But what should you actually test Web applications for?
- Functionality testing ensures that all aspects of the site function properly: that objects such as pictures, text, and forms appear correctly, links work, form submissions succeed, etc.
- Compatibility testing ensures functionality with different browsers and operating systems.
- Usability testing measures the ease with which a user can accomplish predefined tasks.
- Stress testing determines the system's breaking point based on predefined failure criteria.
- Load testing generates user traffic on the Web site to determine if the site is capable of handling a predetermined peak load.
Essentially, testing should exercise and evaluate site performance from a user's perspective. Some companies know the value in testing peaks in traffic after a promotion or a Super Bowl advertisement, for example, but what about the unexpected catastrophe, or a run on the stock price? Reliability and load testing can be important supporting factors to a company's crisis-communication and investor-relations plan. It is also important to remember that functionality and scalability testing are not separate testing activities, and need to be performed concurrently for a realistic emulation of the live Internet.
Testing performance under a virtual load of Internet traffic prior to being posted on the public Internet is vital to gain a clear understanding of how a Web application will perform when it does go live. For example, another large, well-known financial services organization detected certain problems only when they tested with 500 concurrent users, problems that would have gone unnoticed if they had merely performed scalability testing. They found that when a certain Web application supported a single user to 499, users were able to receive correct financial information online. However, once the concurrent users reached 500 and above, financial records and confidential information were given to the wrong users. The large financial services company became aware of these issues only when testing functionality under heavy virtual loads.* In turn, they were able to correct the problem prior to the Web application going live and public, and before it would become a significant cost to fix.
Testing Methodologies
Industry analyst firm Newport Group believes that the structure and management of those testing practices is in need of improvement. For example, 51 percent of the e-commerce companies surveyed reported that no test standards were followed within their organizations. And 83 percent of that survey population segment indicated that there were no plans to implement or adhere to testing standards within the next twelve months.
Ultimately, testing Web applications-large or small-early and often-requires a level of expertise that few companies are able to develop in-house. But QA professionals, testing consultants, and testing labs are available to help companies develop large-scale Web applications that are reliable, scalable, and in keeping with a company's critical business objectives. The testing and test planning can be done remotely without the need to travel to a physical site. This can keep costs down considerably, even those with very large-scale Web applications.
KeyLabs, for example, evaluate testing methods before actually running the test-monitoring how a site performs in combination with the test solution, the number of machines participating in the test, and the geographic locations where the load generators are located.
To minimize the risk of major structural issues being discovered late in the lifecycle, consultants should advise clients to perform scalability testing well before the bulk of the program's data coding has been completed and design decisions have been finalized. This makes it possible to flesh out all the scalability issues from the top to the bottom-in each application tier, subsystem, or component-well before final integration takes place.
To provide real-world tests, I advocate taking a structured approach-doing multiple tests, testing on different days, and conducting multiple test rounds. This gives site and application developers time to analyze performance statistics they are getting, optimize the back-end processes they are running, and fix bugs. It often takes two or three rounds of testing for companies to reach their targets. To thoroughly troubleshoot before going live, I recommend not just one test, but a true testing process that builds solid Web sites.
Industry analyst firm Hurwitz Group supports the integration of testing saying, "Functional testing is the traditional turf of QA. No matter how well the QA team performs its functional testing tasks; however, the ever-broadening scope of e-Business puts new demands on the entire development team…Quality processes must adapt and evolve to extend functionality and quality into this new environment…Technologies and procedures that were once limited to QA require distribution across the lifecycle."
In planning for the unknown factor of the World Wide Web audience, consultant professionals should also account for the international factor. Because Europe and the Asia Pacific regions have rushed to embrace the Web almost as quickly as the US, it is all the more important for consultants to factor global testing into their development cycle. Global access to their customers' Web applications means different amounts of latency, meaning the time between initiating a request for data and the beginning of the actual data transfer, which cause the connections between the browsers and the server to take different lengths of time. A realistic test environment that factors in these latencies and time lag is critical for real-world testing.
Scalability: How Large Is Large?
In mapping out a test plan, a consultant's first step is often accounting for scalability. The scale of the testing process varies, depending on the complexity of the Web site. Most new large-scale companies are initially shooting to test between 1,000 to 5,000 concurrent users; however, this is a test that typically many organizations cannot conduct themselves. It would cost them more to test accurately in-house than to build out the back-end production infrastructure.
In determining that the tests will be effective for the site in question, consultants should keep in mind one of the biggest factors-the number of users that can run on a specific load generator or specific piece of hardware. Every Web site is different, and the actions that the user takes on the Web site are different. Load generators emulate the multiplicity of users and their various Web browsers and connection speeds. A seasoned testing consultant or testing organization should plan ahead to create test scripts that are able to test the Web site as efficiently and realistically as possible and to secure several, often geographically dispersed, machines across multiple networks to generate the virtual load.
In large-scale testing, as with any Web application testing, resource efficiency is key to increasing the return on investment. If a virtual load can be generated on two machines instead of ten, or on a single laptop instead of a handful of expensive systems, it makes a big difference. As we have discussed, it is important to assess functionality under load, and if a testing consultant is working with a customer who has limited hardware resources or budget, the ability to maximize testing solutions becomes paramount.
With successful companies relying more than ever on their Web applications for customer relations, cost avoidance, and revenue generation, the pressure is greater than ever to get it done right. Companies are under unprecedented pressure to build robust applications and plan for the unexpected spikes of the live Internet. Getting the Web application performance testing process "right"; however, does not have to be a difficult process. Following the best practices outlined in this article will help align QA with the rest of the development process, ensuring the success of testing efforts and preparing the Web to face the masses.
* The virtual-heavy-load-testing software this company used was RadView's WebLOAD software.
