Has Web Development Changed the Meaning of Testing?

I recently had a wonderful experience at an industry conference and learned some interesting things I thought I'd share with you. Usually I hate going to conferences on the topic of testing and quality assurance (they can be so boring when under-qualified, over-hyped, or overly-academic people get up to speak). However, I recently returned from the Practical Software Quality Techniques (PSQT) conference in Bloomington, Minnesota, held in conjunction with the PSTT conference (where T=Testing replaces Q=Quality). And, I am even more happy to say, the P=Practical word in the title was particularly well chosen. Presentations were by "been there, done that" kinds of people.

I went out of my way to choose sessions on Web application development (there were eight parallel tracks), mainly because I don't know much about Web development. My chief interest was to what degree Web development differs from traditional software development. The answer I learned, partly to my surprise, is "plenty."

What's different about Web application development? Web application development tends to be:

• requirements-free: There are seldom any stated requirements at the outset of a project. (One speaker described that phenomenon, only partly in jest, as "Specs? We don't need no stinkin' specs!")
• rapid-development: There is seldom time to do anything more than what one speaker called "slam it and jam it," using a process he called "FAD--Frantic Application Development."
• short fuse and relatively small: Most projects are expected to last about three months, using a maximum of four people.
• testing-primitive: There are few generally accepted processes, and the tools that are available often don't fit the problems well.
• user-intimate: Although you never meet your users, they are only a mouse click away from letting you know what they don't like--and from going over to the competition.
• a "living project" (two speakers used this term): Maintenance, usually by the developer, is a "must," and in fact "all products are in permanent beta test" (modifications are ongoing).
• performed with limited management support: Most managers think Web development is easy.
• highly aware of performance: Performance is vital--and extremely unpredictable.
• concerned with content: Content is king; it doesn't matter whether the application is "pretty" or not, if the content isn't interesting or useful.
• a whole new field: Developers are more "Wild Web developers" than software professionals.

In fact, that last item is key to understanding the difference between the Web and traditional software. One speaker expressed this difference very well, saying, "We've moved back two decades in testing approaches" (that is, no one is building the notion of Web applications testing on the lessons learned from the history of software testing), but went on to say "I'm not sure that traditional approaches will ever be applied." In another session, the issue became even more clear. Is it pure backwardness to build Web applications without requirements, or is it simply true that in any rapidly evolving discipline, it is impossible to know what the requirements are until the first (several?) prototypes are built? And in the absence of firm requirements, isn't it impossible to perform the rudiments of traditional testing and quality, such as requirements-driven testing? As the Web applications field matures, it will be fascinating to see what comes of these issues.

What characterizes a Web application? Andrea MacIntosh of QA Labs, Inc., said that the typical architecture is a browser tied to a Web server tied to an application server tied to a database. And what characterizes Web applications testing? The same speaker said that it's about multiple environments (you can't test a new version of a site on the live site), multiple platforms (you can't necessarily control the browser/operating system/languages/tools choices of your users), a new approach (you test "back to front," meaning starting with the server side and working out to the client side), and the focus is different (you must "think user," structure by whatever canned components you can use, and prioritize your testing according to where the risk of failure is largest). Two other speakers, addressing the issue of performance (and not specifically testing), said the same thing about "back to front" approaches. Brian Hambling and Angelina Samaroo, of ImagoQA, advised starting performance work at the back end, focusing especially on any legacy systems and their (baseline) performance. (Incidentally, regarding performance requirements, most speakers agreed that a maximum response time is about eight
seconds--anything more than that requires telling the user what is going on).

What about the state of the Web applications practice? Arthur Hicken of ParaSoft noted that "the average Web site has one bad link for every three and a half pages, and twelve errors for every page of HTML." "Ninety percent of browser incompatibilities," he went on to say, "are caused by using nonstandard HTML, and the other 10 percent are because the browser does nonstandard stuff." (He noted that many people blame Microsoft for deliberately doing nonstandard things so that their competitors could not get their code to work, but Hicken blamed the competitors themselves for their own nonstandard practices). And Mark Bonewell of Medtronic described his own lessons learned from a first-time experience testing a Web application. He noted that there were almost no requirements (by the time he said it, this had become a familiar theme at the conference) and that it was the first time that most programmers on the project had done Web development.

As I said, I usually hate going to conferences on the topic of testing and quality assurance. But Web testing conferences should offer new and changing information for some time to come. Each conference, like each Web development project, will have to address fluid and unstable "rules" as the field develops.