Stranger in a Strange Land: Bringing Quality Assurance to a Web Startup

It perhaps goes without saying that a Web startup is not an environment in which quality testing is typically found. Development is fast and loose. Many developers are inexperienced. Marketing is pushing to be first to market. One might be tempted to label the environment as chaotic

When I accepted the opportunity of being the first test engineer at TRIP.com, only 25 people worked for the Web startup. The developers had produced some exciting applications and felt they were ready to "grow up and play with the big boys." The development team thought they were intellectually prepared to introduce standards and procedures.

In reality, development was frenetic, and the developers didn't have a clue as to how to stop and analyze their processes, much less how to impose discipline on them.

For my part, I was a complete stranger to Web development. I truly felt lost in the wilderness without a map or a compass. For years I had been testing databases, fourth-generation languages, and client/server software on UNIX, NT, and Windows platforms. I spoke ODBC, but not JDBC. I knew my customers. In my experience, the software development cycle had stretched on for months or even years–during which your typical Web application has gone through numerous incarnations.

This is the story of how I learned about Web application development, preached the quality gospel, and collaborated with the software and product developers and marketing managers to implement development standards and project processes that build quality into our applications. TRIP.com now employs over two hundred people, has over four million registered customers, and has introduced such cutting-edge products such as FlightTracker and intelliTRIP. Myself, I've moved on to a new startup. Once again, I feel like a stranger–this time in the strange land of eXtreme Programming.But that's another paper.

Any DotCom is a work in progress. Even once we had a successful project process at TRIP.com, we faced continual challenges such as an inadequate test environment. Even when the whole company understands and is committed to the importance of quality assurance testing, unexpected events lead to surprises. The key is to keep plugging away at the following tasks:

• Get Buy-In
• Work Smart 

1. Get Buy-In
I won over my managers, developers, and marketing counterparts by following these tenets:

• Identify a top manager in your organization who believes in your cause and will champion it. In my case, it was the Vice President of Web Development and Chief Cat Herder (yes, that really was her title). When I was hired, this person did not believe that five developers could keep one tester busy full time. But, in time, she became my biggest ally. She not only pushed the developers to work for quality, but she also lobbied the management team for testing resources.
• Partner with someone outside of your organization, such as a project or operations manager. Educate your ally to garner his or her help. We had a topnotch project manager who, once she understood what QA and testing would do for the company, did much of my job for me. She enforced processes such as document review and signoff, helped implement and police the defect tracking system, and tied up a million details involved with every big production launch. She became the prime channel of communication between marketing and development.
• Educate everyone you come into contact with at the company about software quality assurance?what it is and what it will do for them and the site. Early on, I held a "Lunch 'n' learn" professional development seminar. The company bought lunch, so attendance was good. I explained why testing is essential and what it involves. Lots of meetings and one-on-one encounters are needed to get everyone on board and to establish priorities.
• Support Information Systems, the group that administers the production site. These people will benefit from not having their pager go off so much when applications are tested before being launched to production. The Vice President of Technology and the IS director fully supported me and refused to launch any update that had not been fully tested. This kept the rest of the organization from steamrolling over me, giving me a chance to prove the value of testing. I don't bug IS for the things I can do myself, but I let them do their job. For example, I installed Y2K patches on the test machines, but IS controlled all the UNIX and NT account management.
• Understand the developers' point of view. You may have young, brilliant developers who don't communicate in ways you are accustomed to. Our original developers were mostly very young and inexperienced. Some had not finished high school!. Others weren't old enough to drink! The culture was anti-corporate, and they said what was on their minds. I found that if I listened, I learned, and they in turn were willing to listen to my ideas. I learned everything I now know about Web applications from the developers themselves. I presented a "Quality Hero" award each month to a developer who took exceptional measures to prevent defects and improve quality. The prize was just a Nerf gun and the developer's name appeared on the Quality Hero Award plaque, but it raised the visibility of high quality process and techniques.
• Brainstorm with developers and others about problems that may not come out in testing. For example, I didn't know that if you change a URL, search engines may not be able to find your site. When we implemented a content management tool that required changing every URL, this was important information!

I. Work Smart
Here's my advice for making the testing organization lean and mean:

These tools won't necessarily meet your needs–just be open and creative when evaluating tools. Investigate alternatives–for example, if you create unit tests to reproduce each bug you find in testing, you may not even need a defect tracking system!

Software QA and Testing Resource Center: Everything from basic definition and articles on how to test Web applications to comprehensive lists of Web tools to links to other informative sites.

Articles by Cem Kaner

Of course, user conferences are invaluable for both information-gathering and networking

In short: Dig your heels in and refuse to launch until some semblance of a test environment is established. Remember, it is harder to get the test environment once the new application is in production. Make it a requirement of release.

• Evaluate tools. Put as much time as you can into tool evaluation, such as those for automated testing, defect tracking, and configuration management. Identify the vendors who can help you the most, and get as much information from them as you can. Ask fellow testers for their recommendations and experiences. Install new tools and try them out. Select tools that are appropriate for you and your company. It doesn't do any good to buy a tool you don't have time to learn how to use, especially if your testing team is small. I ended up choosing tools that are lesser known but stil meet our needs. For example:
• automated testing at TRIP.com (we use this at tensegrent as well), we used WebART, an incredibly inexpensive, easy-to-learn, but powerful tool sold by OCLC Inc. (a non-profit company). They gave me invaluable advice and provided insights about Web testing. Pick everyone's brain including vendors!
• For defect tracking at TRIP.com, we chose a Web-based tool, TeamTrack (now called TTrack), from a startup company called TeamShare. It, also was far less expensive than its competitors, but it was easy to implement and customize. At tensegrent, which is a brand-new startup on a small budget, we use the free Bugzilla and it is fine for our needs.
• configuration management, we again turned to a smaller, innovative company which produces an inexpensive, easy to implement and learn yet robust tool, Perforce. At tensegrent, we use freeware, CVS–it lacks some features, but the development team is small and can work around its drawbacks.
• Search the Web for resources. I would have quit TRIP.com after a week if I had not found an excellent Web site that points to information about testing Web applications and lists of tools. These sites, in turn, led me to more tools and information Here are some examples:
• Hire good help: It proved impossible at first to find experienced test engineers in our area, so we at TRIP.com hired bright but inexperienced people with the right qualities that make good test engineers: enthusiasm, dedication to the end user, and determination. A caveat: inexperienced testers who have no programming experience have a harder time learning a scripting language for an automated test tool. However, by using a combination of outside classes, hiring consultants and patient, one-on-one training, our testers learned UNIX, SQL, test scripting, HTML, configuration management, and other technical skills. You're going to spend money either way - paying high salaries for experienced test engineers, or training novice testers. Once you've turned them into pros, remember to keep them challenged, happy and well-compensated so other companies don't poach them!
• Get input about quality from all departments in the company. I formed a quality board with members from sales, marketing, customer support and travel to gather fresh ideas about error prevention and prioritization of regression testing. Whenever major problems occured, we held a quality review panel where representatives from development and information systems heard short presentations from the people who experienced and fixed the problem. The panel studied the issues and recommended steps to prevent such problems from recurring. This was a big effort and to be truthful, it was difficult to get follow-through. But give it a try. We also held post-mortems after all major launches to see what lessons could be learned. By employing these methods, we learned some valuable lessons!
• Insist on a test environment that is exact replica of, but is entirely independent from, production. You can't emulate a production load without the equivalent of production hardware and software. Since the production architecture is likely to change in response to increased traffic and other considerations, this is a moving target. The test environment will need to be updated in synch with the production environment. The architecture is key too. If the production servers are clustered, your testing had better be done in a clustered environment. If part of an application runs on a stand-alone machine, it must do so in your test environment. Establishing and keeping up development, test and production environments was a huge challenge. Even when the entire company is sold on the idea of a proper test environment, there are business and technical reasons (read: excuses) that get in the way of reproducing the production environment in for testing. Don't be complacent, and never give up. Make sure you have the best test environment you can get for each application going into production, and work actively with your information systems team to get the environment you really need. Even small applications can deceive you. For example, we launched a simple application, SantaTracker, on Christmas Eve so kiddies could watch Santa's sleigh fly around the country. The test environment was broken, so we were not able to test on the same architecture that it was to run in production, but we weren't concerned. After all, it should have worked exactly like our regular FlightTracker application! Right? Wrong! It was a disaster!
• Learn about the development tools. They present their own quality issues and offer some solutions, too. For example, the first version of our content management tool did not have any version control. It took more than a year to upgrade to the release that offered this capability, and even then it didn't enforce version control. We had to constantly police the process to ensure that developers version their code. The software on which our Java applications were based had complex configuration parameters we didn't fully understand when we first put it into production. We had tested our intelliTRIP product with a production load in terms of transactions per second, but never with a realistic number of concurrent users. As a result, the servers kept crashing on the first day we put it in production. If we had understood the configuration and the user session management parameters better in our Java-application management tool better, we could have prevented this problem

II. Define Processes
Collaborate with your counterparts to formalize your processes:

• Get control of the production environments. Work with your information systems team to create a production update procedure. When I started, developers launched their own changes to production. It was hard to wean them away from this bad habit. Even after we thought we had implemented good production update procedures, we lacked the discipline to enforce it under pressure. For example, since we did not have good configuration management, it became impossible to build baselines of intelliTRIP, so developers would simply move new classes into production to fix problems. Only?after two years did ?we begin to be able to require developers to build scripts and installation documentation before we accepted any software from development for testing.
• Get involved from the beginning of each project. This is hard work. It forces you to juggle many tasks, but it is essential. Participate in all documentation reviews: Those for requirements, functional specifications, and design specifications. Make sure the documents are complete and clear. Look for ambiguities, gaps, lack of detail. All parties, including marketing, development, test, customer support, sales must agree on a vision for the product. This vision is a short phrase that describes the main thrust of the product. With intelliTRIP, development and test were told to produce a server-side version of the original client-side product as quickly as possible. Sales and marketing believed that the purpose of the product was to quickly locate fares from airline Websites. Since development and test wasn't told that part about quickly, we released the product even though we knew that is was sometimes slow to return results. This type of disconnect can be prevented by including a vision statement in the requirements
• Define quality. Work with marketing and product development to define quality IS for each product: Should the priority be good, fast, or cheap? (You can only have two of the three!) Even if you choose fast, don't sacrifice the process. At TRIP.com, we once implemented a promotion that marketing believed to be simple and wanted to rush to production. Since the product manager did not hold documentation reviews and get signoff, the HTML pages produced by the developers had to be changed three times. This took much longer than a documentation review meeting. There is no need to get bogged down in process either. If you find that is happening, change the process, or train people how to use it properly.
• Document the internal processes of both test and development. You can't expect marketing and product development to follow best practices if you don't do it yourself. At TRIP.com, one of the development directors, with the help of the technical writer, led an effort to define and document the development process. By the way–??no matter the size of the company, unless you are using a lightweight technology such as eXtreme Programming, you need at least one experienced, skilled technical writer in your development organization.
• Enforce the process. If your QA team is large enough, dedicate one person to administering and enforcing configuration management and delivery of installation scripts and documentation. At TRIP.com, we expanded this Configuration Manager role to that of a deployment engineer who works closely with developers to produce the builds and installation procedures. Don't accept software to test if it is not accompanied by all the documentation and software you need to promote, test, and launch it to production
• Innovate!Look for new ways to present documentation such as functional and specifications. Web applications require a new approach. You have creative people at your company who can help! Get input from as many different groups as you can. If your company really wants to get innovative, consider lightweight methodologies such as eXtreme Programming (XP). These methodologies can meet the need to get to market quickly, accomodate rapidly changing requirements but still release a high quality product.

Summary–As You Grow
All companies change as they grow beyond the 'startup' size and environment. My team and I endured countless frustrations when fast growth at TRIP.com led to temporary chaos. As your organization grows, educate new employees about project process and quality practices. Listen to them and take advantage of their fresh outlook and new ideas. Take the initiative. If a gap results from a re-organization, fill it yourself. For example, when we were temporarily without a project management function in the company, the development director and I set up a weekly tactical meeting with representatives from all departments so that everyone could stay informed and juggle resources. Quality assurance can be a frustrating job, especially in a Web startup. Pick your battles. Keep striving for better quality. Above all, enjoy the experience!