Exploratory Validation: What We Can Learn from Testing Investment Models

Over the past few years, the airwaves have been flooded with commercials for investment-support software. Do your research with us, they promise, and you can make scads of money in the stock market. How could we test such a product? These products provide several capabilities. For example, they estimate the value or direction of change of individual stocks or the market as a whole, and they suggest trading strategies that tell you whether to buy, hold, or sell. Every valuation rule and every strategy is a feature. We can test the implementation of these features, but the greater risks lie in the accuracy of the underlying models. If you execute the wrong trades perfectly, you will lose money. That's not a useful feature, no matter how well implemented. Cem Kaner reports on work he's been doing in this area, presenting this case study of exploratory, high-volume test automation, done for the purpose of validation rather than verification.