In this interview, STARWEST keynote speaker Bj Rollison talks about the things that really matter in testing. He covers the value of testing certifications, the impact of ISO standards on the industry, and the advance of test automation.
Josiah Renaudin: Today I’m joined by Bj Rollison, a testing mentor and keynote speaker at our upcoming STARWEST conference. His presentation is titled “Things That Really Matter in Testing—Today and Tomorrow.” Bj, thank you very much for joining us.
First, could you tell us a bit about your experience in the industry?
Bj Rollison: My career in the computer industry started in 1992 working for a startup in Japan, building custom systems from the motherboard up. In 1994, I landed a position on Microsoft’s Windows 95 International team as a test lead. Over the last twenty years I worked on several products, and also spent about eight years in Microsoft’s Engineering Excellence group teaching and driving practices across the divisions. Since leaving Microsoft last year, I have focused on teaching fundamental testing principles and practices to burgeoning developers to hopefully improve their unit tests.
Josiah Renaudin: You mention in the abstract for your keynote that you’ve often seen people in testing get stuck in trends and topics that make good fodder for debate but do very little to enhance personal careers or the testing profession as a whole. Can you provide examples of these trends?
Bj Rollison: Yes, one example is ISO standards. Sometimes, minimum standards are necessary. But generally they are always minimum standards, and I have never seen ISO standards required in the development of consumer software industry. Some companies may require standards to comply with some governmental requirements, but ISO standards have no impact on the majority of testers in the industry.
Josiah Renaudin: Is agile a modern version of this style of trend, or do you feel it has more value?
Bj Rollison: Agile is a wonderful concept for improving efficiency in a business. But every successful business constantly strives to improve efficiency. Another such concept was poka yoke, or mistake proofing. It was a wonderful concept for an assembly line but didn’t quite catch on in the software industry.
The point is that some concepts for improving efficiency in one business sector may not apply to other types of businesses. The practices necessary to build a successful operating system usually far exceed those of building a game for a smart phone.
Josiah Renaudin: What is your definition of the “traditional” testing role, and how has it evolved over time?
Bj Rollison: The concept of a “traditional” testing role is a complete farce. The traditional testing role was created by very senior and experienced developers such as Boris Beizer, Glenford Myers, and Jerry Weinberg. When I was hired at Microsoft, all testers were expected to have an understanding of operating systems and programming languages. Was that the “traditional" testing role?
In the mid- to late ‘90s, Microsoft and other companies bought into the concept of hiring super-users and customer-like users as testers to find bugs. I suspect many people consider this as the “traditional” testing role—especially since the industry once again seems to be requiring testers with higher technical skills and greater competency who can provide more value throughout the entire development process. The key is to forget “traditional” roles. The industry changes very quickly. If you get mired down in some abstract notion of a “traditional" role, you may soon find that your value to your company or project becomes obsolete and unnecessary.
Josiah Renaudin: How has test automation affected this industry? For a standard organization, what should the split be between automation and manual testing?
Bj Rollison: First, there is no such thing as a “standard organization.” Secondly, anyone who suggests some magical ratio between automated tests and manual tests is a charlatan or snake oil salesman. Test automation is largely misunderstood. Some people do not or cannot differentiate between automated tests and macros. Also, a lot of time is wasted on overly simplistic tests automated just for the sake of automating a test. This is especially true of automated tests that attempt to emulate customer interactions, or GUI automation. Well-designed automated tests with specific purpose can be very helpful in a project, and can also free up a tester's time to test and find ways to improve their product.
Josiah Renaudin: How does exploratory testing affect a software tester’s role?
Bj Rollison: All testing is inherently exploratory in nature. The tester’s role has always been to explore the software to evaluate its capabilities and to expose potential flaws. The more a tester understands the software and the platform it is built upon, and about the targeted customers of that software, the more effective that tester will be. Testers must continually grow and learn.
Josiah Renaudin: More than anything, what message do you want your audience to walk away with at STARWEST?
Bj Rollison: There are some talks at conferences that are intended to convince you of something: a cool tool, a new process, or an ideology. Many times the presentations are too narrowly focused and the information is presented in a way that worked for the presenter’s project. That’s OK, but I don’t think that is why most people attend a conference. Conference attendees want to learn something new and learn how they can apply it on their project.
Some concepts transfer across product lines very easily; some are harder, impossible, or unnecessary to adapt between different products. The advice I would give to the audience is to not look for shrink-wrapped ideas, tools, or processes that you will take back to your project and change the world. Instead, open your mind to new ideas. The speakers will tell you a story of their experience. It is likely not exactly the same as your experience. When I attend a conference, I always ask myself, What can I learn from this person’s experience, and how can I apply it later to improve myself, my team, and my project?
Bj Rollison has more than twenty-five years of experience in the software industry. His twenty-year career at Microsoft began in 1994 on the Windows 95 international team and ended—several projects later—on the Windows Phone team. At Microsoft he held several leadership and individual contributor positions. But, the most rewarding was as a Test Architect in Microsoft’s Engineering Excellence group where he designed and delivered hands-on training to Microsoft’s 6,000 test engineers. Bj now teaches software testing and test automation practices at University of Washington and Duy Tan University in Vietnam. Bj enjoys sailing, stamp collecting, gardening, and spending time with his family.
User Comments
BJ, love your take on "traditional" testing. To me it's a common term thrown around as a means of identifying the role of testers in an organization usually by people with no idea of what that role really is. It's an easy thing to fall back on and not put the work into identifying what the organization really needs.
"and I have never seen ISO standards required in the development of consumer software industry"
Just because you have never seen it doesn't mean it doesn't happen. In fact we've seen this happen in government contracting where the idea of 'CMMI Rating' became something that people lobby into contracts to reduce competition.
ISO has a pretty good reputation with some of its standards industries around the world use them and strive to include them in some way. The danger is that when you talk about minimal standards as BJ mentions, you run into the problem that you see on many of these home improvement shows. Homes which may be 'up to code' but code may not be strong enough, or in some cases, work being done which may look like it passes code until you dig below the facade. Software development is no different in this respect.
I believe BJ referenced the government as a caveat. <br><br>
Though I do see your point about “minimal” standards. I find with the age of Agile, organizations confuse light standards with no or inadequate standards and this includes what they decide to document. As a traveling consultant with 26 years’ experience, I have seen a rapid depreciation of standards and the rise of poor testing practices masquerading as the "agile' process. You need only move outside of traditional software companies to see this trend first-hand.
Agree with the point that Automation versus Manual testing is always a grey area for top management in almost all the organizations. When we talk about automation, management always thinks about cost cutting and resource cutting. In my experience of 15+ years in Software indistry, I hardly see a relevant and required automation test suits which can reduce the testing time and improve productivity.