Balancing Mobile Application Quality through Proper Testing: An Interview with Jeff Payne

Josiah Renaudin: Today I'm joined by Jeff Payne, the CEO and founder of Coveros and a keynote speaker at our upcoming Mobile Dev+Test Conference. He'll be speaking on mobile application quality. Jeff, thank you very much for joining us.

Jeff Payne: Thank you for having me.

Josiah Renaudin: No problem at all. First, could you tell us just a bit about your experience in the industry?

Jeff Payne: Sure. At Coveros, we build security-critical applications using agile methods. These days, many, many critical applications are using or have mobile application interfaces. Everything from banks to e-commerce sites to critical infrastructure, et cetera, et cetera, et cetera. It seems like pretty much these days if you don't have a mobile interface for your app or you're not building apps specifically for the mobile space, you're behind.

Josiah Renaudin: I want to talk a lot about your keynote today. How much more complex has application development become with the advent of mobile platforms? Compared to developing on standard PCs, what new intricacies has mobile development brought to the table?

Jeff Payne: Sure, it's very complex. It's a fascinating topic because of that complexity. Some of the areas that we're seeing people are struggling with, both from a development and a testing perspective, are first compatibility and interoperability. There are so many devices out there, and because of the business model for these devices and the operating system and the applications, they're constantly being updated by the developer of the device or the OS or the apps you're using. It's very difficult to put out an application that's going to work on many, many, many different devices and across many, many different versions of those devices. Compatibility and interoperability is huge. That's one area that's big.

A second one is usability, particularly when you're talking about your phone. You're talking about a tiny, tiny little interface, right? When mobile devices first came out, mobile phones first came out, a lot of companies jumped in and built web-based interfaces for their phone, for their app, so that you could look at their website, say for instance on your phone. But it was so tiny nobody could read it. People began to realize that, in order for these devices to be productive, really need to concentrate on usability and user experience, and figure out how to build user interfaces to them that are actually usable. Those two areas, along with performance and security, are just really big in the mobile space.

Josiah Renaudin: It's absolutely more complex, but we've also seen a lot of improvement in the development area. How much has mobile testing improved since we first introduced to smartphones? What strides have we taken to guarantee more consistent quality throughout?

Jeff Payne: Sure, that's a great question. First of all, initially, mobile testing, like any kind of testing on a new environment, was pretty, I'll call it, ad hoc. A lot of clicking around and just trying to make sure that things worked from a user's perspective.

Now you're seeing a lot more structured testing. You're seeing a lot more automation. You're seeing a lot more of what I'll call rigorous exploratory testing, so informal or exploratory types of testing but that are done with rigor associated with them. Very much a better understanding of the importance of the user experience, and testing not only for features and functions, but for usability and how good is the functionality of this thing in terms of usability.

All those things are hugely important, as well as trying to figure out how you deal with the fact that this software that you're building and you're testing sits inside of a device, and ultimately it's got to work in that device. How do you test that, right? How much of the testing do you do outside of the device? How much do you do inside of the device? What kind of a testing process do you follow to make sure from an end-to-end perspective the software works?

Josiah Renaudin: You just talked about this when you were discussing the improvements with mobile testing, but can you also talk a little bit more in detail about some of the challenges or problems that mobile devices bring when it comes to testing?

Jeff Payne: Yes. Tools is one. There's not great tools out there yet. You're starting to see some tools out there that are interesting and useful. Some of the existing security tools and quality tools are starting to incorporate or provide mechanisms for you to do more automated testing on the mobile device itself.

We're also seeing a lot of what I'll call cloud-based testing solutions, so a combination of either putting your application out there and letting the testing community test it as a crowd-sourced model, or there's cloud-based emulators that emulate various platforms that you can run and actually automate tests for so that you don't have to have all those emulators yourself for various environments.

There's also remote device farms, if you will, of devices and configurations that are set up that you can access over the cloud to actually test physically on various devices that you might not own yourself. All of those are really helping the testing direction that we're going in terms of mobile.

Josiah Renaudin: Absolutely. Mobile's growing. It's growing all the time. We're seeing more devices, more OSs. There are plenty of devices that mobile developers and testers need to keep in mind, but there's only so many resources that you can allocate to those different devices. How could a team intelligently decide which resources go where to appease so many different platforms and so many different operating systems?

Jeff Payne: I can't tell you that. Nobody will come to the keynote [laughs].

Josiah Renaudin: Give me a hint. Give me a hint here.

Jeff Payne: No, seriously, that is one of the biggest problems. Because of the complexity and the various platforms and the issues around usability and performance and security, all of those are challenges in the mobile space that are, to some extent, go above and beyond what we've seen in traditional applications. That means that for a fixed testing budget, you're going to have to really think through what to do.

We found a couple of things that would be useful. One is the tried and true risk assessment approach, right? Trying to figure out, based on what your application does and the types of users you have and the types of risks that are associated with it and what might happen when it fails, what are the various types of testing that you're going to want to concentrate on more? For instance, if your application is an online banking app, obviously security is going to be of utmost importance. If it's Angry Birds, security is probably not something that you're going to spend a lot of time with.

Josiah Renaudin: Probably not.

Jeff Payne: Yeah, one is risk management or risk assessment. The second one though that I'm pretty excited about is what we're calling persona-based testing, which is thinking about the user and the user experience associated with that device and figuring out, given the users that I have and given the different platforms that are out there, where do I want to place my bets? I can only test so far and so often, and I can only test certain use cases, if you will, and against certain uses of the product.

Can I define those and pick the top five, say, and build a matrix of devices and environments that are supported by those users and most typically used by those users and the various types of testing that are going to have to be important from a risk perspective and drive my testing both by risk and by the types of people who I'm trying to attract to my application and I want to use my application?

Josiah Renaudin: Now I won't ask you to spoil too much more of your keynote, but I do want to know more about what you see for the future of testing. What about the current state and even possibly the future of mobile testing has you excited? Are you seeing any new techniques or breakthroughs that are streamlining the process?

Jeff Payne: Yes, one risk we haven't talked a lot about that's huge in the mobile space is security. It's real simple. We've got these devices, whether they're phones or tablets or phablets or whatever they are today, that you're using, and unfortunately they're not very safe. They're not very secure. We leave them laying around so there's physical risk that they'll disappear.

We download applications all the time from app stores to them that may include malicious code. If anybody believes that Apple or Google are adequately assessing the security of all these applications that are showing up, they're fooling themselves. Some of them are malicious and they're sitting there on your phone, right next to your banking app or some other critical app that you have.

We are seeing a lot of work being done on trying to figure out how to analyze applications, whether they're ones you're building, for your own safety, or ones you're downloading onto your phone, to make sure that they're secure and that you're not taking undue risk associated with building and deploying and downloading applications onto your phone that sit there with your other critical apps. That's an area I'm really excited about.

I mentioned cloud-based testing before. I think there's a lot of promise in that regard to help lessen the burden of dealing with multiple devices and configurations and operating systems, things like that, without having to go buy all that stuff yourself.

I guess the last area is automation tools are getting more sophisticated. Emulators and other things that help you test outside of the environments, the device environments, are getting more sophisticated. That means that automated testing is getting more sophisticated which is great, because that can help us deal with some of the complexity that we have in these devices and testing these devices.

Josiah Renaudin: Absolutely. Now, just in brief, what overarching message do you really want to leave with your audience in San Diego?

Jeff Payne: Sure. The overarching message is I really want people to understand that this is not easy, and a traditional testing approach, where you just focus on requirements and test the functionality of the device, is not going to work in this space. You have got to look at the non-functional requirements, performance, security, usability, configuration, and compatibility. Those types of overarching quality principles are going to be as important, if not more important, than just the functionality of your app. You've got to allocate time and attention to those or you're not going to be very successful selling your application.

Josiah Renaudin: Fantastic. Jeff, thank you very much for stopping by and speaking with me today. I look forward to hear more about mobile application quality at your keynote in San Diego.

Jeff Payne: Thank you very much, Josiah. I appreciate it.

Jeffery Payne is CEO and founder of Coveros, Inc., a software company that builds secure software applications using agile methods. Since its inception in 2008, Coveros has become a market leader in secure agile principles and was recognized by Inc. magazine as one of the fastest growing private US companies. Prior to founding Coveros, Jeffery was chairman of the board, CEO, and cofounder of Cigital, Inc., a market leader in software security consulting. Jeffery has published more than thirty papers on software development and testing, and testified before Congress on issues of national importance, including intellectual property rights, cyber terrorism, and software quality. Follow Jeffery on Twitter @jefferyepayne.