Using the Cloud to Build the Perfect Mobile Tester: An Interview with Rachel Obstler

Rachel Obstler will be presenting a presentation titled "Mobile Testing in the Cloud" at STAREAST 2014, which will take place May 4–9, 2014.

About "Mobile Testing in the Cloud":
Today, organizations are rapidly deploying mobile versions of their customer-facing and internal applications. With the prevalence of more agile-based approaches and the challenge of an ever-increasing diversity of devices and OS versions, testers are being asked to accomplish more testing in less time. Rachel Obstler shares how leading enterprises are improving the efficiency of their mobile testing using automation, and how they identify the right processes and tools for the job. Sharing some fascinating statistics from their recent mobile quality survey of more than 69,000 mobile app developers and QA organizations in the top US enterprises, Rachel dives into the challenges identified in the survey and shares five clear ways to improve your testing process: implementing a collaborative agile process, optimizing with a development tool that naturally facilitates testing, using a combination of real and emulated devices—and when to use them, and more.

Cameron Philipp-Edmonds: Today, we have Rachel Obstler. She will be speaking at STAREAST 2014, which is May 4 through May 9, and she is giving a presentation titled "Mobile Testing in the Cloud." Rachel is the senior director for Keynote's DeviceAnywhere platform. Rachel has more than ten years of experience in the wireless industry working for companies ranging in size from startups to multinational corporations.

She most recently managed the wireless data quality products at Telephia Inc. Rachel spent three years working with Lucent Technologies, where she managed and launched a billion-dollar product line of CDMA-based stations. Anything to add to that?

Rachel Obstler: No, I think you covered it.

Cameron: OK, sounds good. Since you're doing a session titled "Mobile Testing in the Cloud," I'd like to ask you about improving the efficiency of mobile testing. Why is the testing of mobile more important than ever before?

Rachel: Obviously, over the last couple of years mobile has increased in importance. We call it the mobile imperative. What we're seeing out there is not only our people demanding access via mobile, but they are demanding quality access via mobile. I'll throw a couple stats.

What we're seeing is that 80 percent to 90 percent of apps were actually deleted from users' phones last year. If you don't like an app, you get rid of it. People are saying that apps should load within three seconds. I've also seen stats that say apps should load within two seconds. So their expectations of how quickly you want to have an app available to you are really drastically changing.

Another stat: Only 16 percent of people will try an app more than once if it doesn't work the first time. So that's the bad side, the side that if it's not working well, people won't come back. The good side: There's stats like two-thirds of users are more likely to buy if they have a good experience.

It's ever more important that you provide, again, access for your mobile, and not just access, but quality access.

Cameron: The consumers are more strict and stringent with their demands when it comes to mobile than they are with other platforms.

Rachel: We are seeing that somewhat. It's changing a bit. We're going to talk a bit about a survey that we did but we see that at least in QA groups, the expectation of mobile quality is actually higher even though they have the same time to test and the same or less resources to do it with.

Cameron: How has cloud affected the ability to test mobile?

Rachel: Cloud gives you a lot of options, and one thing I'll talk about in my presentation as well at STAREAST is not only can you use a cloud to do things like host data, you can actually use the cloud to do things like host devices. You can have devices available to your testing groups in the cloud.

What that means is that you don't have to have a device or thirty devices or ten devices sitting at everyone's desk. You can enable all of your employees, and a lot of our QA teams today are located all around the world, so you're not going to have just people in one office. You're going to have people in multiple different offices that all need access to devices.

What it allows you to do is give them access to devices from wherever they are and not have to keep the headache of managing sets of devices for every employee.

Cameron: Your tutorial also mentions five clear ways to improve the testing process. Can you identify just one and kind of elaborate briefly on why it's important for improving the process?

Rachel: Sure. Using cloud devices is one. We've already talked about that, so maybe I'll talk about test automation. Getting back again to this idea of the expectations of quality being higher for mobile, but the testing time not getting any longer and the resources not getting any larger. What that means is it puts incredible pressure on the testing team to be able to deliver quickly and with quality.

When you have that type of pressure and testing, a really good way to help that is to automate as much as you can. And especially automating the more repetitive tests.

Cameron: I'm so glad you brought up test automation because that kind of leads into my next question here. Your tutorial speaks to the ability to improve efficiency of mobile testing through automation. So does manual testing still have a place in improving efficiency?

Rachel: Yeah, I would say it in another way, which is you can also improve the efficiency of your manual testing. Obviously, all testing has to be done either by a machine or automated or manually. And there's always going to be a place for manual testing. It doesn't make sense typically to automate testing of maybe a brand-new feature, right, so a first timer using it.

Or things like exploratory testing, when you're trying to break something or you're trying to really act like a user would. These are not things that typically you want to automate. You want a real user performing them to the extent that you can make that testing, maybe through efficient access to devices in the cloud and very available for people, then you help them make their manual testing more efficient.

Cameron: Your company did a mobile quality survey, and you mentioned it briefly. Can you tell us a little about the survey?

Rachel: Sure, I'll just use some of the highlights of the survey. We had about sixteen hundred people respond to the survey, most of whom were QA professionals. We also had people from development, IT—but the same type of organizations. What we saw, as I mentioned earlier, is that the quality expectations of mobile apps were actually higher, but the amount of time that organizations got to do the testing and the resources that they got were not any higher—and, in some cases, lower.

We saw that, interestingly, many QA groups are still organized individually, so there will be many QA groups across an organization, but that tool decisions are more often made centrally. So there may be a central tools group or a central organization that's thinking about tools for many different QA groups. That can always lead to friction when, say, the needs of an individual or a group are not necessarily being met by the centralized tools division.

We also saw that companies are still doing the majority of testing for mobile testing on real devices, but that—and as a result, maybe—we also saw that access to a variety of devices was the number one important feature for functional testing. Then lastly, we also saw that our respondents had not done a huge amount of automation in the mobile space so far. Many of them have started, had done a small amount of automation, but not a large amount.

Cameron: If they were doing test automation, how efficient was it?

Rachel: We did not exactly ask that question, so it's a tough one to answer. I can speak for some of our customers, because we do provide products like that, that they find that automating even, like, say, a really simple bill acceptance test can actually improve your ability to get back to your development team and when the build is accepted from—we saw someone get it from two weeks to essentially a day. So automation can make a big difference.

Cameron: You have five clear ways to improve the testing process, but what is really the main takeaway?

Rachel: I think I'd like them to go away—especially if they're not as familiar with mobile testing yet, a lot of companies are bringing this in to different groups and it's becoming something that more and more QA groups have to start taking care of—that they come away with some clear thoughts or ideas of how to think about tooling for mobile: what are the aspects of tooling, what do they need to think about when they're trying to choose the appropriate tool for their organization.

Cameron: Now, let's get to some more general questions. You have over a decade of experience in the wireless industry, and a lot has happened in ten years. I think back to my first phone—it was a brick. What do you think will happen in the next ten years?

Rachel: It's really such a hard question to answer, but the one thing that I'm pretty sure about is that the mobile—and again, it doesn't need to be a phone, it could be a tablet—but anything that's mobile is truly going to become the center of the communication world.

Now, you still see a majority of revenue being generated on desktop, not necessarily yet on mobile. You still see traffic's pretty even at this point. I think it's pretty close to being even, but I think we're really going to see the switch or you're going to see the majority of not only traffic, but purchases such as all interactions, social interaction, et cetera, really happening and the mobile device being the center for that.

Cameron: And do you have a mobile device that you're kind of partial to?

Rachel: I use an iPhone. I've gone back and forth since I'm in the industry of also trying an Android device, but I have an iPad and an iPhone right now.

Cameron: Hypothetically, let's say you have to hire the perfect mobile tester. What attributes does he or she have?

Rachel: I think one of the most important things is either an affinity for or knowledge of different mobile devices. When you're testing on a mobile device, especially if you're doing something like writing automation on a mobile device, it's really important to understand how they act.

You're writing a test and then in the middle of a test, a message pops up. That happens on mobile devices all the time. It could be anything. It could be someone sending a text by accident to that phone. It could be a software update that's now available. Really understanding how phones work and kind of those idiosyncrasies of phones is really helpful when you want to create a really good testing system for automation as well as manual testing.

Cameron: What are some attributes that would be detrimental for a mobile tester—that may be good for a regular tester but maybe not so much for a mobile tester?

Rachel: A lot of it depends on how you're doing your mobile testing. I know someone who's in QA who said that after testing all day on a mobile device, literally a device that they held in their hands, that they couldn't see. They actually lost their vision. It happens when you go skiing, you can get snow blind. Kind of the same idea with mobile devices—if you're going to test on the actual device, the screen is really small and it's actually not that easy to see it. So it's a bit of a challenge.

it's challenging sometimes to test on something with a much smaller screen. But I think what that leads to, again, is trying to find a platform where you can see it on your computer instead of in your hands, and that makes it a lot easier to use.

Cameron: Is there anything else you'd like to say to the attendees of STAREAST and the attendees of your presentation before they come to the conference?

Rachel: Sure. I'd like them to come to my session, which is going to be Thursday, May 8, at 1:30 p.m., and it's called "Mobile Testing in the Cloud."

Cameron: All right, thank you so much. Once again, this is Rachel Obstler and she'll be speaking at STAREAST 2014, which is May 4 through May 9. Thank you so much, Rachel.

Rachel: Thank you, Cameron.

Rachel Obstler is the senior director for Keynote's DeviceAnywhere platform. Rachel has more than ten years of experience in the wireless industry, working for companies ranging in size from startups to multinational corporations. She most recently managed the wireless data quality products at Telephia Inc. and has experience managing the products of various wireless technology companies, including Metawave Communication (smart antenna technology) and Enuvis (location technology). Rachel spent three years working with Lucent Technologies, where she managed and launched a billion-dollar product line of CDMA Base Stations.