STAREAST 2015 Interview with Jason Arbon on Innovation from the Tester’s Viewpoint

Jennifer Bonine: Hi, and welcome to the first of our virtual interviews for STAREAST. Thanks to all of you for joining us today and being here. Some of you may have been watching this morning and saw Jason do his keynote. We actually now have a chance to talk a little more with Jason about a few of the topics that are maybe top of mind for the folks out there watching that want more information.

Jason, let's talk about, first of all, you're the CEO of a company called AppDiff. Explain a little more to us what that is, just for people who may not have heard of it yet or want more information on what that is.

Jason Arbon: I'm happy to, and I'll give you money for marketing. Basically, AppDiff is rethinking how we do test automation today, particularly for mobile. The idea is that most of testing activity is geared toward finding what's different in applications and then deciding whether it's a bug or a feature. The idea is to get rid of that first part of that problem, where we spend 90 percent of our time hunting down the changes we have.

Jennifer Bonine: What changed.

Jason Arbon: Exactly. So, push that burden back onto computing resources on the cloud, let the cloud figure that out for you, and then humans sit back and they don't do the busy work anymore and they get to make the high-level, human-level kind of intelligent decisions as to whether what's a bug or a feature.

Jennifer Bonine: Yeah. It's like when we used to have, when you think about it, old simple technology. DocDiff, you know, when you're looking at two documents and you want to know what's changed and quickly identify versus having to manually go through all of that and look at it. I mean, this was awesome.

Jason Arbon: Yep. Read one, then read the other one and then try to figure out what's different.

Jennifer Bonine: This is great. The concept of that for your mobile.

Jason Arbon: Yeah, it's not an innovative concept, it's just no one's really built it yet.

Jennifer Bonine: No, it's awesome. I think it's great. And then we know today, for those out there looking at mobile … It's been around a while, but people are exploring it at different paces. If I'm out there and I'm struggling with "How do I start thinking about my mobile strategy?" When you work with people, how do you tell them to get started on their strategy and how to think about it? Because there's lots of way you can go.

Jason Arbon: Yep. For folks that want to start thinking about mobile strategy, first of all, get a phone. It's amazing how many people come talk to me and they're like, "How do I get started on my mobile strategy?" and they're carrying like a BlackBerry or they have a flip phone. I say, "Get a phone."

Jennifer Bonine: A smartphone.

Jason Arbon: Second step thing is to really look and see, keep track of what's going on in the app stores. Download the apps, play with them, see what the competition is doing. Read the app store reviews because that's the voice of the user. That's a good indication of what the testers need to be focused on because testing is traditionally about defending the user from bad development, bad code, bad design. A lot of times, the most important thing is not just to start testing the app, but to understand what users want in the app. The best way to do that today for free is to read reviews on app stores. So when you download Angry Birds, take fifteen to twenty minutes and read the app store reviews. It's a good way to get an introduction.

Jennifer Bonine: What do you guys say today in terms of your attention span for an app? Whereas in the past, you could have software that didn't work great, but people were more patient, you could say, heck with it. But with a mobile app, we're finding, I think, that people aren't as patient about it working and being what they want. About how long before someone decides, “I'm deleting it off my phone”?

Jason Arbon: That's an excellent point. IBM has done some research recently—it's half marketing money but it's half real—is that people will dismiss, people will judge within the first thirty seconds. Often it's actually even less. That first experience has got to be awesome. It's not good if it's a crashing experience, but more importantly, I think it's not just about mobile. It's, first of all, people are like in line at Starbucks when they're evaluating their app and checking out your new app—they're entertaining themselves for a few seconds.

Jennifer Bonine: They're a distraction, there's multiple things ...

Jason Arbon: And they don't have a lot of time. And secondly, the traditional web desktop software, you were sitting down. So people would take the time to ramp and learn on it, but on mobile, they're just on the go and in the real world. There's just not the amount of time, and there's not the patience, so again, quality is critical, especially in those first on-boarding experiences for an application.

Jennifer Bonine: Yeah, because you've got to get it right. Otherwise, people, they'll lose interest, they'll get bored, they move on.

Jason Arbon: If you're a banking app, they might consider switching to a different bank.

Jennifer Bonine: Yeah, because there isn't that same loyalty anymore because of that. There's so much abundance of technology out there that people are very willing to switch if they're not getting what they need out of the apps.

Jason Arbon: Absolutely.

Jennifer Bonine: And change.

Jason Arbon: That's what I'm hearing a lot from companies that care about testing and quality, is that they're really trying to care about quality because they realize that the mobile, the mobile app particularly, is their business now. A lot of companies are getting most of their traction and money transactions happening through mobile, so they care about it now and they realize that could be the beginning or end of the next phase of their business.

Jennifer Bonine: Well, and if you get it wrong, if you're publishing inaccurate pricing, if you put bad data out there, it's instantly available to the world. It's not like it's a limited scope of release and if someone doesn't like it, with all the ways you have to publicly voice that opinion through cyberspace, people know instantly versus word of mouth.

Jason Arbon: Right. They can post a review in five seconds. And even if your app is confusing, even if it's correct and it's confusing and it is misleading or is problematic, people will instantly be talking about it. And you can ... a lot of times app teams say, “Well, those users are wrong.” Right? The thing is in mobile, there is not a wrong user, because they all have a voice.

Jennifer Bonine: Right. And they have that independence in terms of how they're looking at it. One of the other things too, now that we're talking about apps is, there's a lot of data collected out there on all of us as consumers. There's this abundance of data around what we're looking at, how much time we're spending on a site, what we're focused on, what we buy, what we get rid of, what we delete. How do I know, as a tester out there, I could get overwhelmed with how much there is? How do you help people learn how to focus their energies and what data to pay attention to and what to look at?

Jason Arbon: No idea. You're just screwed.

Jennifer Bonine: The bad news.

Jason Arbon: The good thing is there is a lot of data. That's the good news and the bad news. The reality is, it's usually pretty easy if testers talk to their product managers or their developers to ask what instrumentation they've put inside the app, and then again, a lot of those companies—a couple of examples are Flurry, there's Google Analytics, there's a couple of other ones like AppScene—it lets you see what people are moving within your app. But they visualize this data, too, so you don't have to live your life in a spreadsheet anymore or a CSV file.

There's a lot of visualization technology now. Particularly in mobile. Again, I think Appsee is a great one—they don't even owe me. And again, Flurry and stuff gives you good insight. Testers need to ask the product manager for access to it and then take fifteen to twenty minutes to look around. That's usually all it takes. Usually, the big issues with testers and data is that they either don't know about it or they're too intimidated to look at it. But it's actually pretty easy if you get into it.

Jennifer Bonine: Okay. What would you say for the folks out there watching that maybe their companies are just embarking on this mobile strategy, they've never done mobile before, they're trying to figure out the best ways to get information, how to adapt or change what they've been doing in the past because maybe they're been traditionally been working on other types of more stagnant applications that don't change as often? They haven't had to worry about mobile. What resources would you say to take a look at in terms of helping get them up to speed in this space?

Jason Arbon: The book I wrote on app quality.

Jennifer Bonine: There you go.

Jason Arbon: But literally, that's why I wrote it is to share information. I would say the most, not even a resource, but a quick rule of thumb is that everybody that's transitioning from web to mobile will first build a responsive mobile web site, and then they'll go, "Oh, it's really easy to build a hybrid app." Where you put the website inside the native app, if any company is serious about mobile, it always happens, they end up running a native app. Even Facebook went through this transition a few years ago and even companies for internal apps are doing the same migration. At every step it's easy, it's easy in a way, but if you really want to get aggressive and smart about it, if you're serious about mobile, I would suggest that people just leapfrog and go straight to native app development because you're going to get there anyway.

Jennifer Bonine: Skip that interim because you'll go there.

Jason Arbon: And if you don't think you'll get there, then maybe don't bother with those intermediate steps. That's controversial and I'll lose half my friends,but that's the reality.

Jennifer Bonine: Your suggestion, go straight to native.

Jason Arbon: Yep, because everybody goes to that.

Jennifer Bonine: Because eventually, that's where you're going to end up, so just make the leap there.

Jason Arbon: In addition, in terms of resources, also another thing, a lot of the smart teams and pretty good teams who are resourced well, they'll want to build their own lab. Don't do it. I've done it ... there's actually an article on TechCrunch that talked about my lab. I built it at Utest with Jeff Carollo—he did all the work, but the batteries explode, you can't keep up with the new devices, the idea is, don't build your own lab because you'll end up shutting it down. Go straight to cloud. Xamaran's got a good cloud, there's AppThwack and Testdroid. There's just no reason not to use these public cloud labs.

Jennifer Bonine: Instead of building your own because ...

Jason Arbon: Yep, it's a lot of money and time and ...

Jennifer Bonine: It is. And a lot of people struggle with that. I need all the devices, I need the lab, and then the problem is, to your point, it just magnifies what you have to maintain in terms of it. And look how fast it changes.

Jason Arbon: If you look at Xamaran, these companies will have 300 to 2000 devices available at any given time.

Jennifer Bonine: Yes, and they update them and they're available as soon as the updates come out within a couple of hours, they're all updated and correct.

Jason Arbon: Like you said, because of the rapid-fire of mobile, the hardest problem when testing is to focus on what's changing in the app. It's not just building all that infrastructure, so outsource as much of that as you can, offload it to somebody else and focus on your app and your core value in that app.

Jennifer Bonine: Yeah. And use your partners that are good at those other things to do those things and focus on the app.

Jason Arbon: Exactly.

Jennifer Bonine: What are you seeing in terms of, obviously, we know what the ... what it looks like today, but technology changes so quickly, and we have new things entering the market all the time and consumer demand for more and more is coming out. You have a session on Thursday that talks about 2025, which sounds like forever away, but ten years in this space, if we look at when Apple came out with the first smartphone that they had, I mean that wasn't even ten years ago.

Jason Arbon: Right, right.

Jennifer Bonine: You think about the change from then to now, now you say ten years from today, what are you seeing or what do you think we'll see in terms of ten years from now around this space?

Jason Arbon: I think it's actually really brave. Ten years is a perfect window, though, because people will forget in five years what you've said.

Jennifer Bonine: So you're safe.

Jason Arbon: Exactly. The biggest transformation isn't even going to be devices, the size of devices, it's going to be ... the two main things are going to be that everything is going to be connected and aware and that's basically more infinite, uncountably infinite testing space, but the good news is along with that is going to be coming near infinite compute. The compute will become free, they might even give you a Starbucks or a latte to use their infrastructure at some point.

The key thing is focus on how you can leverage those resources the connectivity of people of the world, and assuming they have infinite compute to solve testing problems. What it's really going to look like is we're going to lose UI, I think. I hate to keep referencing him, but he's a genius, James Whittaker has an example about his Jacuzzi. He thinks in five years, his Jacuzzi’s going to talk to all the other Jacuzzis and ask what's the cheapest, best, most effective cleaning fluid for his Jacuzzi. And it will order it from Amazon and it will just show up at his door.

Things will get smart and aware. We've got a lot of software testing, but that means there's a lot of compute going around. I do question, I do have to say, I'm dangerous, but I don't know why James Whittaker is so worried about cleaning his Jacuzzi all the time.

Jennifer Bonine: Is there something underlying there?

Jason Arbon: Is there something we don't want to know?

Jennifer Bonine: That he's always worried about his Jacuzzi.

Jason Arbon: And I'll add that as we reach infinite computer, infinite computing, machine learning will actually become very real and legit. I worked on some of the neural network, testing the neural network stuff at Bing in the past, but they were kind of dumb. But there's enough computing power and like you said, enough data today that these things will become semi-sentient and we look at, the prediction is that by 2025, a machine, an average phone will be about as smart as a dumb monkey. We always joke in testing that there are dumb monkeys or we're building dumb monkeys but in 2025, we're going to wake up and the machine will have the intelligence of a dumb monkey. So where do we fit in that equation? I think we just got to move up the evolutionary chart and use our human intelligence.

Jennifer Bonine: In terms of what we're seeing, more people, we don't go more than a few feet from our phones at all times, except for you ...

Jason Arbon: Except for me today.

Jennifer Bonine: When you left it, that's why I was saying it's ironic if you were watching the keynote and you heard the phone ring, that would be Jason's. So he left his, but they say on average, we're so connected to our devices and it's just a part of us and we're all now expecting besides just a generic device, that it has all of this personalization. I'm even looking on your watch, you have Mickey as the face, but Apple's offering personalization on the face of your watch and you can change it by your mood or the day or whatever you want. What are you seeing in terms of having to look at creating that level of personalization and experience in the apps beside just in the generic experience for the user?

Jason Arbon: Right, exactly. I think that we're very ... we're under-utilizing as an industry and software developers and testers, we're under-utilizing the amount of data these things have. I give an example in the talk today, if you go to Google.com, literally today, you can do it today, everyone thinks Google is super smart, super aware, contextually aware, and if I'm here at the conference and I typed in "D" it'll propose things like, it might not even auto complete to Disneyland, even though we're next door to it, right? Or if I start with an "S" it won't say "STAREAST."

So how contextually aware is even Google today? But we really need to bring those signals of location and usage across apps and the personal history and also driving interest based on social activity. When I Tweet about STAREAST today, guess what? Google should know that I'm Tweeting about it and if I search for conference, that's the first one that shows up. Although it should always be because of good SEO.

But we're just, we're heavily under-utilizing the amount of data and signals that are on these devices. And it's almost trivial to bring those in the devices and make them more intelligent. What will get scary, though, is when these devices are semi-smart monkies and what they're going to do with all that data. And they can think a thousand times faster than us.

Jennifer Bonine: How do you think people are going to feel about that as it seems like it knows more and more about you. Because I talked to people today and even when they go on the web and they're searching, and they know for some reason I've signed up for this conference so in the sidebar, hey SQE ...

Jason Arbon: The ads follow you around the web.

Jennifer Bonine: And the ads follow you around the web and people get a little frightened by that. Now we're talking about things like televisions are going to have cameras built in so they know who's watching them. And they'll personalize commercials. If my four-year-old is watching, she's not going to get a truck commercial to buy a Chevy two-ton truck or whatever, she's going to get the commercials, go to Disney World or go to Disneyland because they know that that's what she's going to want to see and pitch to me. As we get smarter and smarter, and the devices have more and more information about us, how do you think people are going to feel about that intrusion into their, oh my gosh, everyone knows everything about me?

Jason Arbon: I wasn't a big deal at Google, but this conversation happened a lot at Google, too. Do you personalize search or not? I worked on, I lead a search team for personalization there and this was a big issue. You don't want to scare people and when you get it wrong and they really, they think you're dumb when you get it wrong but it could also be ... upsetting.

I think it's a very transitional period we're in, like you were saying earlier, just technology is moving faster and faster because we build tools but then we use those tools to make the next generation tools, so we're exponentially accelerating our development, both in tools and testing technology. What's going to happen is that we just need to adjust ... our adjustment period is usually like a decade to adopt a new technology—we're only given a few years now.

When you look at it today, there's a lot of outcry when these things happen. Then it goes away even quicker and quicker and quicker. Look at even Facebook, when they made dramatic changes to how much data they share, there's outrage for a few weeks, then their usage keeps going up. I think it's all about adding value to the user. If you're adding value, people will chill out and will relax actually. I think it's not a long-term problem.

Jennifer Bonine: Right. Well and I think what people don't realize today is that data's out there. It's just about, to your point, how much are we actually tapping into it and leveraging it. Because the data's there, and it could be being used today, we're just not leveraging all of what is there and what is available.

Jason Arbon: They're just shocked that it's put together by somebody.

Jennifer Bonine: Then one of the other things, too, that I've heard around this space is you know, for example, Apple, if it integrates to other apps that you can download, you now like having one thing to connect through and making your apps open and accessible to connecting to other things. FitBits and FitBit bands, being able to connect to other mobile apps that they allow in, so their technology to connect in to create a single space.

Jason Arbon: I think the key thing with that is giving the user, making sure they're aware and giving them the controls to turn things on and off. I really think, though, in a couple of years, everybody will have everything open. It feels like I'm crazy for saying that, but everything will eventually be open. We let people adjust to the heat of the water, as they're getting cooked but they need to adjust slowly and at their own pace. I think the visibility and the access controls are the key with that.

Jennifer Bonine: Yeah, having the ability to turn it on or off but knowing that it's capable of doing that, because I think some companies go with "Well, it's mine and I don't want to share and I don't want to let other people in. " But I don't know how viable that is as a strategy if you don't allow others to connect in and make it a piece of a greater thing that people use.

Jason Arbon: If you look at today's services, recently, a couple of weeks ago, I wrote to Google. I was building stuff on top of Twitter, Twitter data and we lost a deal with Twitter, to do data exchange and sharing, but we redid the deal again. Even Twitter realizes hey, there's value in sharing that data, personal data with Google and integrate those things. So it's happening again. I think it's just inevitable that everything will be open or people won't use it because they'll get a lot less value out of that.

Jennifer Bonine: Exactly. Coming back to that value proposition, and what does it do. To that point too, not what you do now, but there's talk about crowd testing, so some people may not have heard about this or are just hearing about this now. Explain what you think the value of that is or if you think there's value in that, as a strategy.

Jason Arbon: Yeah, so I ... I always think about it as a known thing, because I was doing it for three or four years.

Jennifer Bonine: But there's a lot of people that don't. I was actually talking to some people the other day and they were like, "Oh, I've heard of crowd sourcing, I've never heard of crowd testing." It's kind of, for some people, it's new. Just like mobile, where we're thinking, "Oh gosh, we've been doing mobile for years," and people are going, "Ooh, we're just finding out about this."

Jason Arbon: I have a tendency to move off projects quickly before they become mainstream, actually.

Jennifer Bonine: Right? You're ahead.

Jason Arbon: For better or worse.

Jennifer Bonine: Right.

Jason Arbon: It'd be smarter if I was in the happy medium. They key thing with crowd testing is that it ... the key value proposition is that it's kind of like Amazon with machine computing. A good example, people like stories viscerally, so I was working on Chrome browser, I was testing the Chrome browser over at Google and we looked at all this set of bugs, we had like … if you look at 100 percent of the bugs that in the Chrome browser that we ended up finding, only about a third roughly were coming from automation. About a third were coming from manual testing, and realize we had a big budget approval, we had a lot of smart people get through the interview loops and they weren't finding all the bugs. This other third that users end up finding, were just leaking out all the time.

We had a good program where we could get that feedback, we were very open so we could get that feedback but crowd testing is a good way to get users in a mass, because you've only got twenty testers or something. How do you get 100 or 1000 people to give you feedback in the real world? And they'll have the real world context. For web apps, they have different accounts and different sites that you may not be able to generate in testing and particularly in mobile, they're in different locations and they have different devices and different available memory.

If your phone goes off, I'm not going to be happy again. That would be rude.

Jennifer Bonine: Another strategy, right? Something to think about if people haven't thought about it or haven't heard about it to help you.

So for those watching that didn't get to see that, that was Dan North who Jason interrupted by his phone going off during his keynote.

Thanks, Jason. Time flies. We're already done. If people want to get a hold of you that have more questions for you, things they didn't get to ask, what's the best way to get a hold of you?

Jason Arbon: The best way to get a hold of me is just send mail to [email protected]. After these events, I usually do get a flood of email and I spend the next few weeks catching up, and I love to meet a lot of new people and talk about new problems.

Jennifer Bonine: That's how you get a hold of Jason after this.

Jason Arbon: Can I say one quick thing?

Jennifer Bonine: Sure. Absolutely.

Jason Arbon: For all the geeks out there, the machines will rise up, they will dominate us, but I'm telling you, testing is harder than development. And guess what these machines need? They need smart testers to give them the tools to figure out if the next version of their machines are better than the last so we'll enable the end of humanity, but there may be a statue for testers in the virtual world of the future when humans are gone.

Jennifer Bonine: There is hope. Thanks, Jason.

Jason Arbon: Cool. Thanks so much.

Jennifer Bonine: Thanks for being here.

Jason Arbon: Thanks for putting up with me and my distractions.

Jennifer Bonine: Absolutely.

Jason Arbon: Thank you.

Jason Arbon is the CEO of AppDiff.com, focused on automagically identifying differences in mobile app UI, UX, and performance. Jason is also the creator of the new mobile web search and discovery app mobo (herecandy.com). Jason was formerly the director of engineering and director of product at Applause.com/uTest.com, where he created the App Store data analytics service and led overall product strategy to deliver crowdtesting to top app teams via more than 100,000 community members. Jason previously held engineering management and innovation roles at Google and Microsoft. He coauthored How Google Tests Software and authored App Quality: Secrets for Agile App Teams.