Innovation from the Tester’s Viewpoint: An Interview with Jason Arbon

Josiah Renaudin: Today I'm joined by Jason Arbon, founder of appdiff.com and a keynote speaker at STAREAST. He'll be speaking on building mobile app quality strategy as well as innovation from the tester's viewpoint. Jason, thank you very much for joining us. First, could you tell us just a bit about your experience in the industry?

Jason Arbon: Yes. So I've been around the block I think, and I feel old... That question makes me feel pretty old, actually. I started as an intern at Microsoft in testing, working on Exchange Server when that was actually cool. I worked on a bunch of different teams at Microsoft. I worked on like BizTalk server, OneFS, Bing, MSN, a bunch of different stuff. I quit there twice. I tried to quit and they kept coming back.

So that was over about six years. And then I joined Google, and... The last team I worked on at was Bing, actually, at Microsoft, and then I went to Google and worked on the Chrome browser. People know that stuff the best. So I worked on the Chrome browser, was a test automation guy for Chrome browser, and I also headed up in ChromeOS, and I also headed up a personalized search team that nobody knows about, working on just Google search for the last two years while I was there. And then after that, I went to utest.com, which I think most people in the testing world know about.

So it's a crowd-sourced testing solution where you have, basically, the world's largest test team on the planet of 100,000 testers and companies give us their apps and then we take their apps and we test them, and give them the bugs back, and charge a little commission in the middle. So that was pretty cool, I did that for about three years. And I left there a few months ago now—it's been about three months—to start my own new company called appdiff.com, and it's trying to automate and speed up testers. So I've gone from an intern, lowly intern, to engineering and management, to product development and now my own little start up.

Josiah Renaudin: And you have both a tutorial and a keynote at STAREAST that's coming up, so I want to start off by focusing a little bit on the tutorial. Can you talk about the differences between native, web, and hybrid apps?

Jason Arbon: Actually, if you see me at the conference, just be very nice because I'm going to be very busy [laughs]. The tutorials are all day and they're two in a row and two talks, so ... So everyone asks this question, which is awesome actually, about the native, web, and hybrid apps, and I can give you the quick answer but the real answer is, all these things are blending, frankly, over time.

Native apps are looking more hybrid, and web apps are looking more native, so we're frankly ... Very simply, web apps are just web pages we're all used to that basically show up and look nice and friendly on an iPhone or an Android phone. The typical web page that scales down and looks pretty. The native apps are basically this generation's version of desktop applications, but nobody wants to admit it. They want to think it's something new. But basically, the idea is you write a bunch of compiled code and it's bundled into, like, a little executable that goes in the app stores and people download the things to their device. So that's a native app, and they look just like the difference between a Mac app and a Windows app look different. The buttons look different, stuff like that. So do the native apps across Android and iOS. They look different. And they're a little faster because they don't have to hit the web every time you click a button, but ... Those are native apps.

And then the hybrid apps are exactly what you'd think, they're just a blend of the two, right? So they try to take the best of both worlds but they usually end up kind of being, like, somewhere in the middle. The good thing about a hybrid app is that it's basically a native app with a web browser stuck inside the middle of it, and so—kind of parasitically a little bit—so you basically take a native app, stuff a web browser in there and then load your web page up inside that native app. The big advantage for that is that if you have a web page, you already pretty much have most of your app developed, first of all. Secondly, if you put it inside of a native app, people can find you, users can find you in the app stores and download you and they have a little bookmark, essentially, on your homescreen. Instead of being a bookmark inside the Safari. So that's basically a web native and a hybrid app.

The funny thing is that this question comes up all the time, and I've heard it for about four years and I think, “No one will ever actually be comfortable with the answer.”

Josiah Renaudin: Yeah, I mean, I write about it and I think every once in a while, I still get a little confused about the differences, but I think it's an important thing to kind of delineate before you get into that sort of discussion.

Jason Arbon: Agreed.

Josiah Renaudin: Now, how important is it to test your app under real-world conditions? And kind of branching off of that, do you think enough teams perform these types of tests?

Jason Arbon: To answer the second part, nobody tests this stuff enough. It's very important, but nobody tests enough. Even the big app companies, I've worked at them, I've worked with a lot of them through Applause and Utest, and frankly the world today is really ... It's getting more and more complex and even if you have the infinite resources, infinite money, and infinite time, people still ship apps that perform very horribly today under real-world conditions. All you have to do is look at the app store, look at the star ratings for even the most popular apps, and they suck, frankly.

There are a lot of issues, a lot of people's complaints on specific devices. So you've got devices, locations, networks, you've now got connected homes, your phone talks to your watch talks to your nest ... And there's a lot of personalization going on, too, so that experience, even if you get all the software perfect, you think you get it perfect, people expect and demand a personalized experience these days and so you almost have this other dimension of how do you make sure that the content and that experience is personalized for that location or particularly for that user? So the test matrix is just exploding every twelve months, it seems to be doubling, and so the really...

Sorry, I'll say there's probably five real ways to tackle that today. But no one really does it well; even the most experienced teams with the most money, the most time, and the most vested interest in getting it right still can't do a very good job. The most effective approaches today are really around beta testing, so if you don't have a beta testing program, even if it's friends and family checking out a build before you go into the app store, you're kind of crazy. It's the best, really good way to get not just feedback on your app, but making sure it works on those devices. Make sure it works in their car. Make sure it works on their home WiFi. And these kind of things...

Another one is the hardware automation. So today ... We're really lucky, but most people don't know about it, is that if you have some test automation sitting around, some automation scripts, you can, through a web page, run that automation on 350 Android devices tomorrow. Like, literally. So you can get a pretty good sample, but most people don't really leverage that infrastructure yet. So you can run on around 400 different devices and get results back in half an hour or an hour. But most people don't even know that exists.

Another one is the crowd-source testing. And that's why I worked at Applause and Utest for a while. The most effective ways to use them for this kind of problem is to ... The cool thing is you can basically rent friends in the Cloud that ... so like, you get ten people a week to play with your app, but they're a different ten people every week, right? They're not the same testers in your lab or the same test team or the same vendor, right? You can actually rotate people through each week to get different devices, and different contexts, week over week over week. That's a great way to kind of get rid of a lot of the bugs and find out how it really works in the wild.

Another one is the user feedback tools. I think this is one people don't really realize, is there is some new tools and SDKs for mobile apps where if people have problems in the wild, there's a movement toward giving them the feedback button and then just asking the app team for help. Right? Like a little call-for-help button. And you can actually engage in a conversation inside the app with your user to get him past that roadblock and also get repro steps and understand what they were trying to accomplish.

And then the last thing, now, I won't... I'll shut up after that…

Josiah Renaudin: No, you're fine!

Jason Arbon: ...is to monitor the app store reviews. This is a thing I've been hounding on forever but people are starting to care now, starting to realize it, but the best source of information on, like, what issues are happening in production in the wild and the cheapest way to do it is to read your app store reviews. People just don't read them all or treat them with enough seriousness. So read those app store reviews, they're basically free bugs and they also tell you what people are trying to do and what they're frustrated with, because testing fundamentally has always been about emulating the end user before the end user gets the software, and the funny thing is today we have end users actually talking to us because of app stores, but people aren't reading the feedback. So...

So those are kind of the five big interesting ways I think to approach that real world testing problem.

Josiah Renaudin: Absolutely. And like you said about reading feedback, as a writer I'm often getting edits from my coworkers or from other people, and that kind of feedback is what you need in order to progress. So to be able to look at those reviews and see those bugs is, like you said, it's free advice. It's a great way to expand upon your application.

Jason Arbon: Exactly, and you want to get that feedback before you get the public comments on the bottom of the article, right?

Josiah Renaudin: Absolutely! That's always important. Those hurt. A lot [laughs]. Now regarding your keynote, can you discuss how to identify low-hanging fruit for innovation?

Jason Arbon: Yeah. So, the easiest way is kind of just like the start-up world, frankly. Everyone says the easiest way is just to look around. Just take a step back and look around at what's going on around you. So, what are you doing? What are your peers doing? What is your team doing? And just look for every place where there's friction, so it's very expensive or it's very slow or it's really annoying, and just take a step back and think about those problems. Usually, people think they're troopers and try to fight through those problems or, you know, grin and bear it, or they're trying to be heroes by slogging through all these things, these hard problems, like a long test pass, or a lot of manual configuration and these kind of things. So the quickest way to identify that low-hanging fruit is just take a step back and just look at where the pain is on your team. People don't usually take that time, they just focus on ploughing forward and just trying to work harder instead of, frankly, kind of smarter. So that's the first thing.

The other is I would say the—we'll talk about it a little later I guess, hopefully, this conference—but the big thing to me is that testing is the most underserved part of computer science and computer engineering and software engineering. So like, everyone's gone off to the glory things of building apps, but they're easy today, there are billions of them, obviously it's pretty easy to build an app today. The hard part is, is that no one knows how to make sure that they work well, and they're of high quality. So the good news is that all the little frustrations we see or on our app team, the low-hanging fruit, can have a very significant impact, because there's been so little innovation frankly in the last ten years in testing and quality. So just look around, honestly, is the best advice.

Josiah Renaudin: Speaking of advice, what tips do you have for presenting new, innovative ideas to your company, and subsequently, if these ideas do fail, how do you fail gracefully?

Jason Arbon: Right. So. I'm not a very graceful person, but I've failed pretty gracefully. So, the first piece is how do you present new innovative ideas? I think people don't realize. I think that they ... If they have the innovative idea, like it's something cool, they want to share it immediately. And that's actually probably not the best idea, because, like, everyone knows, you know, everyone says, "Ideas are a dime a dozen," especially in software, especially if you look at a company like Google or Microsoft or something I've worked in. Ideas are actually the cheapest thing around. Everyone's got an idea.

So fundamentally, I suggest people show, don't tell, frankly. Go off and experiment and build that thing, the best way to keep your failures quiet is to not tell anybody about them. So I spend a lot of time at Starbucks drinking coffee and hacking and experimenting and brainstorming and only, you know, one out of five things that I hack and play with may turn out to be somewhat interesting. And only when I have something to show somebody do I talk about it. People usually think that “oh, that Arbon kid in the corner is usually... almost all his ideas seem to be useful and half-working!” Right? At least half-working. But it's because I've just quietly failed a bunch of times in my own sadness, right? Over a latte. So show, don't tell.

A quick story that might put that into some good context is—this kind of was embarrassing for me, but it's the way I really learned this point—when I started at Google, about three weeks after I joined, I had this cool idea for personalized search, blah blah blah, thought it was a really cool thing. I built a twenty-five, thirty-page SlideDeck and PowerPoint, and then I was brave, you know, and it was Google, so I pinged the head of search, the VP, this guy named Udi Manber, and he's an awesome, sweet, nice, super smart guy.

The first thing he said though ... He took my invite for a meeting, which was awesome, but on slide three he was like, "Hold on, did you work at Microsoft?" And I was like, oh, this is not going to go well ... Uh uh. But he says, "I think this idea has a lot of promise." He says, "I care about it enough that you should do it right" and he says, "you can put the calendar invite anytime on my schedule, whenever you're ready, but take a step back and, you know, go build it!" Get something demonstrable, right? Don't bring back slides but bring back something that either half runs or I've got feedback from a lot of people or something like that. And so, I was both embarrassed and kind of inspired at the same time.

So since then I've really learned that lesson of, you know, show, don't tell, right? And talking is easy but doing is the hard bit. That's what matters.

Josiah Renaudin: You just discussed how you brought up this interesting, innovative idea to someone, and I want to know what can a successful innovation— especially one that you pitched—do for your career as a tester?

Jason Arbon: I guess it seems that my career is going on a good trajectory. On paper it looks like it probably is doing okay. So an innovation that's impacted my career ... A recent example is probably good. I landed at Applause, or it was utest.com back then, a crowd-source testing company where almost all the revenue came in through manual testing of web pages, right? That was the job and that was the deal. So I create a little labs team and actually, not that people don’t know this but, I left Google with four people at the same time to create a little labs team for Utest, up here in Seattle, and our charter was to help the company somehow, but we had eighteen months of runway to figure out what that was going to be.

When I landed, I was more disruptive I think than the CEO and Founder were even expecting and the CTO at the time. They were expecting and assuming our team would work on adding value to web testing and manual testing, right? And instead what I really thought we should be doing is focusing on mobile—this is, you know, three years ago—I think mobile at some point is going to be a big deal, but we're waiting to see. But I saw the transition and those metrics and I knew mobile was going to be huge, and how do you scale things? You scale them usually with data, not just making a human 10 percent more efficient or 20 percent more efficient.

I spent a lot of late nights working on this data analytics problem and analyzing all the app store reviews. But the founder called me at night because he saw I was online and working at ten o'clock at night, and he goes, "What are you doing?" And I just told him, "Don't worry about it, I'm not doing anything, I'll tell you if I have something." But what's interesting is, so this is why I took the risk, right? You take that risk for innovation and time and some little bit of career risk, but I finally got something and some results. Within a year and a half, a few things happened.

Once we got good data and people thought it could be useful, turned it into a product so we've been reselling that, IBM is reselling it, even, IBM is picking up our data and selling it as their own these days, you know, people subscribe to the data for a lot of money, and that data also has come back and helped make the crowd selection. So how we pick people for test cycles at Applause. It's got some magic, mystery sauce now so it picked the right testers for the right projects behind the scenes, algorithmically. And it all came from this common investment. Also my career path, specifically. I got a lot more, you know, kudos, and freedom and flexibility by taking that risk and creating something interesting, but I also moved from being director of engineering to being the director of product. And for the whole company, not just for the labs team.

Within a year and a half, I was running product for the entire company and doing partnerships with other companies and pushing to acquire a couple of companies ... So, I mean, within a year and a half, that little innovation led to just a lot of credibility and a lot of momentum career-wise. And now, whether it's good or bad, I have enough confidence, enough people have enough confidence in me that I'm starting my own company. So that's probably pretty big impact in terms of moments of innovation and how it impacts your career and your job and your job function.

Josiah Renaudin: Absolutely. Now more than anything, what message do you really want to leave with your audience at STAREAST, either your keynote audience or those attending your tutorial?

Jason Arbon: It's pretty simple: I think that software testing and quality is the most underserved thing in the industry, so it's really easy to innovate and it's really easy to add value. I just think for a lot of reasons testers are risk averse, or they're quiet, or they're introspective, and so I just want everybody out of their shell. Just wake up, and realize there's so much opportunity, business opportunity, career opportunity, impact-on-the-world opportunity … just be brave, take the step back and think about how to innovate, and then you can add value. It's really, really easy if you just take a step back and think about it with a little bit of confidence.

Josiah Renaudin: All right. Well, thank you so much, Jason, I appreciate you stopping and talking with us and telling us all these interesting stories, and I'm very interested in hearing more from you at STAREAST.

Jason Arbon: Cool! Thanks! And again, forgive me, I'll be tired and busy, but it should be a good conference.

Josiah Renaudin: Absolutely.

Director of product strategy at Applause.com/uTest.com, Jason Arbon worked previously on Google Chrome, ChromeOS, personalized web search, and Microsoft teams including Bing, Biztalk, WindowsCE/Internet Explorer, and WinFS. A frequent conference speaker on software quality, Jason has also delivered mobile app quality and testing workshops and keynotes at software company internal engineering conferences. He founded herecandy.com, a mobile app company that made web browsing and sharing smarter on mobile devices. Jason co-authored How Google Tests Software and published App Quality: Secrets for Agile App Teams. In his free time … oh, wait, he doesn’t have any.