Create a Mobile App Quality Strategy: An Interview with Jason Arbon

Jennifer Bonine: We are back with another virtual interview at STARWEST. I am so excited. Jason's one of my favorite folks to have on our interviews, so we're lucky he's back again this year. Thanks, Jason, for being here.

Jason Arbon: Thanks, Jennifer. It's good to see you.

Jennifer Bonine: For those of you that don't know out there, why I'm so excited for this is because Jason started a company called Appdiff. If you out there have not heard of Appdiff, you need to listen up and hear what Jason's doing and what his company's doing, because it is so cool. If you have mobile apps that you're running out there for your organizations, this is someone you should be aware of and be talking to. Jason, maybe give us a little overview of Appdiff.

Jason Arbon: Thanks, Jennifer. Yeah. Appdiff was really painful to build, but it's super easy to use. Basically we're tackling the two biggest problems that mobile apps have today, well app teams have. One is measuring the performance of their application. That's really difficult. Most people barely have a handle on that. The other big thing they want to do is speed up their agile cycles, their lean kind of ... that CI and that kind of world. They get more builds than they can deal with.

What we do is we basically build a bunch of little robots that live in the cloud. All you have to do is upload your app and that's it, or you can tell us where it is in the app store and we even work with just that. We do a complete performance analysis of your application and show you where the slowest pages are, where the fastest pages are. We basically spoon feed it to you. There are no charts and graphs. You can get those, but really we just show you the picture of the slowest part of your app to motivate people to actually fix the stuff.

Secondly, we do a basic regression automation suite. What happens is the robots look at everything inside your application. We'll check between ten and thirty thousand data points on average for every build. Then the magic happens when you give it the second build. In the second build it does the same thing, the robots spin up and they run around all through your app. We choose machine learning and fancy things like that to make it look more like a human going through the thing, to click on smart things, not dumb thing. Still not brilliant yet, but they're good. Basically they're a five or seven year old kid. That's what we describe them.

Jennifer Bonine: Yeah. You can have your kids test this stuff. Sometimes they're smarter than adults now with tech.

Jason Arbon: Right. I don't know. I'm comparing it to my children, so it may be.

Jennifer Bonine: Your children are probably advanced.

Jason Arbon: No! I don't know. I don't know. They're advanced in bugging me and stealing my iPhone. Basically the idea is that it runs through the second build the same way and with ten to thirty thousand data points, and then it does the difference. The cool thing is it tells you if anything got slower or faster in your app. It also finds out if anything new's been added. The worst thing as a tester is a new build comes out in the morning and you don't know if there are things to test. If something is missing, it'll also tell you that. It'll catch regressions. They don't know what the right behavior is. I don't pretend that they do. Machines are not taking over the world yet, but they just at least point testers within fifteen minutes of a build to stuff that changed.

Jennifer Bonine: Right, It just gives them that road map, right?

Jason Arbon: Yup.

Jennifer Bonine: It's a road map for the testers as well as the developers to speed up how long it takes them to do that analysis and research.

Jason Arbon: Exactly. The most embarrassing thing as a tester is that a day or two days later you discover the feature that they chucked in.

Jennifer Bonine: Exactly. This way you're going to know. The other cool part about that is a lot of times you get these builds, and you'll the go onto the next build, and you didn't have time to finish testing the old build. Now you're going to know which build the stuff changed in, because it's going to tell you. You're not going to have to go, "Oh. Was that two builds ago or was it three builds ago that this changed?" It's going to tell you.

Jason Arbon: Right. The new developers, basically if you're doing a CI system where we catch those changes in fifteen minutes, the testers can look at a few minutes later. You catch it. They can regress it before they go to lunch. We call it the go-to-lunch scenario. You want to catch the most important stuff before you go to lunch or dinner.

Jennifer Bonine: Yeah. That's amazing. With mobile we all know, like you said, the problems with mobile. Now that we're in this mobile space it used to be with technology we had days for people to find stuff on the web or whatever, but we don't have that time anymore. When we push it out people are finding this stuff right away. We want to be able to get changes out to them as fast as we can, but yet with good quality.

Jason Arbon: Exactly. The tragedy of today ... I get a little dramatic sometimes. The tragedy is that we have all these really smart people doing manual and automated test cases, but guess what they're doing all day because of CI and CD. They're doing the same little, tiny things every day.

Jennifer Bonine: Every day, over and over.

Jason Arbon: Every single button on the menu item has to be checked every day. The bots basically just do the basic stuff so that smart people can actually use their brains and find those hard cases, do the complicated stuff, do you things that are related to the business.

Jennifer Bonine: If you're running on IOS, Android, it's going to run on all of those?

Jason Arbon: Yup. iOS and Android. Yup. All flavors.

Jennifer Bonine: All flavors.

Jason Arbon: Different languages, different network speeds.

Jennifer Bonine: That's one of the things when I talk to people that they have a challenge with too, is even if they have iOS and Android, they've decided their Android look, and feel, and navigation may be different than their iOS navigation. They're struggling, because even if they're running automated tests, they've got two sets of scripts that they're maintaining. It takes a long time for them to go through and check both of those. To your point, they're doing it twice. Their manual testers could be doing it multiple times for both of those, iOS and Android. You're saying those bots will help identify that. They can go in, they're going to pinpoint ...

Jason Arbon: They compare Android versus Android; you have one test suite. The cool thing is you don't even have a test suite for the basic stuff. It just runs. The coolest thing about that, to geek out, test nerd, test nerd, is that there's a new feature in the app. It tests it even if you don't know it's there.

Jennifer Bonine: That's awesome.

Jason Arbon: Traditional regression tests, forget you have to write all the stuff and run ...

Jennifer Bonine: You have to know it's there. This is detecting what's not even there.

Jason Arbon: You have to know it's there to check it. This detects what's even new in the build.

Jennifer Bonine: Wow. ... that you don't even know is there. That's what's good for us as testers. Their fear at night, what keeps them up, is, "What happened that I don't know happened? What's missing?"

Jason Arbon: They're missing something.

Jennifer Bonine: You can't sleep. I know you all feel that at night. That is so cool. You know what would be real cool though is? I also have websites. What if I want to run this on a website?

Jason Arbon: I don't talk about it officially. Yeah. We officially don't support web, but it ran on the web first. We're just focused on apps, because apps the most acute problem. If I talked about the web, I probably wouldn't have gotten as many investors.

Jennifer Bonine: Yeah. It would be cool, very cool to do that. As you said, you do have investors now. There's people that are taking notice of this, large organizations that are figuring out that they need it. Definitely for those of you watching out there that have mobile, that have some of these basic challenges that we know everyone has, you should definitely check this out. Jason, where can they find more information and find you in the company?

Jason Arbon: Appdiff.com. Me, I'm [email protected]. It's pretty easy. I'm also easy to spot, and I'm wandering the halls, and I know we're in the virtual, but we also got a cute little team in the expo. We have a little booth this year. I feel very professional. It feels like a real little deal.

Jennifer Bonine: Yeah. Those people out there, something good to check out there, something you should know if you have the mobile piece. It always goes so fast, Jason. I can't believe that we're almost at the end.

Jason Arbon: That's cool.

Jennifer Bonine: Any last thoughts or things you'd like to share with the folks out there in the virtual world?

Jason Arbon: Yeah. The biggest trend I've seen that really surprised me actually was that people are doing a lot more competitive analysis these days. A lot of the people we're working with today, they go, "Cool. You can automate my app. Cool. You can get all these performance metrics." They're super excited.

Jennifer Bonine: I want to know about their app.

Jason Arbon: They want to know about their competitors. These are people getting very sophisticated. A cool thing we do is we can run on your competitors too and we can run a differential. We can say, "Your cart functionality versus the other guy's functionality, who's faster and a win/loss thing.

Jennifer Bonine: That's awesome. That is so cool.

Jason Arbon: Competitive analysis I think is the next wave post automation.

Jennifer Bonine: Right? Now you're going, "Okay. I got my stuff in order, but what are they doing?" Right?

Jason Arbon: How do you motivate your team and also how do you show off too? If you really are faster, tell your entire company.

Jennifer Bonine: Tout it. Right? If you're on the mobile team, you've got to tout that this is working really well for you.

Jason Arbon: Tell them to bug off. We're faster. Chill out.

Jennifer Bonine: Exactly. Jason, thank you so much.

Jason Arbon: Cool. Thanks, Jennifer. It's good to see you.

Jennifer Bonine: Check Jason out and his company. You guys won't be disappointed. Thanks, Jason again for being with me.

Jason Arbon: Awesome. Thank you. Appreciate it.

Jennifer Bonine: I appreciate it.

Jason Arbon: All right. I'll see you again next year, I'm sure. All right. Thanks.

Jennifer Bonine: Thank you.

Jason Arbon is the CEO of Appdiff, which is redefining how enterprises develop, test, and ship mobile apps with zero code and zero setup required. He was formerly the director of engineering and product at Applause.com/uTest.com, where he led product strategy to deliver crowdsourced testing via more than 250,000 community members and created the app store data analytics service. Jason previously held engineering leadership roles at Google and Microsoft, and coauthored How Google Tests Software and App Quality: Secrets for Agile App Teams.