Jon Hagar is a systems software engineer with over thirty years of experience. In this interview, Hagar discusses how reviews impact mobile app development and testing, security issues in the mobile and embedded world, and why you need to do more than just test the requirements.
Jon Hagar is a systems software engineer with over thirty years of experience. His new book is Software Test Attacks to Break Mobile and Embedded Devices. In this interview, Hagar discusses how reviews impact mobile app development and testing, security issues in the mobile and embedded world, and why you need to do more than just test the requirements.
Jonathan Vanian: Why don't you start off by telling us a little bit about yourself and your career thus far?
Jon Hagar: Well, I've been working in software and software testing for about thirty-two, thirty-four years, something like that. It's been long now that I'm starting to forget how long I've been doing this. Most of that time, I've spent in the embedded software world, and recently, I've gotten interested in the mobile and smart phone world, which is a very close cousin, it turns out, to the embedded world. Written a lot of papers; do a lot of research in that area.
JV: Got it.
JH: It's kind of one of my passions.
JV: Can you explain the link between embedded and mobile; how you found out they’re similar?
JH: Sure, and when I say embedded, it's probably best to start with a little definition on that.
JV: Yeah, go ahead. That'd be helpful.
JH: Embedded usually refers to software that's embedded, hence the name, into some unique hardware, which is fundamentally a little different than your typical mainframe or IP kind of system or PC, which has generic types of hardware. So, the embedded systems have this very unique hardware component to them, such things as planes and automobiles, and one of the definitions I give people is that in embedded, a lot of times the user of the software isn't really even aware that they're interacting with software compared with the PC where it's pretty obvious you're using software.
That's kind of the embedded world. The mobile smart phone world came along in the last few years. There were mobile phones for a long time before that, and they were more of the classic kind of mobile phones before you the smart phones; kind of the classic embedded device. People were probably aware there was some software going on there, but it wasn't say like your PC software.
As the smart-phone revolution took place, where you had many of the characteristics of the PC world in the phone itself, people realized, “Oh yeah, there's apps in their software and I'm kind of using this little hand-held computer.” But a lot of the developers used the same kind of approaches that they did in the IT world when they were developing apps.
The problem with that is there's some fundamental problems that very much correspond to the embedded world, which is how I became interested in that; such things as batteries become a big issue. Battery life is a problem for many smart phones, and there's an awful lot of things, bugs even, that can impact an app and how it drains a battery. As that is one example of the kind of thing that the embedded people are always worried about. I became interested in smart phones because it seemed like it shared a lot of the characteristics of the embedded world plus a lot of the characteristics of the typical software world; I started doing research on those.
JV: In the past several years how much has mobile grown?
JH: If you read a lot of the literature, the phones themselves, in terms of just raw numbers in the world, are poised to surpass, if they haven't already done so, the number of regular PCs. Everywhere, everybody's got one across the world. You have this proliferation of the hardware, and then, along with that, kind of a frenzy like what we saw back in the dot-com days of the Web and, maybe, the PC craze before that, where everybody wants to now have a mobile presence.Everybody is creating an app. There's tens of thousands, if not millions, of apps out there. When I started looking at, particularly, mobile app smart phone software, it was pretty obvious to me it wasn't really well tested. People were just putting things out there. I've heard some friends call it the Wild Wild West. Kind of like the web in the early days; people just didn't care about quality.
Then you look at a lot of the reviews in social media that sprang up at the same time. The users can quickly brand an app as good or bad, and it can basically kill an app if enough reviews get posted online in the app store that say, "Hey, this thing doesn't work" or "This doesn't work on my particular platform."
JV: That’s such an interesting concern. That instant review, the instant critique.
JH: Yeah, and you see things posted in there that where I came up with my book was by reviewing a lot of the public available information by places like app stores and news reports across the mobile smart-phone space and the embedded space, and I tried to get a handle on the kinds of errors that were, I'll say, being found by users out in the field very commonly. From that, followed along with Dr. James Whittaker's work of attack-based testing, with the idea that you not only want to work and test the functionality of some software, show that it does the requirements is the classic one, but that you also want to come along and really try and show that something doesn't work. The break-it philosophy.
JV: I was wanting to get into that; why do we need to break mobile and embedded devices?
JH: My answer is that if we, as the test community, don't, the users will, and we see the bad reviews. In some of the mobile and embedded space, we see various horrors stories, even lawsuits, from systems not working correctly. So, there's in my mind a fair amount of the need to have the right amounts of testing. I'm not saying that every mobile smart phone app out there needs to be tested at the same level as, say, the space shuttle was tested, but you need enough testing early on so that when you get it out in the field, hopefully rapidly, that you don't get those bad reviews, because otherwise, you're going to probably not be around to have another go around of changes.
It's a balancing game between the level of cost and schedule and how much testing, where I think, unfortunately, some places don't do enough testing.
JV: From some of the folks that you talk to, are the reviews really making them want to double down their testing? Is there a sense of urgency out there?
JH: I'm not sure the urgency is there yet. I think some of the mobile app developers have started to realize they need more. This was what we saw back in the dot-com days and the PC world where you could get away with almost anything. As the users become more sophisticated and the social media increases, I think a lot of the people will start going, "Gee, we need that right level of testing," and again, there's no one size fits all of that.
JV: What are some things traditional programmers misunderstand about mobile?
JH: I mentioned the one about batteries. That's one I found quite a bit and offer up a bit of specific attack on. The other one that's cropped up quite a bit recently and scares me quite a bit is the whole security end of things, in everything from mobile banking to other fields. I was watching news pieces where people were walking around stores and being tracked; Is that what they want?
There's a whole set of security issues; the limitations of the devices themselves. You have small screen size. That's a limitation. You have, relative to a PC, smaller amounts of memory. Yes, there's a lot of memory, but smaller amounts of memory and storage capability. I think one of the things a lot of the PC people forget, because they're used to just very high bandwidth in a direct hard-line connection, is you have the connection to the mobile service provider, which can come and go. You can get low bandwidth, high bandwidth; it can change as you move around. How does that effect the app and the software? It becomes an issue that everybody needs to think about.
JV: There's another body that the thing needs to communicate with that might not have been there in the PC world.
JH: Yeah, where you get it to pop out and lose some data or something like that.
JV: How about other major security issues?
JH: Well, and it's evolving quickly. In the book, I had some attacks. I think those are still good and valid. Your basic login signature kind of thing, but things are changing rapidly. You have the biometrics concepts that we see a little bit with some of the Apple products, where you have fingerprint recognition. I think those things are going to both help security but then introduce a new level of, I'll call it, concerns for testers. If you've got critical data that you're exposing, that may not be what you want.
You also see spoofing. There's something known as GPS spoofing that's very sophisticated but nonetheless possible. Do you want to have yourself hijacked when you're driving your car?
JV: That's a pretty major concern, I would suspect, that people would have.
JH: We know from reports people have cracked into cars. We know people have cracked into pacemakers, another kind of embedded device. Again, a little bit of knowledge for me just makes me very scared, and I've been studying this a fair amount, but I still have an awful lot to learn. I think for those of us in the mobile security community, it's going to be a lot of years and a lot of hard work.
JV: You have a chapter in the book on smart hand-held mobile systems. What is a smart hand-held mobile system? What would constitute something to be smart?
JH: The most obvious one are the Androids and the iPhones and those kinds of things that I think probably a lot of people have, but there's also other things that are out there that maybe we don't think about as much. A lot of the cars these days come with a network online and information and entertainment systems built right into the car. A car is certainly mobile. It's somewhere in between a phone and an embedded device.
I recently flew the new Boeing airplane. It's got an online information system.
JV: Were you thinking about testing the whole time when you were flying in that plane?
JH: Well, a little bit, and I don't want to tell stories too much, but the system actually had some bugs in it. So I was like, "Oh, this is kind of interesting." We had some interesting conversations with the flight crew. It was a very nice airplane, but as any new high-tech thing, there's things to be worked out.
Then, you also have the whole medical device industry, and they're mobile in a very different kind of way, and they're very smart too, so a lot of us that are working in this want to say it isn't just the phones that's the big gorilla in the room. There's these other things too that need to be thought about for testing.
JV: What do you want readers to take away from the book? What is your goal with that?
JH: Again, it's that concept that you need to do more than just test the requirements. You need to look at the other qualities, security being one, usability being another.
There's a variety of qualities, and that there's these approaches, these patterns as I call them in the book, that the attack-based testing community that James Whittaker started up has realized, along with the concepts like exploratory testing that we really need creative testers to go out and push into the areas that the developers didn't think about and provide that information back to the stakeholders. The developers can say, "Hey, we can crash it," or "We can get these strange results. Maybe this is something the user is going to be unhappy with." It's that trying to break-it-attack approach that's the main message I'm trying to get people to take away.
JV: Right, and how long have you spent researching for this book?
JH: Well, in some ways, all thirty years.
JV: These are your memoirs.
JH: Yeah. I built up a lot of mental models on embedded systems over that time. A few years back, probably three or four years back, I started some research. There was actually a Star West Best Paper Award that I got in 2010 that was based upon that research of taxonomies of common errors, and as I started thinking about those common errors, I realized, gee, this attack-based approach is good for that.
Somewhere in about 2010, I started incorporating the mobile smart phone into the taxonomy. I'd say probably the last four or five years of research and then probably one-and-a-half years of writing. It's a lot of work to do a book.
JV: Thank you for taking the time.
Jon Hagar is a systems software engineer and tester supporting software product integrity, verification, and validation with a specialization in embedded software systems. For more than thirty years Jon has worked in software engineering, particularly testing/verification and validation. Embedded projects he has supported include the space shuttle, large rockets, spacecraft, and ground systems as well as testing new smart phones. Jon has managed and built embedded test labs, including supportive automation development. Jon publishes regularly with more than fifty presentations and papers in software testing, verification, validation, agile, product integrity and assessment, system engineering, and quality assurance. He teaches classes at the professional and college level.
