Government Regulation and Medical Apps: An Interview with Jonathan Kohl

In part two of an in-depth discussion on what exactly it’s like to work on software development and testing for mobile medical devices, Jonathan Kohl, a contributor to StickyMinds, explains the challenges of working with a government agency, meeting goals and marks when working under regulations, and the acceptance of mobility as important technology for health care.

Jonathan Vanian: You mentioned how challenging it was dealing with the fast environment of mobile in contrast with the slow moving government agency using old technology. What were some of those challenges and how did you reconcile them?

Jonathan Kohl:  Here's an example: Our biggest fear when we were getting close to a release was a new operating system release appearing out of nowhere, which would require a lot of extra work. We have to try it out on devices, go through a regression suite, and determine if we can move to it or not. If it looks solid, then we update all the devices, analyze what risks the new OS brings to our app, do a full regression, and focus on some of the new risks that are introduced. Invariably, there are failures and code rework, so it is almost like doing a new feature release. We would want to move on and get closure on a release, but the agency is like other similar agencies. They have a huge responsibility, are trying to do the right thing for the public that they represent, and they aren't as well staffed as a new social-gaming startup might be. Their work takes time.

We reconciled this by being as lightweight as possible. We were using a modified version of Scrum, which is quite lightweight as a methodology, but with more roles and documentation overhead, with sprints dedicated to regulatory compliance or to testing and bug fixing and related processes. We had to work hard with our internal auditor to make sure that we had just enough process and documentation to satisfy the auditors, but our primary focus was on working with the technology and staying current. Our burden was to make something safe and usable that could enhance people's work and improve patient care. In some cases, what we were providing might even save someone's life. So we wanted to make sure we were developing and testing the absolute hell out of these things; we moved rapidly and used a ton of models of coverage and user scenarios in all sorts of situations.

Take network testing for an example. A hospital is a terrible place for network communication, so we tested wandering around a large concrete and steel building, and tested different network strengths, and transitions from different network types, such as various WiFi strengths, transitioning between WiFi beacons in a large building, and moving from WiFi to different cellular networks. Imagine that you are a doctor on rounds, you have to rush around a building, and you need to be connected to get information at any given time in different locations. You go from an underground parking lot, which is surrounded by concrete and steel, and you may have no connection, or a slower cellular connection such as EDGE. You move into an elevator and lose your connection altogether.

Next, you pop out of the elevator and walk down a ward with your device automatically connecting to three different WiFi connections on your trip. Now you are in a room with a family and you need to get dad's cardio study up to show the family why he has been rushed into emergency surgery. Emotions run high; you are stressed and family members are shocked, confused, anxious, and want to know immediately what is wrong. You don't want an app that crashes because it couldn't handle those transitions.

Now imagine requiring access in any of those points I described above. You may be in a transition point between network types or sources or in an area with poor network strength. Maybe you are walking down the hall getting a second opinion from a colleague; we tested this and developed code in our app to handle these situations well and we tested many other mobile-specific scenarios heavily. There are a whole host of other things to consider—low batteries, syncing with PCs, and different lighting conditions (radiologists like the dark, but we had one radiologist test the software when he was in the park with his children under bright sunlight.)This project heavily influenced mnemonic, which outlines many of the mobile-specific models of coverage we used.

We recorded what we did with those kinds of scenarios using session-based testing and we had to make sure our session sheets were detailed enough so they could be followed by another person. The session sheets also had to follow document management guidelines, with proper oversight, storage, titles, headers, footers, etc. So we had to have really solid debriefs and more editing on the actual docs than I was used to.

So we had a balance. We needed to be up to date on technology, be compliant with our regulatory oversight, and always trying to find a happy medium. We had a lot of skilled people who could work efficiently, get up to speed on technology quickly, and with whom we collaborated closely on every aspect of development and regulatory compliance. That helped tremendously. We also used mobile technology to stay in touch and communicate when we were on the move, to record videos for demonstration purposes, or to record bugs. We could get a message with a video or screenshot to our device from someone doing field testing in seconds, and from there figure out how to address it.

JV: You mentioned that some issues the FDA was concerned about were bench tests and clinical trials. How did you pull off testing when you had to meet these multiple goals/marks?

JK: They were just other models of coverage to manage and complete. As a tester, I don't just look at the requirements and test off of them. That isn't nearly enough. I always use multiple models of coverage to get the most out of our testing efforts. I also tend to do lightweight risk assessments on most projects I'm on, which help us determine what models of coverage to use to mitigate those risks. The nice thing about an FDA or other regulated process, is that this work is not optional, so you get more support for being more thorough and creative. So in short, as a tester, I always have multiple goals/marks in my work. It was just far easier to sell when it was a requirement. The clinical trials were fascinating to manage: Radiologists and others who do diagnostic work are amazingly talented. They would see a pathology instantly where I would see nothing out of the ordinary. It was an honor and a pleasure to work with them and learn from them. Their input helped our other types of testing enormously, especially with user scenario testing. We got a sense of their emotional state and urgency and their motivations and fears when they use this kind of software, so those perspectives informed other areas of our testing to make it much more effective.

Years ago I read a piece by Brian Marick where he described that less depth across multiple models of coverage tends to be more effective than really thorough depth in one particular model of coverage. I've used that as a guideline ever since. I never just take the requirements or spec and use that as my only model; it's often foolhardy, frequently too narrow to catch anything but the obvious, and I find it boring. I've had much more success testing with multiple models. It helps uncover missing requirements, larger project assumptions, and helps us find problems that other people hadn't thought of looking for. That's how we add value as testers, by looking for things others wouldn't think of. In medical, or other mission critical areas, it is incredibly important to test as much as we can in as many ways, approaches, or perspectives in order to find important information quickly and get it dealt with. That's my dad, mom, sister, wife, or me who will be negatively impacted if we get it wrong; or if not mine, someone else's family.

JV: It’s common knowledge that the government is used to dealing with old technology and may not be as up-to-date with the latest trends in computing like the rest of the software world. That being said, do you get a sense that the government is taking steps modernizing the way they work with software?

JK: Absolutely. Many of them use mobility tools in their own lives, or for productivity, so they understand it. They are trying their best, but government agencies don't get the funding they should, especially in difficult economic times. I have worked in several regulated environments and I usually find the auditors to be fair-minded people. They are a bit scary because they have an important job and walk with a big stick so to speak, but most of them are intelligent and working hard to do the best they can.

It can be frustrating that they don't move at the pace we would like them to, or in line with technology, but they have a very important job to do. Without oversight, we'd be back to the days of snake-oil salesmen and leeches. Sometimes their decisions are controversial and people get upset, but to my view, these agencies do more good than harm.

JV: This article from the Healthcare Information and Management Systems Society points to the ways the medical app industry is maturing and the author believes that the new FDA regulations will give it credibility. How do you think the mobile medical app world might change once the regulations go in place?

JK: I see less to do with the regulations here in this piece  and more of an acceptance of mobility as important technology for health care. It's not that the mobile medical app world will change as much as the medical world will be changed by mobility apps and services.

With the apps I was working on, we had to have FDA clearance for them to be sold to healthcare professionals in the USA, as well as with related regulatory bodies in other countries. That was a requirement for usage. To see practitioners faces light up as they used our software and recognized its potential was huge. It’s hard to describe the feeling of being part of something that important. So my read on this is that now that the FDA has developed regulations, there is much more opportunity to create applications and systems that will make our lives better. I don't say that lightly.

As the article stated, mobility to the medical industry is like ATMs to the banking industry. Just imagine how we are able to be productive with mobility apps and services when we travel or are merely away from our desks, and imagine what that could do for your doctor. One of my doctors uses a MacBook Pro during her consults and is very tech savvy. She loves the idea of mobile devices in the medical profession, because she'd like to get the benefits of technology she enjoys at home in her office. Last time I was in to see her, she left the room to get a second opinion. She rushed back in and said: "Don't look this up on your iPhone while I'm gone!" She didn't want me to read something on the internet and get all freaked out while she had stepped out. I have more access to information in that case than she does, but the information I have access to is often wrong or suspect, or not applicable without an expert eye. We laughed about it; her husband is also in the software industry, so she understands. Imagine if she used mobility technology to get a second opinion with me as an active participant. Or to take it further, imagine using mobility to get expert oversight or participation during important surgery. The foremost expert in the world on this surgery is attending and directing without having to travel and be there with me as I get the surgery. Amazing. Some of my colleagues are developing this kind of software today, as I write this.

The flipside of this are the inevitable apps that promote quackery and junk science, and governments want to crack down on things like that. They don't want people to get apps that cause harm to themselves or others, so they are trying to manage this. One change we might see in the mobile apps space is more oversight for store submission related to health applications for the mass market.

However, the exciting thing is that we will see more medical apps and systems taking advantage of mobile technology. With the power of mobility, we should see better service for people who require health services, which at some point or another is all of us.

Finally, since I don’t have a medical background, I'd like to thank my friend and software colleague Jaret Hargreaves, (MSc. Biomedical Engineering), for his review and clarifications of my answers.

Read more in the continuation of this interview: Testing Mobile Medical Apps: An Interview with Johnathan Kohl.