Testing Mobile Medical Devices: An Interview with Jonathan Kohl

With the Food and Drug Administration gearing up to create an agreed upon set of guidelines to regulate the mobile medical device app market, software developers who take on such projects need to be aware of what it’s like to work with a government agency right by your side. For an in-depth discussion on what exactly it’s like to work on software development and testing for mobile medical devices, I contacted Jonathan Kohl, a contributor to StickyMinds, who has recently written about exploratory testing in an FDA-regulated environment. Part one of our conversation focuses on the software project he worked on in addition to what it’s like to work under FDA regulations.

Here’s an edited version of our interview conducted through email.

Jonathan Vanian: Can you tell us a bit about the device you once worked on and its related software?

Jonathan Kohl: I worked on medical visualization software for popular smartphones and tablets.

In the old days, we used to get scans printed on film. The classic example is an X-ray. If you injured yourself as a child, you probably went to see the doctor and they took an X-Ray, then put the film up on a lighted board and showed you where you broke it. Well, those days are gone; all the scans are now stored as image files on servers and sometimes the only way you can show a patient what his pathology looks like and why doctors are recommending a certain course of treatment is to book time on the actual scanning machine itself. If you're lucky, the specialist you see has a machine in their office and they can show you in a consultation just after the scan. If your medical scans are sent to an expert, you won't see them. By the time this information gets down to your family doctor or the specialist who ordered the scan, they may not have any images to show you. So this application and system that I worked on provided access and visualization so that a medical practitioner could pull out a smartphone or tablet and show you exactly what was going on during the consultation. This is a huge improvement for patient care because digital technology has caused us to lose something now that the medium has changed.

Another use of the software was for remote diagnosis assistance. With certain pathologies or events, getting a course of action for treatment in place needs to be done as soon as possible. So if a patient has a medical emergency, gets rushed to the hospital, and a specialist or radiologist is on call, the specialist can quickly look at the scans and recommend an immediate course of treatment to start with while he or other experts rush in to help. In smaller centers, the hardware they use might be located in a larger city, so getting access to studies might require file transfer or sending images in an email. That isn't always secure and some file sizes can be quite large, so having secure, remote access to a larger system can be important.

The software does more than just show images. For example, it creates 2D or 3D rendering and allows people to zoom in and rotate images to look at them from different angles and cut sections out to get different views, among other things. It’s almost like a game—you load a medical study and start gesturing to view it in different ways.

JV: You mentioned in our email correspondence that the latest news regarding the FDA stems from last year when they finally decided to approve apps on the devices due to concerns with the devices themselves (screen resolution, size etc.). Can you elaborate on that and what some of the concerns were?

JK: I'm no expert on this topic, but the biggest concerns I recall had to do with display characteristics, which includes the size of the screens, resolution, color depth, and other characteristics. Another concern was with the security of patient data. Display characteristics is a concern because they wanted to be sure that experts could make the same diagnosis on a small screen with a lesser resolution than a larger screen with a higher resolution. There are medical-grade monitors and an entire industry of hardware that are being developed and regulated just for diagnostics. Security is also important; you don't want your name, physical attributes, patient ID, insurance number, and other personal information to be either intercepted by or accidentally sent to someone who shouldn't see it.

In our case, we were classified as a Class II device, which the FDA does not “approve” but allows for the products to be “cleared.” This is not an endorsement but it allows regulated devices to be marketed.

 JV: I know it’s been a while, but can you walk us through what it’s like to go through the FDA review process and the agency’s requirements?

JK: As for the requirements, I urge anyone who is involved in this to study the source themselves.

Again, I'm not an expert. We had an internal auditor who oversaw this. I plugged in from a testing perspective, so here are some of my experiences. (Note: This is not official, or thorough, these are just some aspects that I remember.)

It depends on how important or risky they rate your app. They have three categories: major, moderate, or minor level of concern. Essentially, if a malfunction can kill or seriously injure someone, then it is major. If it can harm someone, it is moderate. And if a malfunction won't cause someone harm, it is minor. Major level of concern designation is for devices that have direct interaction with the patient—they administer medications, monitor, help with bodily functions etc. We fell in the moderate level of concern designation because the app could be used to interpret patient data. The overhead goes up significantly the higher up the chain you go.

Some of the review process has to do with your own software development process. Is it repeatable? Do you have mechanisms in place to manage requirements, code, and testing? Do you have a code repository? How about code, design, and risk reviews? There is a lot to manage with regards to having a process and roles. Do you have someone who manages the requirements? How about the documentation for auditing purposes? They are fairly flexible on process, as long as you demonstrate some level of maturity with regards to oversight and control in what you do, and they are getting used to things like Agile and other lightweight methodologies. The biggest thing people think about is documentation, and yes, you do need a good deal of that. You need to have a document control system, including storage, access control, and the format of the documents needs to be consistent with proper titles, headers, and footer information. Some of these requirements are very similar to an International Organization for Standardization (ISO) certification.

You have to do a solid risk assessment where you essentially outline threats to patients and practitioners and list ways you mitigate them, through process, testing, etc. For example, a device malfunction might lead to a lack of ability to diagnose. So you mitigate that through requirements work, user scenarios, code design, code reviews, unit testing, user scenario testing, failure recovery testing, and things like that. If your risk assessment is light, then you will fail your audit, so it needs to be honest and thorough.

You also need plans that show how you will create the product, and how you will mitigate those risks through process, tools, testing, etc. So from my perspective, I was responsible for creating a test plan. You also have to make sure you have verification testing in place, so you look at the requirements and specs and test off of them, as most traditional testers would. If you notice in the FDA regulations, they talk about testing approaches like unit testing, system testing, functional testing, and integration testing. That helps testers who like to use different models of coverage because it is easier to sell using this kind of approach to other stakeholders. That was just our starting place though; we went far beyond based on what we discovered through our risk assessment process and for attributes and affordances that were unique to mobile devices. A lot of time was spent on failure modes and how the app could handle them.

We required validation and verification in two specific areas:

• Clinical studies: Highly trained radiologists recorded their findings on FDA-cleared products and on our mobile apps on devices, and demonstrated there was no appreciable difference in the findings.
• Technical bench testing: A third-party organization conducted display validation using the devices to generate test images.

Next, for auditing purposes, someone from outside the team needs to be able to select tests and run them and will also look at failures. They will open up a random bug report and follow it. Can they repeat it? What was done to fix it, and what was done to verify it was fixed? They will look at the entire system from risk assessment to plan to tests to failure report and retesting for traceability. What test failed? How did you know there was a failure? How can you tell whether it was truly fixed or not? What safeguards do you have in place to ensure the fix stays in the code? It's like any auditing situation: You have processes and safeguards in place and an outsider can have a look at what you are doing and understand them. Accounting firms do the same thing and I looked at the FDA guidelines much like GAAP (generally accepted accounting principles) and the FDA is quite permissive in exactly what you do, as long as it follows the spirit of the regulation. They don't demand one certain way of developing software because they understand that there are many acceptable ways of doing this well and technology can dictate certain practices. Their "lease burdensome approach" concept is really nice. I've worked with other agencies who weren't as open minded. I had one agency try to force a Best Practices model of software development and testing that was created by an accountant. It was so out of step with reality it was absurd. They had no clue about iterative, incremental methodologies, the agile movement, automated testing, let alone risk-based or exploratory testing. In contrast, I have found the FDA process to be much more reasonable.

JV: Was the testing more difficult in a mobile medical app environment as opposed to more traditional types of testing?

JK: To me, no. I am probably the furthest thing from a traditional tester you will find though. I did meet some testers who didn't like the process. They liked to create a new release in a test case system, copy their last test plan, change a few details and regression test off of scripted test cases without much thought. They felt like it was too much work, and resisted the process because it wasn't easy and because it required thought. I also found some of my agile testing purist colleagues who complained about the extra documentation and having to break the rules of Scrum to make it work. We were pragmatists though and figured our way through it trying to keep the spirit of Scrum alive while working in a regulated system. I found the FDA-mandated approach to be in line with how I tend to work anyway (just with more overhead), so it was nice to plug in my risk assessment into our internal auditors work and collaborate. However, the stakes are much higher, and there was a higher process and documentation burden, which could get tiresome at times. You want to move on and work on the new phone or tablet that came out, but you have to spend a couple of days buttoning up docs.

You have this extra part of your release that depends on oversight from a regulatory body (and we worked in several countries, so we had several) and that changes the pace of a release as opposed to a non-regulated environment. You don't get to move on as quickly, and if you fail your audit, or you get a pass with strict orders to fix some areas of weakness, you will spend more time with existing products.

The difficult part was the high stakes of what you are doing. You feel a tremendous amount of responsibility because this software affects people's lives. The flip side of that is you know you are working on something that can really help people, not like some get rich quick social networking mobile app scheme. Not that there's anything wrong with sexy startups trying to make a lot of money, but you feel you are helping humanity by contributing to health technology. That is a great feeling.

The hardest part for me personally was dealing with test data for certain kinds of scans. For some, I just saw a body, and I was interested in the pathologies. For others, it was a person, and I felt disturbed. I couldn't test mammography scans very much; I found it very upsetting when I saw the pathologies in the scans. I kept thinking that it was someone's wife, mother, sister, or aunt. It was also difficult to work on some of the tests when a family member was going through a health scare at the same time. I didn't count on the emotional or visceral impact some of the work would have on us—to look at scans of sick people all day. We scheduled time on other duties to mix things up in part to help deal with the emotional toll this could sometimes have.

Read more in the continuation of this interview: Government Regulation and Medical Apps: An Interview with Jonathan Kohl.