Medical Device Software - Final Field Testing Best Practices

R

robertjbeck

There is a disconnect between best practice for final testing, such as field testing or beta testing, and what can be done with medical device software. For instance, one official definition of beta-testing is that it is done in the field with pre-release software with a limited set of customers/users. How does this fit with a typical design control scenario, where the software is released as a product after extensive verification/validation but because of its complexity requires clinical use to learn about potential issues/problems? Is a small clinical trial a valid method? Does anyone have any ideas or experience with FDA regarding this type of situation?
 
M

maaquilino

There is a disconnect between best practice for final testing, such as field testing or beta testing, and what can be done with medical device software. For instance, one official definition of beta-testing is that it is done in the field with pre-release software with a limited set of customers/users. How does this fit with a typical design control scenario, where the software is released as a product after extensive verification/validation but because of its complexity requires clinical use to learn about potential issues/problems? Is a small clinical trial a valid method? Does anyone have any ideas or experience with FDA regarding this type of situation?

Beta testing, as described for most software, isn?t done the same way for medical device software. What one would call beta testing for a non-medical device software, is called PQ for medical device software (it?s been called other things also, such as UAT or FAT testing, depending on what industry you?re in). When developing medical device software, after OQ testing the user testing is done during PQ testing, which is testing done by users, or potential users of the system. Once PQ is completed per its? protocol and is determined it is fit for use, the system can be cut over for a process validation in the ?live? environment. At that time a clinical trial can be done.

Clinical trials usually aren?t done to learn about potential issues/problems. They?re interventional and observational studies done based on research plans or protocols which add to medical knowledge and generate safety and efficacy data. In the medical device world, a clinical trial wouldn?t be done unless the software has first been verified and validated. You?re asking people to use a system that hasn?t been fully tested on humans who could be hurt or worse if the system fails; the liability issues would be tremendous. I?ve dealt with this in the past and had to deal with it on the current project I?m on. The company developing the device, their first medical device ever and who have already stated they won?t be doing any clinical trials, wanted to put two units in the setting they will be sold in, before the final verification and validation were done, and run them to see how the users liked them (they have also had extensive interaction with potential users, focus groups, etc. already in order to define their requirements and get feedback on the system). They can?t do this, as 1) it?s a clinical trial which they said they weren?t going to do and don?t have an IDE for; 2) they?ve already had extensive feedback from their focus groups, which are made up of potential customers for the device; and 3) if someone were to get hurt during these ?clinical trials? using a system whose software has not been fully verified and validated, that someone could end up owning this company.

That said, the FDA does allow for feasibility studies, under an IDE, and what they call ?just in time? testing. The state ?And in these cases early clinical use of the device in a limited number of subjects is needed to provide initial insights into clinical safety and device function, inform subsequent clinical and non-clinical testing and/or improve device performance through iteration before finalizing the design. A guiding principle of the early feasibility study is a term we call just-in-time testing and this is a term that we borrowed and modified from industrial production. Just-in-time testing applies to the type and timing of non-clinical testing needed to justify study initiation.? So the FDA allows feasibility studies to be done, the results of which may end up altering the design of the device, but they still require some testing to be done prior to that, and an IDE. FDA states ?An investigational device exemption (IDE) allows the investigational device to be used in a clinical study in order to collect safety and effectiveness data. Clinical studies are most often conducted to support a PMA. Only a small percentage of 510(k)'s require clinical data to support the application. Investigational use also includes clinical evaluation of certain modifications or new intended uses of legally marketed devices. All clinical evaluations of investigational devices, unless exempt, must have an approved IDE before the study is initiated.?

Here?s some links that you might find useful; the FDA comments above were taken from these links.
http://clinicaltrials.gov/ct2/about-studies/learn#WhatIs

http://www.fda.gov/Training/CDRHLearn/ucm372150.htm

http://www.fda.gov/medicaldevices/d...investigationaldeviceexemptionide/default.htm
 
R

robertjbeck

Good answer. The problem I face is that the software involves enough real-world complexity that regardless of the level of pre-release testing, there are going to be lessons learned from actual use, and these lessons will involve changes to the software as the best response. Maybe putting these in terms of a feasibility study is a valid approach (from a regulatory point of view). The software development team and the marketing group think otherwise.

This situation is similar to selling non-software device for 'research use only' to get real-world experience that cannot be obtained any other way.
 
R

robertjbeck

Your reply triggers another question .. what if the manufacturer sells the product to another organization that runs the clinical trial/observation study? does it make sense to deal with software issues as post-market problem reports? Does the manufacturer have any obligations about the trial?
 
M

maaquilino

Good answer. The problem I face is that the software involves enough real-world complexity that regardless of the level of pre-release testing, there are going to be lessons learned from actual use, and these lessons will involve changes to the software as the best response. Maybe putting these in terms of a feasibility study is a valid approach (from a regulatory point of view). The software development team and the marketing group think otherwise.

This situation is similar to selling non-software device for 'research use only' to get real-world experience that cannot be obtained any other way.

There will always be lessons to be learned from a product after release, whether it?s a medical device or not ;) If it were me, if I thought I?d learn a lot through a clinical, such that it would change my design significantly, then I?d go for a feasibility study. If I thought I?d ?maybe? learn something, and it ?might? add or modify some functionality, then I?d 1) get some user feedback during the design phase and have some of those users involved in the PQ testing. I?d prefer to find my issues sooner rather than later, and I feel there?s more hassle when you have clinical studies. Just my experience; others may have had much different ones. Plus, imo, the whole purpose of designing software is so that it?s fit for its? use, and I really can?t know that for sure unless I?m talking to the people who will be using it. We did that with the current project I?m on; there were focus groups, discussions with people who would be the actual users, a trade show that brought in tons of feedback, etc. It all helped us design a product that the users really see a benefit in and are anxious to get to use on a daily basis.


Your reply triggers another question .. what if the manufacturer sells the product to another organization that runs the clinical trial/observation study? does it make sense to deal with software issues as post-market problem reports? Does the manufacturer have any obligations about the trial?

In the regulatory world, the manufacturer is still responsible for their device, which includes the software. You can sell to whomever you want, but if you made it, it?s your responsibility to ensure it works as designed.

Are you asking what if the other organization does the clinical study on the device/software that they were sold? Or if they?re using the device/software to run a clinical study? These links may help answer either or both of those.

http://www.fda.gov/downloads/Medica...onandGuidance/GuidanceDocuments/UCM279103.pdf

http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm373750.htm

http://www.fda.gov/MedicalDevices/D...ubmissions/PremarketApprovalPMA/ucm050419.htm

http://www.fda.gov/MedicalDevices/D...YourDevice/InvestigationalDeviceExemptionIDE/

http://www.fda.gov/Training/CDRHLearn/ucm209135.htm

As this deals with some RUO and IUO products, it has some good info in it.
http://www.fda.gov/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm253307.htm
 
Top Bottom