Medical Device Software - Final Field Testing Best Practices

R

robertjbeck

#1
There is a disconnect between best practice for final testing, such as field testing or beta testing, and what can be done with medical device software. For instance, one official definition of beta-testing is that it is done in the field with pre-release software with a limited set of customers/users. How does this fit with a typical design control scenario, where the software is released as a product after extensive verification/validation but because of its complexity requires clinical use to learn about potential issues/problems? Is a small clinical trial a valid method? Does anyone have any ideas or experience with FDA regarding this type of situation?
 
Elsmar Forum Sponsor
M

maaquilino

#3
There is a disconnect between best practice for final testing, such as field testing or beta testing, and what can be done with medical device software. For instance, one official definition of beta-testing is that it is done in the field with pre-release software with a limited set of customers/users. How does this fit with a typical design control scenario, where the software is released as a product after extensive verification/validation but because of its complexity requires clinical use to learn about potential issues/problems? Is a small clinical trial a valid method? Does anyone have any ideas or experience with FDA regarding this type of situation?
Beta testing, as described for most software, isn?t done the same way for medical device software. What one would call beta testing for a non-medical device software, is called PQ for medical device software (it?s been called other things also, such as UAT or FAT testing, depending on what industry you?re in). When developing medical device software, after OQ testing the user testing is done during PQ testing, which is testing done by users, or potential users of the system. Once PQ is completed per its? protocol and is determined it is fit for use, the system can be cut over for a process validation in the ?live? environment. At that time a clinical trial can be done.

Clinical trials usually aren?t done to learn about potential issues/problems. They?re interventional and observational studies done based on research plans or protocols which add to medical knowledge and generate safety and efficacy data. In the medical device world, a clinical trial wouldn?t be done unless the software has first been verified and validated. You?re asking people to use a system that hasn?t been fully tested on humans who could be hurt or worse if the system fails; the liability issues would be tremendous. I?ve dealt with this in the past and had to deal with it on the current project I?m on. The company developing the device, their first medical device ever and who have already stated they won?t be doing any clinical trials, wanted to put two units in the setting they will be sold in, before the final verification and validation were done, and run them to see how the users liked them (they have also had extensive interaction with potential users, focus groups, etc. already in order to define their requirements and get feedback on the system). They can?t do this, as 1) it?s a clinical trial which they said they weren?t going to do and don?t have an IDE for; 2) they?ve already had extensive feedback from their focus groups, which are made up of potential customers for the device; and 3) if someone were to get hurt during these ?clinical trials? using a system whose software has not been fully verified and validated, that someone could end up owning this company.

That said, the FDA does allow for feasibility studies, under an IDE, and what they call ?just in time? testing. The state ?And in these cases early clinical use of the device in a limited number of subjects is needed to provide initial insights into clinical safety and device function, inform subsequent clinical and non-clinical testing and/or improve device performance through iteration before finalizing the design. A guiding principle of the early feasibility study is a term we call just-in-time testing and this is a term that we borrowed and modified from industrial production. Just-in-time testing applies to the type and timing of non-clinical testing needed to justify study initiation.? So the FDA allows feasibility studies to be done, the results of which may end up altering the design of the device, but they still require some testing to be done prior to that, and an IDE. FDA states ?An investigational device exemption (IDE) allows the investigational device to be used in a clinical study in order to collect safety and effectiveness data. Clinical studies are most often conducted to support a PMA. Only a small percentage of 510(k)'s require clinical data to support the application. Investigational use also includes clinical evaluation of certain modifications or new intended uses of legally marketed devices. All clinical evaluations of investigational devices, unless exempt, must have an approved IDE before the study is initiated.?

Here?s some links that you might find useful; the FDA comments above were taken from these links.
http://clinicaltrials.gov/ct2/about-studies/learn#WhatIs

http://www.fda.gov/Training/CDRHLearn/ucm372150.htm

http://www.fda.gov/medicaldevices/d...investigationaldeviceexemptionide/default.htm
 
R

robertjbeck

#4
Good answer. The problem I face is that the software involves enough real-world complexity that regardless of the level of pre-release testing, there are going to be lessons learned from actual use, and these lessons will involve changes to the software as the best response. Maybe putting these in terms of a feasibility study is a valid approach (from a regulatory point of view). The software development team and the marketing group think otherwise.

This situation is similar to selling non-software device for 'research use only' to get real-world experience that cannot be obtained any other way.
 
R

robertjbeck

#5
Your reply triggers another question .. what if the manufacturer sells the product to another organization that runs the clinical trial/observation study? does it make sense to deal with software issues as post-market problem reports? Does the manufacturer have any obligations about the trial?
 
M

maaquilino

#6
Good answer. The problem I face is that the software involves enough real-world complexity that regardless of the level of pre-release testing, there are going to be lessons learned from actual use, and these lessons will involve changes to the software as the best response. Maybe putting these in terms of a feasibility study is a valid approach (from a regulatory point of view). The software development team and the marketing group think otherwise.

This situation is similar to selling non-software device for 'research use only' to get real-world experience that cannot be obtained any other way.
There will always be lessons to be learned from a product after release, whether it?s a medical device or not ;) If it were me, if I thought I?d learn a lot through a clinical, such that it would change my design significantly, then I?d go for a feasibility study. If I thought I?d ?maybe? learn something, and it ?might? add or modify some functionality, then I?d 1) get some user feedback during the design phase and have some of those users involved in the PQ testing. I?d prefer to find my issues sooner rather than later, and I feel there?s more hassle when you have clinical studies. Just my experience; others may have had much different ones. Plus, imo, the whole purpose of designing software is so that it?s fit for its? use, and I really can?t know that for sure unless I?m talking to the people who will be using it. We did that with the current project I?m on; there were focus groups, discussions with people who would be the actual users, a trade show that brought in tons of feedback, etc. It all helped us design a product that the users really see a benefit in and are anxious to get to use on a daily basis.


Your reply triggers another question .. what if the manufacturer sells the product to another organization that runs the clinical trial/observation study? does it make sense to deal with software issues as post-market problem reports? Does the manufacturer have any obligations about the trial?
In the regulatory world, the manufacturer is still responsible for their device, which includes the software. You can sell to whomever you want, but if you made it, it?s your responsibility to ensure it works as designed.

Are you asking what if the other organization does the clinical study on the device/software that they were sold? Or if they?re using the device/software to run a clinical study? These links may help answer either or both of those.

http://www.fda.gov/downloads/Medica...onandGuidance/GuidanceDocuments/UCM279103.pdf

http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm373750.htm

http://www.fda.gov/MedicalDevices/D...ubmissions/PremarketApprovalPMA/ucm050419.htm

http://www.fda.gov/MedicalDevices/D...YourDevice/InvestigationalDeviceExemptionIDE/

http://www.fda.gov/Training/CDRHLearn/ucm209135.htm

As this deals with some RUO and IUO products, it has some good info in it.
http://www.fda.gov/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm253307.htm
 
Thread starter Similar threads Forum Replies Date
B Software as a Medical Device - Language Requirements EU Medical Device Regulations 6
B Software as a NON-medical device Medical Information Technology, Medical Software and Health Informatics 21
A Medical Device Software POC Medical Device and FDA Regulations and Standards News 6
D One Software as Medical Device product or two? EU Medical Device Regulations 4
dgrainger Informational MHRA's Software and AI as a Medical Device Change Programme UK Medical Device Regulations 0
J Software as a Medical Device - SaMD IEC 62304 - Medical Device Software Life Cycle Processes 4
B A.I. diagnostic software is considered as medical device in FDA? US Food and Drug Administration (FDA) 6
V Medical Device Literature Translation Software ISO 13485:2016 - Medical Device Quality Management Systems 1
Q Software as a medical device vs software not sold as medical device: local regulations for sale? EU Medical Device Regulations 4
T IVDR Medical device software CE Marking (Conformité Européene) / CB Scheme 8
N ISO 13485 7.3.9 Change control in medical device software ISO 13485:2016 - Medical Device Quality Management Systems 6
V Software as medical device (SaMD) replicated for multiple clients through APIs IEC 62304 - Medical Device Software Life Cycle Processes 5
R Medical Device Software Certification IEC 62304 - Medical Device Software Life Cycle Processes 1
Q Storing and developing SAMD (Software as a Medical Device) in the Cloud IEC 62304 - Medical Device Software Life Cycle Processes 3
MDD_QNA Medical Device Software - Is a Help Button required? IEC 62304 - Medical Device Software Life Cycle Processes 1
F Software as a Medical Device (SaMD) Technical File Requirements Manufacturing and Related Processes 3
A Software as Medical Device (SaMD) definition and its applicability Other Medical Device and Orthopedic Related Topics 4
T First 510(k) submission - Class II software as medical device US Food and Drug Administration (FDA) 3
Z Software Library - could it be a medical device? ISO 13485:2016 - Medical Device Quality Management Systems 5
M Informational TGA – Submissions received: Regulation of software, including Software as a Medical Device (SaMD) Medical Device and FDA Regulations and Standards News 1
D Requirements for the dimensions / color of medical device labels - Software as a Medical Device IEC 62304 - Medical Device Software Life Cycle Processes 2
D Software as an accessory to a Class I medical device EU Medical Device Regulations 4
R Medical device software without user interface Other Medical Device and Orthopedic Related Topics 3
R Validation of Medical Device Hardware containing Software - How many to Validate ISO 13485:2016 - Medical Device Quality Management Systems 1
Ed Panek Rule 11 Question - CE approvals for software as well as the medical device EU Medical Device Regulations 7
D IEC 60601 for a Software only Medical Device? IEC 60601 - Medical Electrical Equipment Safety Standards Series 5
S Software Release Note - Class A stand alone software medical device IEC 62304 - Medical Device Software Life Cycle Processes 2
M Professional Use Medical Software French Labeling for Canada -- Not Considered Medical Device Canada Medical Device Regulations 2
S Medical Device Software Distribution for a CE-marked medical device via a USB memory stick EU Medical Device Regulations 8
M Informational TGA – Webinar: An update on the consultation for the Regulation of Software, including Software as a Medical Device (SaMD) Medical Device and FDA Regulations and Standards News 0
J Qualification of a Software as a Medical Device (SaMD) guidance under MDR EU Medical Device Regulations 9
P Software requirement and design specifications - Medical device contains both hardware and software IEC 62304 - Medical Device Software Life Cycle Processes 2
M Informational TGA Consultation: Regulation of software, including Software as a Medical Device (SaMD) Medical Device and FDA Regulations and Standards News 0
N Medical Device Accessory Classification - Software as a Medical Device Other Medical Device Regulations World-Wide 5
R SaMD - Software as a Medical Device - Software change control form ISO 13485:2016 - Medical Device Quality Management Systems 3
G EU MDR 2017/745 Rule 11 interpretation - Re-classification of a Software as Medical Device Other Medical Device Related Standards 0
M Medical Device News TGA – Regulation of Software as a Medical Device Medical Device and FDA Regulations and Standards News 0
S Two Questions about Medical Device Software 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 3
K Medical Device Software Update in Field of AIMD IEC 62304 - Medical Device Software Life Cycle Processes 1
M Medical Device News Health Canada - Scientific Advisory Panel on Software as a Medical Device (SAP-SaMD) - Record of Proceedings – January 26, 2018 Canada Medical Device Regulations 0
R Online / Cloud Based Software as Medical Device EU Medical Device Regulations 8
S DMR (Device Master Record) for Medical Device Software IEC 62304 - Medical Device Software Life Cycle Processes 3
S Labelling in Medical Device Software IEC 62304 - Medical Device Software Life Cycle Processes 8
S What is considered the complete software medical device? Medical Information Technology, Medical Software and Health Informatics 6
JoCam Medical Device Software - Apps which can control medical devices EU Medical Device Regulations 13
L Software Medical Device - 7.3.8 - Design and Development Transfer ISO 13485:2016 - Medical Device Quality Management Systems 4
A SOP for software validation of software in medical device IEC 62304 IEC 62304 - Medical Device Software Life Cycle Processes 5
K Registering a Software medical device (SaMD) in China China Medical Device Regulations 5
N Medical Device Software - Switch from Waterfall to Agile methodology Medical Information Technology, Medical Software and Health Informatics 4
V Software medical device labelling EU Medical Device Regulations 8

Similar threads

Top Bottom