SaMD Performance Testing

RedDevil_AK

Starting to get Involved
#1
Hi All,

We have a software intended for use in post-processing of MR images to do image segmentations, and provide the measurements derived from these segmentations. For our performance testing, the company wants to evaluate our output against the predicate device(s) output, ie, using the predicate device as the gold standard. I am not a fan of this approach, as the predicate device may have had an acceptance criteria for their output against manual segmentation (usual gold standard), and if we have an acceptance criteria against the predicate device's output; the output of our software may be some ways off of manual segmentation, which I think the FDA will frown on.

Has anyone had any experience in this field and how did you resolve it?
For performance testing, is it enough to prove that the performance of the software product is "as good as" the predicate device? The evaluating substantial equivalence (sorry, can't post links yet) guidance document does not provide an answer for or against.

Thanks!
 
Elsmar Forum Sponsor

ECHO

Involved In Discussions
#2
Hello!

I have done bench tests to show that our device is "as good as" our competitors (predicate device) in the past but this was mainly for marketing purposes. Some of these bench tests were apart of our verification but our documents never had a "this acceptance criteria was set because of our predicate device" statement in our documents. Do you happen to know why your company wants to do it this way?

For performance testing, is it enough to prove that the performance of the software product is "as good as" the predicate device?
I don't think so. Even if your device is a complete replica of the predicate device, I don't think it will be "enough". You could use it as a check to make sure you got everything covered.
 

RedDevil_AK

Starting to get Involved
#3
Thanks for the reply, ECHO!

Do you happen to know why your company wants to do it this way?
It is mainly because finding figures of merit and acceptance criteria for some of the output measurements has been difficult. There's not much literature out there for some of the measurements, and defining clinical relevance for the acceptance criteria has been tough.
 

Tidge

Trusted Information Resource
#4
As with @ECHO I've seen "non-inferiority" testing, but primarily for marketing reasons. We were still obligated to implement specific designs to sastisfy identified requirements... and have validated test methods and data sets to demonstrate the requirements were satisfied.
 

Junn1992

Quite Involved in Discussions
#5
seems like 'comparing to other product' equates to a "clinical comparison", but still leaves the question of pre-clinical testing open.
 
Thread starter Similar threads Forum Replies Date
A SaMD: Metric for release frequency EU Medical Device Regulations 4
A SaMD: Post Market Surveillance after Decomissioning EU Medical Device Regulations 2
R Cloud-based SaMD Validation IEC 62304 - Medical Device Software Life Cycle Processes 8
Ed Panek Consultant Request SaMD AI "Expert" FDA US Food and Drug Administration (FDA) 2
K SaMD and Customer Integrations Medical Information Technology, Medical Software and Health Informatics 3
C How to place software version for SaMD product in HIBC secondary data structure (UDI-PI)? Other US Medical Device Regulations 4
R SaMD Verification & Validation IEC 62304 - Medical Device Software Life Cycle Processes 6
T IEC 62304 : Risk control for SaMD IEC 62304 - Medical Device Software Life Cycle Processes 8
Z PMS Data collection for SAMD SaaS from clients EU Medical Device Regulations 3
L Essential Design Outputs for SAMD 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 2
A SaaS SaMD - Impact QMS ISO 13485:2016 - Medical Device Quality Management Systems 2
I Software (SaMD) mobile application verification testing: objective evidence Medical Information Technology, Medical Software and Health Informatics 2
J Software as a Medical Device - SaMD IEC 62304 - Medical Device Software Life Cycle Processes 7
H ISO 13485-paragraphs for a SaaS SAMD needed or not? ISO 13485:2016 - Medical Device Quality Management Systems 3
H Customer Specific Change in a SaaS SAMD IEC 62304 - Medical Device Software Life Cycle Processes 1
I Multiple Indications in SaMD Other US Medical Device Regulations 8
Y SaMD Verification and Validation SOP and Plan IEC 62304 - Medical Device Software Life Cycle Processes 8
V Software as medical device (SaMD) replicated for multiple clients through APIs IEC 62304 - Medical Device Software Life Cycle Processes 5
R SaMD - Placing on the Market before DoA EU Medical Device Regulations 2
D CE SaMD modular approach CE Marking (Conformité Européene) / CB Scheme 5
K Regulatory requirement of SaMD with machine learning component IEC 62304 - Medical Device Software Life Cycle Processes 3
Y Distributor for a SaMD Medical Information Technology, Medical Software and Health Informatics 3
Sravan Manchikanti How to interpret '8.3 Control of nonconforming product' for SaMD device while implementing ISO 13485 & MDSAP ISO 13485:2016 - Medical Device Quality Management Systems 7
Q Storing and developing SAMD (Software as a Medical Device) in the Cloud IEC 62304 - Medical Device Software Life Cycle Processes 3
A Demonstration of Equivalence - Need for comparing biological characteristics for an SamD EU Medical Device Regulations 1
N Servicing for SaMD ISO 13485:2016 - Medical Device Quality Management Systems 6
F Software as a Medical Device (SaMD) Technical File Requirements Manufacturing and Related Processes 3
A Software as Medical Device (SaMD) definition and its applicability Other Medical Device and Orthopedic Related Topics 4
N Agile SOP for SaMD IEC 62304 - Medical Device Software Life Cycle Processes 3
M Informational TGA – Submissions received: Regulation of software, including Software as a Medical Device (SaMD) Medical Device and FDA Regulations and Standards News 1
D Not a SaMD - How to be submitted for marketing authorization in EU and USA CE Marking (Conformité Européene) / CB Scheme 5
T Medical device or not - SaMD Other Medical Device and Orthopedic Related Topics 3
M Informational TGA – Webinar: An update on the consultation for the Regulation of Software, including Software as a Medical Device (SaMD) Medical Device and FDA Regulations and Standards News 0
J Qualification of a Software as a Medical Device (SaMD) guidance under MDR EU Medical Device Regulations 9
M Informational TGA Consultation: Regulation of software, including Software as a Medical Device (SaMD) Medical Device and FDA Regulations and Standards News 0
R SaMD - Software as a Medical Device - Software change control form ISO 13485:2016 - Medical Device Quality Management Systems 3
M Medical Device News Health Canada - Scientific Advisory Panel on Software as a Medical Device (SAP-SaMD) - Record of Proceedings – January 26, 2018 Canada Medical Device Regulations 0
K Registering a Software medical device (SaMD) in China China Medical Device Regulations 5
C Supplier Assessment for Hardware supporting SaMD ISO 13485:2016 - Medical Device Quality Management Systems 1
T MEDDEV 2.7/1 r4 vs. IMDRF/SaMD WG/N41 Other Medical Device Related Standards 3
M IMDRF document - Application of QMS to SaMD ISO 13485:2016 - Medical Device Quality Management Systems 2
W IEC 62304 vs. IMDRF SaMD Guideline Risk Class IEC 62304 - Medical Device Software Life Cycle Processes 5
E ISO 17025 Key Performance Indicators (KPIs) ISO 17025 related Discussions 5
L Supplier Performance when your supplier is also the customer ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 17
L Supplier performance evaluation Supplier Quality Assurance and other Supplier Issues 8
K SGS Certification - Advice on their Performance Service Industry Specific Topics 9
D IEC 60601-1-2: Is EMC immunity testing required for a device without essential performance? IEC 60601 - Medical Electrical Equipment Safety Standards Series 15
S Performance Qualification and Process Validation ISO 13485:2016 - Medical Device Quality Management Systems 5
Sidney Vianna IATF 16949 News IATF Performance Complaint Management System (IATF CMS) within the IATF Database IATF 16949 - Automotive Quality Systems Standard 4
malasuerte Performance of ISO 9001 certified vs. non-certified manufacturing companies ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 11

Similar threads

Top Bottom