SaMD Performance Testing

[RedDevil_AK]

Hi All,

We have a software intended for use in post-processing of MR images to do image segmentations, and provide the measurements derived from these segmentations. For our performance testing, the company wants to evaluate our output against the predicate device(s) output, ie, using the predicate device as the gold standard. I am not a fan of this approach, as the predicate device may have had an acceptance criteria for their output against manual segmentation (usual gold standard), and if we have an acceptance criteria against the predicate device's output; the output of our software may be some ways off of manual segmentation, which I think the FDA will frown on.

Has anyone had any experience in this field and how did you resolve it?
For performance testing, is it enough to prove that the performance of the software product is "as good as" the predicate device? The evaluating substantial equivalence (sorry, can't post links yet) guidance document does not provide an answer for or against.

Thanks!

 

[ECHO]

Hello!

I have done bench tests to show that our device is "as good as" our competitors (predicate device) in the past but this was mainly for marketing purposes. Some of these bench tests were apart of our verification but our documents never had a "this acceptance criteria was set because of our predicate device" statement in our documents. Do you happen to know why your company wants to do it this way?

For performance testing, is it enough to prove that the performance of the software product is "as good as" the predicate device?

I don't think so. Even if your device is a complete replica of the predicate device, I don't think it will be "enough". You could use it as a check to make sure you got everything covered.

 

[RedDevil_AK]

Thanks for the reply, ECHO!

Do you happen to know why your company wants to do it this way?

It is mainly because finding figures of merit and acceptance criteria for some of the output measurements has been difficult. There's not much literature out there for some of the measurements, and defining clinical relevance for the acceptance criteria has been tough.

 

[Tidge]

As with @ECHO I've seen "non-inferiority" testing, but primarily for marketing reasons. We were still obligated to implement specific designs to sastisfy identified requirements... and have validated test methods and data sets to demonstrate the requirements were satisfied.

 

[Junn1992]

seems like 'comparing to other product' equates to a "clinical comparison", but still leaves the question of pre-clinical testing open.

 

[RedDevil_AK]

Thank you, all!