Regulatory requirement of SaMD with machine learning component

[Kynetyk]

Greetings everyone,

I am working on a software as medical device that has a machine learning component to it (eg. Tensorflow model). In short, it is an image processing application that evaluates the volume of anatomical structures from medical scans to assist diagnosis.

I'm currently following IEC62304 for the "model code" (model weights + inference code), however I'm having a hard time finding information on how I should treat the "training code" (dataset management, model fitting, model evaluation), since it is not part of the SaMD per say.

I've read in a Johner-Institute blog post on machine learning that the "training code" should satisfy ISO-13485:2016 requirements regarding "computerized system validation" (GAMP5-Category 5). I'm assuming that it refers to section 7.5.6 (Validation of processes for production and service provision), however that section also state that it applies to processes "where the resulting output cannot be verified by subsequent monitoring or measurement [...]". In my case, I can still evaluate the performances of the machine learning component independently of the "training code".

Do you have any thoughts on this ? Any experience/strategy regarding ML and SaMD?

 

[yodon]

While these are FDA guidance docs, you might be able to get some good, general info here:

• Premarket
• Postmarket

 

[Ed Panek]

As a person familiar with training networks to play chess, as you have stronger and stronger networks, it is possible to actually weaken the result with more training. The network with a low learning rate can fall into local minima and be stuck there and without lots of experimenting, it won't be known.

 

[jerjunnluh]

I believe the blog you mentioned is titled as "Validation of Machine Learning Libraries"

In the blog, it said:” If the code for this data pre-processing and the training of the model is not part of the medical device (this is usually the case), IEC 62304 is not(!) applicable. This applies both for the development of this code as well as the use of existing machine learning libraries (SOUP).” That means the model libraries you created (CNNs, RNNs, or other models) can be treated as a common libraries like other libraries you call in your main program.

However, in the same blog it also stated “ The trained model now becomes part of a (medical) device together with the machine learning library (or part of it). This component is now responsible for making correct predictions, e.g., for correctly classify images.

……

IEC 62304 rightly calls this the verification of the SOUP/component, even if the term validation is used more often when talking about libraries.
This verification/validation step involves demonstrating that the component makes the correct prediction with respect to the trained model.”



If you want to evaluate the performances of the machine learning the volume of anatomical structures from medical scans, you still need to follow the process of IEC 62304 to validate your software.



Hope it might help you.