Validation Methods of Machine learning and Artificial intelligence

patilrahuld

Involved In Discussions
Wondering if anyone of you can guide me in the right direction.

I am looking to study AI and Machine leaning validation techniques in pharma. i can find general information on AI/ML validation but not specific to pharma.

Also, if there a noted regulation on AI/ML validation in pharma?
 

Ed Panek

QA RA Small Med Dev Company
Leader
Super Moderator
I am familiar with AI and medical devices in the General Hospital category.

The FDA is evolving on this issue however, I suggest releasing a locked model for customers and not learning. If you try a learning adaptive model the FDA will probably water it down so much as to kill it. You also won't require PCCP or ACP documentation in the submission.

Make sure you keep your data sets separate (Training vs validation) and the data has demographics to demonstrate no bias.

We are working with an excellent consultant on this so I am learning every day from her. PM me for her details.

I strongly suggest obtaining the advice of a been there/done that type person for this as AI is so new to the FDA and they are very anxious about how to regulate it.
 

yodon

Leader
Super Moderator
FDA does seem to be coming to grips with ML / AI. I continue to see new products cleared by FDA.

Obviously, you want it continually learning but I think what @Ed Panek is suggesting with the locked down approach is to have a planned release cycle with the model and not lock it down forever.

FDA has is providing some help:
Be sure to post later on regarding any experiences you have. There are several of us on here that would benefit from more "real world data" (if you will).
 

Ronen E

Problem Solver
Moderator
Also, if there a noted regulation on AI/ML validation in pharma?
I'm not sure what you mean by "in pharma". I think that in most cases, the AI tool would be considered SaMD (so, a medical device), or (FDA-)unregulated SW. Are you referring to an AI tool that is intended to facilitate pharmaceutical decision-making or the likes? I think that would still qualify as a device. Anyway, I'm not aware of any FDA regulation or guidance targeting "pharma AI" (as they do for medical devices). However, pharma isn't my specialty and AI is a fairly new field.
I strongly suggest obtaining the advice of a been there/done that type person for this
I doubt that there are more than very few people who can rightfully claim that they've "been there" where "there" is "pharma AI".
have a planned release cycle with the model and not lock it down forever.
I think the main issue FDA is having with AI is the performance "unpredictability" of unlocked AI "running wild" in the field (how would we validate that the learning itself would never go rogue). No one is expecting the AI to be "locked forever" (actually, if the model can be improved through e.g. additional training, it'd be welcome I think); so when we talk about "locked AI" we mean that the learning (+ necessary validation of the updated model) only occurs at the Manufacturer's, while the model stays static in the field.
 

Ed Panek

QA RA Small Med Dev Company
Leader
Super Moderator
I'm not sure what you mean by "in pharma". I think that in most cases, the AI tool would be considered SaMD (so, a medical device), or (FDA-)unregulated SW. Are you referring to an AI tool that is intended to facilitate pharmaceutical decision-making or the likes? I think that would still qualify as a device. Anyway, I'm not aware of any FDA regulation or guidance targeting "pharma AI" (as they do for medical devices). However, pharma isn't my specialty and AI is a fairly new field.

I doubt that there are more than very few people who can rightfully claim that they've "been there" where "there" is "pharma AI".

good catch. Depending on the consultant, if you ask, "do you have experience in XYZ?" they will generally say "yes" regardless of their actual experience especially with something brand new like AI. No one is an AI expert yet. Its like hiking a trail in the woods. It takes years of hiking that trail in all weather conditions to state you are an expert. Make sure you measure that experience in practical terms.
 
Last edited:

Tidge

Trusted Information Resource
I am looking to study AI and Machine leaning validation techniques in pharma. i can find general information on AI/ML validation but not specific to pharma.

If the OP is asking more along the lines of "trying to study (in a specific field)" and less "specifically trying to release/leverage something in a regulated field" Then I suggest the following:
  • Do the studying of a wide variety of Machine Learning circumstances/applications.
  • Develop a nuanced appreciation of the particular area of interest, to see how the Machine Learning approach could help tackle which parts of that area of interest.
I can only speculate on what role ML might be playing in pharma; my own limited imagination has trouble guessing which possible uses would require "validation" in the classical sense. For example: any new treatments developed with the help of AI wouldn't be accepted just because an AI was validated, and my gut instinct is that it is unlikely that AI would be used to play a role in a manufacturing process (e.g. mixing).

As far as using an AI in the development of solutions: I expect the most difficult aspect would be deciding on the metric(s) for evaluating possible "solutions" to whatever the problem is... the "validation" would (IMO) likely be demonstration that the algorithm(s) have been implemented as required. Outputs of the AI would still require external analysis.

A very casual introduction to machine learning for anyone who is interested is Oliver Roeder's Seven Games: A Human History. I found the writing to be uneven, but there are enough pointers to different methods of Machine Learning (at least in what are essentially zero-sum scenarios of "win-or-lose", but there are applications to other areas) that the book still gets a soft recommendation from me. In order to get that information from the book, it is necessary to step between the histories of some personalities and his mostly personal takes on the games themselves. Roeder is a journalist, and it the book reads (to me) like it didn't get a particularly serious pass through an editor. Nevertheless, it offers accessibility on the topic for a broad swath of people.
 

patilrahuld

Involved In Discussions
I am familiar with AI and medical devices in the General Hospital category.

The FDA is evolving on this issue however, I suggest releasing a locked model for customers and not learning. If you try a learning adaptive model the FDA will probably water it down so much as to kill it. You also won't require PCCP or ACP documentation in the submission.

Make sure you keep your data sets separate (Training vs validation) and the data has demographics to demonstrate no bias.

We are working with an excellent consultant on this so I am learning every day from her. PM me for her details.

I strongly suggest obtaining the advice of a been there/done that type person for this as AI is so new to the FDA and they are very anxious about how to regulate it.


Hi Ed,


Thank you. i will appreciate if you can share any information/documents (non proprietary)? Sorry if i am asking anything that is not ideal but i am stretched in finding any guidance on such.

Rahul
 

Ronen E

Problem Solver
Moderator
Depending on the consultant, if you ask, "do you have experience in XYZ?" they will generally say "yes" regardless of their actual experience
:lmao::yes:
No one is an AI expert yet.
I wouldn't go as far as that. I worked for a company that developed (and is still improving) an AI-based SaMD almost two decades ago, and during that time had the good fortune of meeting a few people who - IMO - can rightfully be dubbed "AI experts". But I agree there aren't many of them yet.

I liked the hiking trail metaphor, having been a hiking guide in my 20s :)
 

Ronen E

Problem Solver
Moderator
I can only speculate on what role ML might be playing in pharma; my own limited imagination has trouble guessing which possible uses would require "validation" in the classical sense. For example: any new treatments developed with the help of AI wouldn't be accepted just because an AI was validated, and my gut instinct is that it is unlikely that AI would be used to play a role in a manufacturing process (e.g. mixing).
From my acquaintance with AI/ML in/as medical devices, they are mostly about making inferences from a big dataset (e.g. a medical image); too big for a human to analyse methodically and consistently (and non-intuitively, i.e. like a radiologist would observe an X-ray image). In that context, validating the AI does not equate validating the device (or the design). While the latter means building a solid (preferably as objective as possible) case that the device fulfills / will fulfil the user's needs, the former simply means that the AI will "spit out" the right answers ("right" in this case means close enough to what a human expert, e.g. a radiologist in the case of making inferences from a medical image, would "spit out"). Technically speaking, AI validation is similar to AI training, and to use a human parallel, it's like the test at the end of the course, that checks that the training was successful. The main difference is that in the training stage the errors are used for improving (the model), while in validation it's not done - there's only a "pass or fail" result. Of course this description is a little simplistic, but you get the gist I hope.

At this stage I don't think AI is seriously used for "developing new treatments" or the likes. I think it's just not there yet. Maybe it plays some modest role in that arena, but I think it's likely that wherever it's claimed to be more than that, it's mostly marketing inflation.
the book reads (to me) like it didn't get a particularly serious pass through an editor.
This seems to be the case with so many nonfiction books today, that would have otherwise been quite good. :(
 

Tidge

Trusted Information Resource
My reply was explicitly avoiding diagnostic(direct or assisted), which would be a medical device and not "pharma".

From my recollection, one of the later chapters in the Seven Games book did mention (casually) the potential application of one of the specific AI learning techniques in the development of new therapies, but it was pretty casual. It has been several decades since I did any investigation into machine learning (for problem solving, and not zero-sum attacks) and the aforementioned book didn't really offer much insight (IMO) into machine learning aside from the applications towards "solving" the titular games.
 
Thread starter Similar threads Forum Replies Date
C. Tejeda Process validation of rework assembly methods (medical devices) Medical Device and FDA Regulations and Standards News 3
P Test Method Validation (TMV) for all Measurement Methods in Rec/Inspection Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 1
A ISO 17025 - Methods validation and clients ISO 17025 related Discussions 3
S High voltage testing - ISO 17025 - 7.2.2 Validation of methods and 7.3 Sampling ISO 17025 related Discussions 3
A Validation of Calibration Methods and Traceability - ISO 17025 ISO 17025 related Discussions 3
D Do Test Methods completed to a standard require validation? ISO 13485:2016 - Medical Device Quality Management Systems 7
N Clean Room Validation and Certification Methods and Definitions ISO 13485:2016 - Medical Device Quality Management Systems 2
S Writing a Test Method Validation Procedure as well as Validation of Test Methods General Measurement Device and Calibration Topics 1
Q Verification / Validation methods - 8D (Eight Disciplines) process Nonconformance and Corrective Action 1
adztesla Sterile packaging validation for medical device - ASTMD4332 and ISTA2A ISO 13485:2016 - Medical Device Quality Management Systems 3
K Validation of cleaning, disinfection and sterilization method EU Medical Device Regulations 4
P Process Validation - OQ & PQ Other Medical Device and Orthopedic Related Topics 2
P Validation for blown film extrusion ISO 13485:2016 - Medical Device Quality Management Systems 0
V PQ parameters for computer system validation of configurable software of laboratory equipment Qualification and Validation (including 21 CFR Part 11) 0
M Number of samples for MDSW validation EU Medical Device Regulations 3
C Computerized System Validation in ISO 9001:2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
S Testing & Validation Phase Design and Development of Products and Processes 1
M Thread plug gauge validation/certification ASME vs Asian JJF standard Manufacturing and Related Processes 0
G PEMS Validation Plan IEC 62304 - Medical Device Software Life Cycle Processes 4
T What does "fully verify" mean exactly in process validation? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 5
P Do Barcode scanners need validation in Medical device manufacturing? Manufacturing and Related Processes 6
Mustapha FAI Template Validation ISO 13485:2016 - Medical Device Quality Management Systems 3
A Sample size selection for process validation - continuous data Reliability Analysis - Predictions, Testing and Standards 9
S Need guidance for Software validation Qualification and Validation (including 21 CFR Part 11) 5
M Sterilization Validation - Carryover Medical Device and FDA Regulations and Standards News 3
S Cleaning validation of single-use tips providing treatment of women’s intimate areas Manufacturing and Related Processes 3
T Verification vs. Validation AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 4
S Confection validation on different batch sizes Qualification and Validation (including 21 CFR Part 11) 0
E Sealing Validation for Medical devices Qualification and Validation (including 21 CFR Part 11) 5
R CNC process validation Manufacturing and Related Processes 1
A Has anyone implemented the Adobe Acrobat Sign Validation Pack to be 21 CFR Part 11 Compliant? ISO 13485:2016 - Medical Device Quality Management Systems 9
C Test Method Validation - ISO Standards Qualification and Validation (including 21 CFR Part 11) 1
J API Q1 - 5.7.1.5 - Validation of Processes for Production and Servicing Oil and Gas Industry Standards and Regulations 4
A Validation Plastic Injection Molding Process protocol ISO 13485:2016 - Medical Device Quality Management Systems 5
M Use of statistical techniques for Process Validation ISO 13485:2016 - Medical Device Quality Management Systems 9
M Risk-based approach to Test Method Validation for Design Verification? US Medical Device Regulations 5
K Design: Verification Vs Validation And Validation Vs Transfer ISO 13485:2016 - Medical Device Quality Management Systems 19
C Medical Device Gamma Irradiation Validation per VDmax25 (ISO 11137) Qualification and Validation (including 21 CFR Part 11) 1
A Human Factor Validation Human Factors and Ergonomics in Engineering 4
B Verification and Validation of calculations (FEA) in OCTG components. Oil and Gas Industry Standards and Regulations 9
B Validation of FEA Analyses in Oil&Gas Industries. There are a lot of guidelines for other activities. There is a similar proposal for O&G? Design and Development of Products and Processes 0
K Analytical Method Qualification Vs Validation expectations ISO 13485:2016 - Medical Device Quality Management Systems 1
B Validation of design for valve api 6d 25 edition ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 0
P Validation for RUO (Research Use Only) ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
B Hi , everyone i need a procedure for validation of design prototype api 6d (valve manufacturing) Oil and Gas Industry Standards and Regulations 1
Ed Panek Validation of Signature Software (Off the shelf) US Medical Device Regulations 6
Q Combining Validation Protocol & Report into 1 template Document Control Systems, Procedures, Forms and Templates 4
K ISO 17025 Method Validation and Verification for Test Lab ISO 17025 related Discussions 5
S Environment Monitoring System Validation ISO 14001:2015 Specific Discussions 1
B Supplier Evaluation report - Validation required or not ISO 13485:2016 - Medical Device Quality Management Systems 3

Similar threads

Top Bottom