AI/ML training dataset

[shimonv]

Hello everyone,

I'd appreciate your feedback on AI/ML training dataset:
1. Do we need a board-certified physician for labeling of training dataset?
2. Are there other requirements / consideration for preparing the training datasets, based on your experience with FDA?



Thank you!

Shimon

 

[Ronen E]

1. The quality of the training (and thus the AI) is only as good as the quality of the dataset expert grading - GIGO

2. Don't forget to leave a fresh dataset (not used in initial training) for validation of the AI model after it's locked.

^ FWIW

 

[QuinnM]

I can't speak to what will work with the FDA, and I need to read the read the new FDA PCCP guidance document, but I can tell you what we do. We have an annotation process and form that covers all aspects of the annotation/labeling activity. This includes requirements for individuals who are performing the annotation/labeling.

 

[shimonv]

I can't speak to what will work with the FDA, and I need to read the read the new FDA PCCP guidance document, but I can tell you what we do. We have an annotation process and form that covers all aspects of the annotation/labeling activity. This includes requirements for individuals who are performing the annotation/labeling.

Cool. So, what determines who does the annotation?

 

[Ed Panek]

Try to present the results based on typical FDA requirements: Sex, Age, Disease states, Etc
Make sure you positive predictive and negative predictive numbers meet your stated specification.
Be prepared with expert opinion to defend the values you've chosen for the model

 

[Ed Panek]

Almost half of FDA-approved AI medical devices are not trained on real patient data, research reveals

Artificial intelligence (AI) has practically limitless applications in health care, ranging from auto-drafting patient messages in MyChart to optimizing organ transplantation and improving tumor removal accuracy. Despite their potential benefit to doctors and patients alike, these tools have...

medicalxpress.com

Yikes

 

[QuinnM]

Cool. So, what determines who does the annotation?

We consider real world activities, that is who is doing this activity now with out AI? What are their qualifications and experience. We balance between who performs the annotation and QA on the annotation. Most, about 90%, are MDs, while others are certified in their field but may not be an MD. We also define years of experience in the specification too. The QA process has been executed by MDs.

 

[Ronen E]

Almost half of FDA-approved AI medical devices are not trained on real patient data, research reveals

Artificial intelligence (AI) has practically limitless applications in health care, ranging from auto-drafting patient messages in MyChart to optimizing organ transplantation and improving tumor removal accuracy. Despite their potential benefit to doctors and patients alike, these tools have...

medicalxpress.com

Yikes

Exactly my point - GIGO

Probably obtaining large-enough, high quality annotated datasets for training (and validation) is the biggest obstacle for developing really good AI expert systems.