Statistical Analysis to Demonstrate Substantial Equivalence to the FDA

Chrisx

Involved In Discussions
#1
Does anyone know if FDA will accept statistical analysis like a t-test to demonstrate that data from performance testing is substantially equivalent to the predicate device. For example, I might find that test results on the predicate device have a higher average value then the subject device. However, this difference might just be noise and not statistically significant. Could I claim with a certain confidence interval (say 95%) that the two are not different?

Thanks
 
Elsmar Forum Sponsor

Ronen E

Problem Solver
Staff member
Moderator
#2
Does anyone know if FDA will accept statistical analysis like a t-test to demonstrate that data from performance testing is substantially equivalent to the predicate device. For example, I might find that test results on the predicate device have a higher average value then the subject device. However, this difference might just be noise and not statistically significant. Could I claim with a certain confidence interval (say 95%) that the two are not different?

Thanks
Hello and welcome to the cove :bigwave:

There are actually 2 questions here:
1. Is the use of statistical techniques acceptable through the 510(k) review?
2. Can performance data be relied upon to substantiate equivalence with a predicate?

My answers:

1. As a general notion, yes, though you'd have to make sure your analysis is rigorous and your reasoning is robust (not sure that a t-test alone would make the case). There are a few professional statisticians around here and they may shed more light on the statistical aspect of your specific situation.

2. I would not take it as my first-choice route. Substantial equivalence, in crude terms, is determined by intended use and technological characteristics. I would resort to performance data only if there are technological differences and I was trying to argue that there are no new concerns regarding safety and effectiveness (even then, that would not be my preferred argument). To that end the data should, IMO, be extensive and the analysis pretty rigorous.

Cheers,
Ronen.
 

Chrisx

Involved In Discussions
#3
Thanks for the info. I tried to keep the post brief and left out a lot of details. You are correct.

1. Yes, a t-test by itself would probably not be sufficient. It would probably be necessary to also show that the data is normally distributed (anderson-darling plot) and also justify the sample size.

2. It seems that for the types of devices we manufacture performance testing is the norm. FDA guidance for these types of devices describes expectations for performance testing.

Thanks,
ChrisX
 

Statistical Steven

Statistician
Staff member
Super Moderator
#4
Thanks for the info. I tried to keep the post brief and left out a lot of details. You are correct.

1. Yes, a t-test by itself would probably not be sufficient. It would probably be necessary to also show that the data is normally distributed (anderson-darling plot) and also justify the sample size.

2. It seems that for the types of devices we manufacture performance testing is the norm. FDA guidance for these types of devices describes expectations for performance testing.

Thanks,
ChrisX
Typically you can use a t-test or ANOVA to show equivalence, though you should use a test of non-inferiority (Blackwelder) to be rigourous about your analysis.
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#5
depending on the device a 'matched pair' test may be more powerful in showing tha the devices are not substantially different. Beta risk is more important in this type of test than alpha risk...the term 'substantial' in the phrase 'no substantial difference'* is related to the size of the difference. There is always a mathematical difference between two things - the answer you must assure the FDA of that that difference is probably sampling error and if not then it is so small as to make no material difference in the efficacy adn/or hazard of the device.


*I believe this is the actual phrase the FDA uses?
 

Ronen E

Problem Solver
Staff member
Moderator
#6
depending on the device a 'matched pair' test may be more powerful in showing tha the devices are not substantially different. Beta risk is more important in this type of test than alpha risk...the term 'substantial' in the phrase 'no substantial difference'* is related to the size of the difference. There is always a mathematical difference between two things - the answer you must assure the FDA of that that difference is probably sampling error and if not then it is so small as to make no material difference in the efficacy adn/or hazard of the device.


*I believe this is the actual phrase the FDA uses?
The latest FDA (draft) guidance regarding evaluation of Substantial Equivalence can be found here:

http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm282958.htm

Unfortunately it doesn't contain the string "statistic" at all.

Since the OP didn't specify the device's product code or the related guidance document, I can't relate to FDA's expectations beyond any generic ones.

Edit: The guidance in the link I provided does say, on p. 18:

If FDA determines that there are differences in the technological characteristics of the new device and the predicate device, FDA reviews and evaluates all relevant information bearing on any such differences in technological characteristics to determine whether they raise different questions of safety and effectiveness for the new device as compared to the predicate device
and in a related footnote:

Manufacturers should be prepared to provide appropriate performance data to address any differences, even ones that appear to be minimal, that could affect safety and effectiveness to demonstrate that the new device is as safe and effective as the predicate device.
IMO, that means:

(a) Performance data is relevant if there are technological characteristics that may raise different questions of safety and effectiveness.

(b) The above mentioned data is not expected in the submission, but should be available upon the reviewer's request.

(c) The data should demonstrate that the new device is as safe and effective as the predicate device, but no specific method is specified or even hinted.

More details on performance data that may be required are available in section F. Requests for Performance Data (p. 19).
 
Last edited:
Thread starter Similar threads Forum Replies Date
S How to perform verification of the Statistical Analysis Software? Qualification and Validation (including 21 CFR Part 11) 2
A Statistical Analysis - Check if these organisms at different concentrations affect the growth of wheat seedlings Using Minitab Software 4
Y Statistical Analysis of Road Traffic Data Statistical Analysis Tools, Techniques and SPC 11
J Capability Analysis - Unusual Statistical Distribution of my Proccess Capability, Accuracy and Stability - Processes, Machines, etc. 5
M PhD Thesis Data Statistical Analysis Methods Statistical Analysis Tools, Techniques and SPC 2
S Minitab and Crystal Ball Statistical Analysis Software Using Minitab Software 13
J Applicable Statistical Analysis Method for this Data Six Sigma 17
I SPC Statistical Analysis in Metrology General Measurement Device and Calibration Topics 2
A What is Monte Carlo Simulation / Distribution in Statistical Analysis? Statistical Analysis Tools, Techniques and SPC 19
C Applying Statistical Analysis Methods to Work Safety Occupational Health & Safety Management Standards 3
S Statistical Analysis of Survey Results Quality Tools, Improvement and Analysis 8
H What is 99/90 Statistical Analysis Statistical Analysis Tools, Techniques and SPC 4
I Using validated SAS JMP to validate Excel for Statistical Analysis? Qualification and Validation (including 21 CFR Part 11) 11
O What Statistical Analysis Technique to use for Batches? Statistical Analysis Tools, Techniques and SPC 6
P Warranty Statistical Analysis in a Manufacturing Company Quality Tools, Improvement and Analysis 1
B Appropriate Statistical Analysis Technique question Statistical Analysis Tools, Techniques and SPC 4
smryan Count by weight statistical analysis Quality Tools, Improvement and Analysis 24
M SPC (Statistical Process Control) analysis and status on two manufacturing plants Statistical Analysis Tools, Techniques and SPC 4
B Statistical Analysis SPC Trend explanation needed Statistical Analysis Tools, Techniques and SPC 4
R Statistical Analysis Problem - Pooling Information for 8 Hour Shifts Statistical Analysis Tools, Techniques and SPC 5
C Statistical Analysis during Document Audit US Food and Drug Administration (FDA) 8
C Statistical Analysis of all Key Processes Statistical Analysis Tools, Techniques and SPC 1
M Statistical Analysis in Low Quantity Production of 10 to 20 products each year Six Sigma 4
antoine.dias Definition Rv and RvK in Statistical Analysis - Inner diameter between the bearing rollers Definitions, Acronyms, Abbreviations and Interpretations Listed Alphabetically 1
J Software Validation - SigmaStat for Statistical Analysis ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
samer Statistical Analysis for Sales Process? Can you help? Statistical Analysis Tools, Techniques and SPC 11
J Statistical Data Analysis - How to satisfy ISO 9001 : 2000, Clause 8 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 4
Q Teaching Executives about the Benefits of Statistical Analysis and Problem-solving Training - Internal, External, Online and Distance Learning 20
A Shainin vs. Fisher vs. Taguchi DOE (Design of Experiments) Statistical Analysis Quality Tools, Improvement and Analysis 10
Y Statistical Analysis applications in Human Resources Statistical Analysis Tools, Techniques and SPC 8
S Statistical Process Control and the need for Measurement System Analysis (Gage R&R) Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 5
R Analysis of Data - The former "Statistical Techniques" procedure ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 4
T Any recommendation on DOE/Statistical Analysis Software? Statistical Analysis Tools, Techniques and SPC 1
M V&V phase: Justification of acceptance criteria (statistical method ) - (Medical Device) Design and Development of Products and Processes 2
Bev D Statistical Alchemy Misc. Quality Assurance and Business Systems Related Topics 1
K Looking for guidance to write an SOP on Statistical Methodologies? Statistical Analysis Tools, Techniques and SPC 7
M Minimum sample size - Guidance and statistical rationale Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
N Design Verification & Process Validation - Statistical sample sizes Design and Development of Products and Processes 2
John Predmore Interactive visualization through graphical simulation of statistical concepts Statistical Analysis Tools, Techniques and SPC 3
Bev D Profound Statistical Concepts Misc. Quality Assurance and Business Systems Related Topics 63
H Statistical Techniques Procedure - What should be included Document Control Systems, Procedures, Forms and Templates 4
O Statistical justification of sampling size in V&V tests ISO 13485:2016 - Medical Device Quality Management Systems 5
optomist1 It’s time to talk about ditching statistical significance Statistical Analysis Tools, Techniques and SPC 6
Marc Steve Prevette's Statistical Process Control (SPC) "Library" Statistical Analysis Tools, Techniques and SPC 0
John Predmore A Balanced view of statistical tests Statistical Analysis Tools, Techniques and SPC 3
V Statistical basis and justification while comparing / changing sampling plans Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 11
S SPC (Statistical Process Control) for Unilateral Tolerance - Questions Statistical Analysis Tools, Techniques and SPC 6
S IATF 16949 9.1.1.3 Application of statistical concepts - Our technicians are quizzed for statistical knowledge IATF 16949 - Automotive Quality Systems Standard 3
K Please help identify appropriate statistical treatment Statistical Analysis Tools, Techniques and SPC 13
ScottK Statistical basis for 30 pieces for FAI 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 7

Similar threads

Top Bottom