FDA 21 CFR 820.250 - Does "valid statistical" always mean math?

Mark Meer

Trusted Information Resource
21 CFR 820.250 (Statistical Techniques) uses the term "valid statistical rationale".

I'm wondering if this necessarily means being able to back sampling plans with statistical formulae? ...or can the "statistical rationale" be indirect, say, from published research?

For example, Faulkner (Behavior Research Methods, Instruments, & Computers 2003, 35 (3), 379-383), conducts a study examining the correlation between number of subjects in a usability study, and the number of problems identified.

Can citing a study like this be an acceptable basis (according to FDA) for developing a human-factors study sampling plan? The FDA themselves cite it in their "Draft Guidance for Industry and Food and Drug Administration Staff - Applying Human Factors and Usability Engineering to Optimize Medical Device Design".

However, my concern is that, if scrutinized by the FDA, such a rationale would be difficult to argue, given that it is based on analysis of data gathered from a single study, and the author's conclusions. ...and not on established statistical methods...
 

Statistical Steven

Statistician
Leader
Super Moderator
Not only does valid statistical rationale NOT mean formulas, in many cases its a rationale why statistics were not applied. For example, your human factors study sampling plan might reference HF75 Appendix A. When a standard exists that references a sample size, that is acceptable rationale for the sample size selection. You might also choose to have a sample size of 3 based on very low variability (justification is just a standard deviation of historical data). What they don't want to see is you do not understand WHY you chose the sample size. for example, we used Z1.4 normal inspection level II without any justification why that is the appropriate sampling plan for the problem at hand.



21 CFR 820.250 (Statistical Techniques) uses the term "valid statistical rationale".

I'm wondering if this necessarily means being able to back sampling plans with statistical formulae? ...or can the "statistical rationale" be indirect, say, from published research?

For example, Faulkner (Behavior Research Methods, Instruments, & Computers 2003, 35 (3), 379-383), conducts a study examining the correlation between number of subjects in a usability study, and the number of problems identified.

Can citing a study like this be an acceptable basis (according to FDA) for developing a human-factors study sampling plan? The FDA themselves cite it in their "Draft Guidance for Industry and Food and Drug Administration Staff - Applying Human Factors and Usability Engineering to Optimize Medical Device Design".

However, my concern is that, if scrutinized by the FDA, such a rationale would be difficult to argue, given that it is based on analysis of data gathered from a single study, and the author's conclusions. ...and not on established statistical methods...
 

Mark Meer

Trusted Information Resource
...your human factors study sampling plan might reference HF75 Appendix A

What is "HF75"?

When a standard exists that references a sample size, that is acceptable rationale for the sample size selection.

I'm not sure what suffices as a "standard" in this case? For example, is an FDA publication a "standard"? Or a research paper (such as the one cited)?
 

Statistical Steven

Statistician
Leader
Super Moderator
What is "HF75"?

Ronen answered this for you.

I'm not sure what suffices as a "standard" in this case? For example, is an FDA publication a "standard"? Or a research paper (such as the one cited)?

Any FDA publication including guidance documents are considered a standard. Other standards would be AAMI, ASTM, ISO, BS, IEC, etc. A research paper is not a standard, but can be referenced as justification for your decision.
 

Mark Meer

Trusted Information Resource
Any FDA publication including guidance documents are considered a standard.

Too bad Applying Human Factors and Usability Engineering to Optimize Medical Device Design is only a draft guidance.

It's a pretty straight forward a prescription:
"For devices intended to be used by more than one group of users that have distinct abilities or use roles, at least 15 participants from each group should participate in validation testing."

What I think warrants discussion is how the FDA justifies their own sampling plans such as this. ...apparently from a single study which made it's conclusions based on data from 60 subjects?

I wonder how likely this would have been accepted as sufficient sample-size justification to the FDA before it was referenced in their draft guidance? ...I suspect it wouldn't have flown...

...but anyone else successfully justifying sampling-plans based on cited research, please weigh-in!
 

Ronen E

Problem Solver
Moderator

Based on several past discussions in the Cove, FDA's practice is to treat their draft guidance as final guidance from the date of publication and until it is officially withdrawn or superseded by a final guidance or a more updated draft. Of course relying on this is a bit risky because FDA can always point that it is "just a draft" - at their convenience, but I think they only resort to that in extreme cases. Mind you, some FDA draft guidance has been such for years (no further versions issued nor the guidance withdrawn), which didn't disturb anyone, including the FDA, using it as an authoritative tool (as much as guidance is considered authoritative).

What I think warrants discussion is how the FDA justifies their own sampling plans such as this. ...apparently from a single study which made it's conclusions based on data from 60 subjects?

One important thing to keep in mind regarding sample size decisions is that cost usually plays an important role. In most commercial contexts it is a matter of balancing statistical rigor and cost. AFAIK this approach is at the core of Z1.4. I assume that the FDA closed-in on the figure you quoted based on a combination of pure statistical considerations / research, and economical ones. Even FDA understands that usability trials are expensive, and they have this motto of aiming to avoid over-burdening manufacturers and hindering innovation/progress.

Cheers,
Ronen.
 
Top Bottom