SBS - The best value in QMS software

Interesting Discussion Sample Size Determination for Medical Device

david316

Involved In Discussions
#11
Some characteristics are subject to inherent variation from use conditions: for example, I can imagine that you need to prove that the hearing aid meets output requirements given a range of sound volume and frequency as well as ‘fit’ in the ear? In this case your sample size is more dependent on the range of each condition than anything else. The number of devices is less important than the number of conditions. Do you just test at the extremes and the nominal? Or across the range in a distribution of conditions that is representative of the conditions that exist in actual use? It is also dependent on whether or not the hearing aid is consistent in its performance across the range. In this case of design verify you can usually have a small number of devices tested across a full range of conditions. I typically would recommend 3 devices just to be safe. The ‘number of conditions tested on each device’ is then the real sample size.
If you only test 3 devices but over a range of conditions...how do you justify that the results from the 3 devices will generalize to all devices produced? i.e. manufactured devices will meet there specification over the conditions with a certain confidence and reliability?

Thanks
 
Elsmar Forum Sponsor

Bev D

Heretical Statistician
Staff member
Super Moderator
#12
If you only test 3 devices but over a range of conditions...how do you justify that the results from the 3 devices will generalize to all devices produced? i.e. manufactured devices will meet there specification over the conditions with a certain confidence and reliability?

Thanks
The short answer is conditional probability.

But there a few things to clarify. First when I say 3 units, it's 3 units at a min or max of the designed tolerances run or used at the worst case use conditions. So there may actually be more than simply 3 units depending on the complexity of the thing and the nature of the use.

I always think about this in terms of Development, OQ and PQ validation. In development we should be screening experiments, full factorials and response surface experiments to determine the critical (input) characteristics and what tolerances are required on those critical characteristics to guarantee that the output meets the requirements. If we utilize physics and good industrial experimentation we can design product that will have no or very few defects. OQ is used to confirm (or validate) that at the worst case tolerance levels and under worst case use conditions we have no or very few defects/failures. If you can meet requirements at the worst case conditions the nominal will certainly meet requirements. This is just physics. of course if you have defects at the worst case conditions you can use conditional probability to determine the actual failure rate. PQ is typically run at nominal. this is where you can and should use the standard sample size calculations as you can are creating things at manufacturing volumes.

I have found that the use of sample size calculations is needed when we haven't done a good job of design. The downside of these sample sizes is that they REQUIRE a uniformly distributed population from which to sample such that the sample is representative of the population. Of course if we haven't done a good job in development our knowledge of the physics is meager and our knowledge of the nature of the distribution of the population is also meager.

All of that said I know that many regulatory groups cling to the old fashioned random statistics and there may be little one can do to change their minds.
 

david316

Involved In Discussions
#13
It can get complicated quickly. If we consider the hearing aid example as you alluded to there may be a number of conditions over which the hear aid needs to fulfil it requirements. i.e. " I can imagine that you need to prove that the hearing aid meets output requirements given a range of sound volume and frequency as well as ‘fit’ in the ear? "

It assume it is very difficult to make a "worst case" hearing aid and you still need to prove the hearing aid meets output requirements over a range of volume setting etc. If environmental conditions affects the devices performance this further compounds the complexity of the problem. In the end the fall back is often to use random statistics but even this falls apart quite quickly because the combinations of tests ends up resulting in large sample sizes which are not manageable. In the end statistical compromises are made.....
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#14
but they don't have to. I suspect that you are confusing development work with validation... this is a sign of immaturity in the development process. think about automotive. they test a vehicle in Alaska and a vehicle in the Mohave desert. the entire system is tested against worst case environmental conditions.

this will sound bold but 'statistical compromises' are physics compromises. No amount of random sampling across nominal conditions can overcome a lack of characterization and knowledge of how the system is supposed to work and cannot predict or confirm adequate performance against requirements. Been in both places.
 

david316

Involved In Discussions
#15
I can understand determining worse case environmental conditions but do you need to test with a system at the worst case of its design. For example does the car have to be at "worst case" with every part made at the extremes of its allowed tolerances or do you test multiple cars (random sample) at worst case environmental conditions?

Consider the case where we are trying to prove the requirement that a car accelerates to at least 100 km/hr in less than 10 seconds over its intended operating environmental conditions.

Thanks
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#16
Complex devices are tested at worst case levels at the component level before being integrated at the system level (in best case development). so the components in a car for example have been put through their paces before a random pilot car is tested under worst case conditions. Not all development is perfect the example is intended to point out the utility of testing under worst case conditions - physics can make random sampling unnecessary. we must use both knowledge of the system and the physics as well as probability to inform sample sizes. Random sampling to find random variation is simply highly inefficient and can provide a false sense of security.

In the last example of acceleration, I would use a vehicle that has worst case (allowable by the specification tolerances) components to see if it can accelerate under the worst case intended conditions. on the other hand I've never seen a critical requirement for acceleration except in race cars for which there are only 1 for each driver...
 

david316

Involved In Discussions
#17
"In the last example of acceleration, I would use a vehicle that has worst case (allowable by the specification tolerances) components to see if it can accelerate under the worst case intended condition "

Wouldn't it be impractical to make a "worst case" car. Especially when you consider the electronic components and sensors that presumably effect performance. Personally the idea of testing a worst case design appeals but at the system level I struggle to see how this can be practically done.
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#18
Not as difficult as you would think. During development we are creating worst case components. In medical devices we are supposed to this as part of OQ. There are some things that difficult to get at worst case levels (like molded parts from a worn mold) and we have to make decisions based on physics.

your argument for the difficulty of making worst case cars is exactly why random sampling is silly. you can random all you want but until you make a large number of cars that have parts that have naturally varied in their inputs you won't randomly get the full range of variation. this scenario isn't design verify or design validation, it's just long term monitoring of performance. in validation and verify you are supposed to be validating that the full range of allowable variation will still meet requirements - how can you do that if you don't create the full range of variation?
 

david316

Involved In Discussions
#19
" in validation and verify you are supposed to be validating that the full range of allowable variation will still meet requirements - how can you do that if you don't create the full range of variation? "

I believe it is common to use a risk based approach. For example, it might be deemed acceptable to be 99% confident with 99% reliability that a part meets it spec based on the risk.
 
Thread starter Similar threads Forum Replies Date
C AS9138 Sample Size Determination AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 1
V When and How to apply the Bayes Success Run approach for Sample Size Determination Inspection, Prints (Drawings), Testing, Sampling and Related Topics 4
Statistical Steven Continuous Data Analysis Sample Size Determination Six Sigma 3
A Destructive Test Sample Size Determination Statistical Analysis Tools, Techniques and SPC 4
P Repetition (sample size) for measuring equipment accuracy determination General Measurement Device and Calibration Topics 1
K Sample Size Determination - Medical Devices Inspection, Prints (Drawings), Testing, Sampling and Related Topics 4
S Determination of sample size for product audit IATF 16949 - Automotive Quality Systems Standard 1
C Stress / Challenge Conditions for Design Verification Testing to Reduce Sample Size 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 11
K How can Acceptance Number and Reject Number be larger than Sample Size on Z1.4 Table? Inspection, Prints (Drawings), Testing, Sampling and Related Topics 8
J Validation Sample Size for Tray Seal Qualification and Validation (including 21 CFR Part 11) 1
D Design Verification Sample Size vs Repeats Statistical Analysis Tools, Techniques and SPC 9
M Minimum sample size - Guidance and statistical rationale Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
P Unrealistic Packaging Validation Sample Size 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 13
J Control chart for huge sample size Statistical Analysis Tools, Techniques and SPC 9
S Is there a standard for sample size during R&D phase Other ISO and International Standards and European Regulations 18
J Sample size definition in an Automotive SMT pilot lot run Misc. Quality Assurance and Business Systems Related Topics 1
E Correct way to certify hydrostatic testing when it is not 100% (and Sample Size) Various Other Specifications, Standards, and related Requirements 6
B Determining sample size for device sterility Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
D Determining of sample size for 'Operational Qualification' AQL - Acceptable Quality Level 5
F Bayes Success run Theorem for sample size during OQ&PQ Qualification and Validation (including 21 CFR Part 11) 4
I Device modifications - Clinical sample size rationale EU Medical Device Regulations 5
J Sample size for creating a data base as a reference to a tested variable Other Medical Device and Orthopedic Related Topics 6
T Sample Size for Design Validation Design and Development of Products and Processes 4
V Acceptable maximum RSD (relative standard deviation) for an sample size Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 1
E Sample size for design verification of variable in single use device Design and Development of Products and Processes 20
R Sample size for clinical validation/investigation EU Medical Device Regulations 4
R Process Validation sample size selection Statistical Analysis Tools, Techniques and SPC 0
P Is it possible to make an educated decision using a very very small sample size? Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
F Sample Size for Biocompatibility Tests Other Medical Device Related Standards 4
V Firmware Verification Testing & Sample Size Software Quality Assurance 1
R Determining Sample Size for Medical Device Component Validation Inspection, Prints (Drawings), Testing, Sampling and Related Topics 0
S Surveillance Sampling Test - Determining Sample Size Inspection, Prints (Drawings), Testing, Sampling and Related Topics 5
H Sample Size Calculation (Confidence Interval and Reliability) - Medical Devices Reliability Analysis - Predictions, Testing and Standards 6
P Cleaning and Disinfection Validation Sample Size of 3? Other Medical Device Related Standards 2
V Sample Size Calculation for Image Analysis - Microscopy Inspection, Prints (Drawings), Testing, Sampling and Related Topics 2
G Sample Size, Significant Figures, Scale General Measurement Device and Calibration Topics 3
L Heated Sealed Packages - Sample Size for OQ (Operational Qualification) and PQ Inspection, Prints (Drawings), Testing, Sampling and Related Topics 11
P Frequency and Sample Size Requirements for an MSA Studies Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 4
N Sample Size Justification for Medical Device Shelf-Life Inspection, Prints (Drawings), Testing, Sampling and Related Topics 13
C Process Qualification Sample size Document Control Systems, Procedures, Forms and Templates 6
G Sample size for IEC 60601-1-11 environmental tests IEC 60601 - Medical Electrical Equipment Safety Standards Series 1
G IEC 60601-1-2 Sample Size issue Other Medical Device Related Standards 1
A How to calculate Mean and standard deviation without sample size in Minitab Using Minitab Software 5
D What is the minimum Sample Size for Weibull Analysis Reliability Analysis - Predictions, Testing and Standards 12
C Sample Size for Stability Testing Other Medical Device and Orthopedic Related Topics 2
M Calculating Cpk when sample size equals to 1 Capability, Accuracy and Stability - Processes, Machines, etc. 12
F Power and sample size for factorial design Using Minitab Software 4
S Sample Subgroup Size for NP/P chart - nPbar>=5 Statistical Analysis Tools, Techniques and SPC 4
U Sample size to prove parts are good with 99.73% confidence Inspection, Prints (Drawings), Testing, Sampling and Related Topics 4
F Sample Size to get Statistically Valid Data Measurement Uncertainty (MU) 6

Similar threads

Top Bottom