Determining Sample Size for Design Verification and Design Validation

B

Burnett

#11
Re: Sample size for Design Verification / Design Validation

Confirming that various performance metrics of the device will be within associated USL/LSL's. For example to confirm for a button push force requirement of 1.0 +/- 0.1 lb I would measure X button samples and confrim that the upper and lower limits from the measurements are within USL of 1.1 and LSL of 0.9. I would plan the test by estimating mean and stdev to compute sample size n based on confidence level (95%) and tolerance interval (99%). Let me know if that's not enough info.
 
Elsmar Forum Sponsor
B

Barbara B

#12
Re: Sample size for Design Verification / Design Validation

There are two completely different meanings of 99% in statistical tolerance intervals and in sample size calculation:
  • stastistical tolerance: 99% = coverage of the population (with respect to the confidence level of e.g. 95% for the estimated mean and standard deviation)
  • sample size calculation: 99% = reliability / power of a statistical test to detect a change in a parameter, e.g. a shift +/-d (critical difference) from the mean in the sample
Sample size calculations are used to assure a specific power and to avoid wrong decisions, e.g. not to detect a shift of +/-d in mean. In a sample size calculation you want to get the number of samples necessary to assure a specific confidence level (1-alpha) and a specific power (1-beta), with
  • alpha: maximum acceptable false alarm rate (type I error), false alarm: test decision is "process has changed" but the process has NOT changed in reality, e.g. decision "mean has changed more than +/-d" when in fact the mean is the same as in the planning
  • beta: maximum acceptable non-detecting rate (type II error), non-detecting rate: test decision is "process has NOT changed" but the process has changed in reality, e.g. "mean has NOT changed more than +/-d" when in fact the mean has changed more than +/-d compared to the planning values

Since the type II error risk (beta) isn't considered in statistical tolerancing this method can't be used to calculate a sample size.
 
B

Burnett

#13
Re: Sample size for Design Verification / Design Validation

Thanks for the detailed response.

So I have to go back to the basic question: If statistical intervals are not a typical V&V approach, what is the typical approach? In other words what basic method should I be using for Design Verification and with what statistical method, with the objective being to verify that the design will produce a device that consistenly meets its specifications?
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#14
Re: Sample size for Design Verification / Design Validation

So what Barbara posted is correct IF you are looking to ensure that no substantial shift in MEANs happened. However, for many processes we can tolerate mean shifts as long as the parts are still in spec. (not to mention that many processes are simply non homogenous and there are 'statistically significant' differences in lot means simply because the factors that affect the mean are not the same factors that affect the standard deviation.

So we must first be sure we are asking the correct question. IF you are really only concerned that the change hasn't driven the process to making parts out of spec then your approach makes sense. 'Power' or Beta risk can be dealt with by replication - multiple lots of the current and new process.

The dilemma we face is that each situation is potentially different - there really is no cook book answer. So in order to provide a better answer we would have to have more details about your process, the change, and some data on how the current process is running.
 
L

LoneQC

#15
Re: Sample size for Design Verification / Design Validation

My comment concerns design verification ONLY.

What I see is that for design verification, one has, at least in our case, a limited number of "near production" devices
I'm in same situation. I have recently inherited QA role and I'm trying to stumble my way through the statistics and sample size.

Our current procedures utilizes combination of Bayes Success Run chart, AQL chart, and LTPD charts. I've been reading Juran's handbook, Guide to Acceptance by Wayne Taylor, along with whatever I can find online. It seems most of these plans make an assumption that lot size is 10x sample size and seems to apply mostly to processes and ongoing manufacturing. Couple of our projects have limited # of units, and sample sizes from these type of charts are not achievable.

Some plans also mention STI (statistical tolerance intervals) within spec? I believe this allows for fewer samples to used with distribution software (Distribution Analyzer or StatGraphics) to show small sample is within specified confidence/reliability interval without justification for # samples used.

For example, Our design verification plans simply increase confidence/reliability requirements with increasing risk..
risk 1 - 90/95
risk 2 - 95/97
risk 3 - 95/99

for 95 confidence/95 reliability w/ variable data
-LTPD (5% defective) chart shows n=20, Ppk=0.81, Pp=0.87..
- software requires min 8 samples to perform calculations
Looks like Bev D mentioned this over here
"Qualification and Validation (including 21 CFR Part 11)" (can't post link)

Not sure this is valid or how to justify..

Or how to reduce sample size for attribute data.
95/97 on Bayes-Success , n = 98
somehow make it a variable type of data or 100% inspection of a much smaller sample??


I seen some sample size calculations for FDA Verification and Validation posted by Bev D, "Determining Sample Size for FDA Verification and Validation Activities", but we do not have a delta, new product, no historical data to compare


Are there any charts or sampling plans that focus on design validation/verification rather than process and mfg?

Thanks
 

Ronen E

Problem Solver
Staff member
Moderator
#16
Re: Sample size for Design Verification / Design Validation

I'm in same situation. I have recently inherited QA role and I'm trying to stumble my way through the statistics and sample size.

Our current procedures utilizes combination of Bayes Success Run chart, AQL chart, and LTPD charts. I've been reading Juran's handbook, Guide to Acceptance by Wayne Taylor, along with whatever I can find online. It seems most of these plans make an assumption that lot size is 10x sample size and seems to apply mostly to processes and ongoing manufacturing. Couple of our projects have limited # of units, and sample sizes from these type of charts are not achievable.

Some plans also mention STI (statistical tolerance intervals) within spec? I believe this allows for fewer samples to used with distribution software (Distribution Analyzer or StatGraphics) to show small sample is within specified confidence/reliability interval without justification for # samples used.

For example, Our design verification plans simply increase confidence/reliability requirements with increasing risk..
risk 1 - 90/95
risk 2 - 95/97
risk 3 - 95/99

for 95 confidence/95 reliability w/ variable data
-LTPD (5% defective) chart shows n=20, Ppk=0.81, Pp=0.87..
- software requires min 8 samples to perform calculations
Looks like Bev D mentioned this over here
"Qualification and Validation (including 21 CFR Part 11)" (can't post link)

Not sure this is valid or how to justify..

Or how to reduce sample size for attribute data.
95/97 on Bayes-Success , n = 98
somehow make it a variable type of data or 100% inspection of a much smaller sample??


I seen some sample size calculations for FDA Verification and Validation posted by Bev D, "Determining Sample Size for FDA Verification and Validation Activities", but we do not have a delta, new product, no historical data to compare


Are there any charts or sampling plans that focus on design validation/verification rather than process and mfg?

Thanks
Hello and welcome to the Cove :bigwave:

Not a statistician (hopefully they will chime in soon), but FWIW:

1. Please note that design verification is about "demonstration", not about "proof". Proof comes at the validation stage. By that time you may have more units available for testing (as well as more stable processes), and statistical rigor is expected. For the verification stage you might get away with lesser statistical rigor (or maybe none at all), as long as the FDA has not prescribed specific statistical / sample size requirements for your type of device. As a general concept, the goal in the verification stage is to gain confidence (first of all for yourself) that the design is good enough to progress into the pilot production and validation stage; or, contrarily, needs another design iteration.

2. Have you considered applying Bootstrap techniques?

Cheers,
Ronen.
 
Last edited:
L

LoneQC

#17
Re: Sample size for Design Verification / Design Validation

design verification is about "demonstration", not about "proof".
I agree, however it seems questions about design verification/validation (DV&V) quickly turn to process/mfg verification/validation and confidence.

For example, I found this question...
For a medical device design verification/validation, what is the sample size a company should be looking at?
and the conversation quickly turned to the usual process validation and acceptance sampling statistics until...

design verification and design validation sample size is that this is, hardly if ever, an appropriate application for sampling. In sampling there is an inherent consideration that you are trying to capture something about variability.

design verification you are answering the question, “Does the design output match the design input?” It either does or does not in an absolute sense. It is not a question checked by building some units and, for example, estimating the percentages that have the design characteristic in question. In fact, design verification, in many cases, does not require a physical unit

design validation you are answering the question, “Does the device satisfy the intended use and user needs?” Here you need a device that is representative of production. Again, however, variability is not a factor since the device design either satisfies the intended use and user needs or it doesn’t. In the case of the user interface, we only need one device.

process validation, we are exploring an input parameter space. This is a very different problem from the design verification or design validation case. The problem is to show satisfactory output at each point (an infinite number) in the input parameter space.
linkedin.com/groups/So-medical-device-design-verification-2399164.S.45405903

I have been struggling trying to make sense of confidence/reliability and sampling for DV&V, but makes perfect sense for process mfg.

So with this, it seems for pure DV&V, there is really minimum requirements needed to demonstrate design meets input requirements. But how to justify in a 510K submission???

Also, it seems part of the confusion is many companies combine the design validation and process/mfg validations so they will be ready to mfg once everything goes through, especially if it is equivalent to an existing product they already mfg. Small companies or start-ups may not have that luxury, they simply mfg prototypes that will be 'representative of production' to verify design and maybe apply for 510K approval before partnering with contract mfg that will then need to perform the process verifications. Or for ex$pensive products whose production is low, MRI, I'm sure they're not going to mfg 1000 units for verification.

Again, how to justify making 10 prototypes 'representative of production' for DV&V for 510K?

Then there is also concern on how to demonstrate equivalence with minimum # DV&V samples. I did read something about FDA requested a company to demonstrate mean values of equivalence factors to be within ~20%, but nothing about # samples.

Thanks
 

Ronen E

Problem Solver
Staff member
Moderator
#18
Re: Sample size for Design Verification / Design Validation

Hi,

I believe you could find real value in my item #2 in my previous post.

3 more general notes:

a) The "&" concept in "DV&V" really - I mean really - annoys me. To me, it's an indication that people either don't get that verification and validation are two materially different things or are just trying to cut budget and/or schedule corners. It's not a modular package.

b) You are correct that process validation is a completely separate topic. For clarity, I advise you to look at it only after you feel confident and comfortable with the design verification topic. If you see someone pushing it into a discussion about design verification, I recommend you skip it (forgive my boldness please). That person is not likely to help you achieve clarity.

c) Except for the general context (a LinkedIn group) I'm not sure where / who your 2nd quote comes from, or what its regulatory (or other authoritative) basis is. I disagree with its content on multiple issues, and I strongly recommend that you take it with at least a pinch of salt.

Cheers,
Ronen.
 
L

LoneQC

#19
Bootstrapping?? - Unfortunately, its been ~15 years since I've done any statistics, and that was school. Now it all looks like hieroglyphics, its starting to come back, slowly...

a) "&" concept in "DV&V" really - I mean really - annoys me. To me, it's an indication that people either don't get that verification and validation are two materially different things or are just trying to cut budget and/or schedule corners.
small company and small budget... Basically its been rolled into a single phase.. Phase1 - Design, Phase 2 - DV&V , Phase 3 - mfg qualifications (V&V) & design transfer...
Verification - verify all design input has been met
Validation - validate/verify all user needs have been met and no user issues, physician feedback, clinical, ect...

fda.gov/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm070627.htm

b) You are correct that process validation is a completely separate topic.
GREAT, how do I find info on design validation only without getting into AQL, LTPD and confidence interval charts for larger lot mfg..

Except for the general context (a LinkedIn group) I'm not sure where / who your 2nd quote comes from
Yeah, not sure of his qauliifications... but I think that is the general consensus around hear.. basically a MINIMUM # samples to show consistent product that would be 'representative of production'...

Would 10 test samples with distribution data showing x% confidence within design criteria be acceptable?
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#20
GREAT, how do I find info on design validation only without getting into AQL, LTPD and confidence interval charts for larger lot mfg..
First remember that AQL and LTPD based plans have no bearing on validation sample sizes. AQL/LTPD are used for acceptance sampling.
in general you will have higher sample sizes if you are only assessing pass/fail to the specs. you can have smaller sample sizes if you assess the continuous (measurable or "variables") data.

Would 10 test samples with distribution data showing x% confidence within design criteria be acceptable?
for validation, perhaps. of course it also depends on your regulatory reviewer and stats group. Some are very loose and some are very strict. usually depending on the potential for patient harm for the device in question.

In this case however, you must also remember that most confidence interval calculations are for sample MEANS. These intervals provide no information on the spread of the individual values which is what your customer will use. the intervals in this case are tolerance intervals or prediction intervals.
 
Thread starter Similar threads Forum Replies Date
B Determining sample size for device sterility Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
D Determining of sample size for 'Operational Qualification' AQL - Acceptable Quality Level 5
R Determining Sample Size for Medical Device Component Validation Inspection, Prints (Drawings), Testing, Sampling and Related Topics 0
S Surveillance Sampling Test - Determining Sample Size Inspection, Prints (Drawings), Testing, Sampling and Related Topics 5
S Determining sample size for inspection to achieve x% confidence re defects Misc. Quality Assurance and Business Systems Related Topics 10
S Determining Sample Size - AQL & LQ AQL - Acceptable Quality Level 10
B Bayes Success-Run on Attribute Data - Determining Sample Size Statistical Analysis Tools, Techniques and SPC 2
A Determining Sample Size in Design V&V activities 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 44
G Determining Sample Size for Equipment Qualification Studies Qualification and Validation (including 21 CFR Part 11) 10
D Determining Sample Size for Capability in Six Sigma Project Six Sigma 5
H Determining the Correct Sample Size when AQL points to two Sample Sizes AQL - Acceptable Quality Level 7
L Determining Cpk based on a given Sample Size, Confidence Level, & Reliability Inspection, Prints (Drawings), Testing, Sampling and Related Topics 2
M Determining Sample Size needed for Operational Qualification (OQ) Inspection, Prints (Drawings), Testing, Sampling and Related Topics 24
T Determining the appropriate Sample Size using an AQL Table Inspection, Prints (Drawings), Testing, Sampling and Related Topics 29
J Functional Test Sampling - Determining Sample Size to eliminate 100% Testing Inspection, Prints (Drawings), Testing, Sampling and Related Topics 2
W Sampling Plan problem - Determining Sample Size for both Attribute and Variables Data ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
S Determining the Total Sample Size using AQL - AQL Lot Quantity question AQL - Acceptable Quality Level 8
1 Determining AQL and Sample Plan - Batch size is from 1-500 AQL - Acceptable Quality Level 24
N Auditing Proof of Delivery signed by the customer - Determining Sample Size General Auditing Discussions 6
P Determining Sample Size - Filling a bag with three different types of candy Inspection, Prints (Drawings), Testing, Sampling and Related Topics 31
Q Determining Sample Size for FDA Verification and Validation Activities Inspection, Prints (Drawings), Testing, Sampling and Related Topics 41
D Determining sample size required based on a 95% confidence level Reliability Analysis - Predictions, Testing and Standards 7
B Determining sample size for drop test and waterproof testing Inspection, Prints (Drawings), Testing, Sampling and Related Topics 6
T Determining Receiving Inspection Sample Size Inspection, Prints (Drawings), Testing, Sampling and Related Topics 1
N Determining Sample Size for Attribute Data Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
C Internal Audit Sampling Plans - Determining Internal Audit Sample Size Internal Auditing 45
Q Determining the Appropriate Sample Size for Process Validation/Qualification Inspection, Prints (Drawings), Testing, Sampling and Related Topics 13
T Using Cpk for the purpose of determining sample size/frequencies Capability, Accuracy and Stability - Processes, Machines, etc. 2
G Determining Sample Size and Frequency for Features on Control Plans FMEA and Control Plans 8
S Determining sample sizes for PQ Qualification and Validation (including 21 CFR Part 11) 4
R Determining Sample Sizes for Testing (Chemistry or Microbiology) Inspection, Prints (Drawings), Testing, Sampling and Related Topics 6
B Selecting a Sample and Determining the Reference Value for a Bias Study Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 2
S Determining the requirements for the products and services ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
Q Determining Adverse Effects of Corrective/Preventive Actions ISO 13485:2016 - Medical Device Quality Management Systems 2
M Determining if an Insulin Pen Testing Machine is a Medical Device? EU Medical Device Regulations 4
M Determining a tolerance value for Measuring devices in-house inspection General Measurement Device and Calibration Topics 12
D Determining the the maximum number of reprocessing cycles of attachments CE Marking (Conformité Européene) / CB Scheme 2
R Determining Uncertainty from Gage R&R Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 1
A IATF 16949 4.3.1 - Determining the scope of the quality management system - supplemental IATF 16949 - Automotive Quality Systems Standard 9
D ISO 9001:2015 4.3 Determining the Scope of the QMS ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
J Determining SPC tolerance Statistical Analysis Tools, Techniques and SPC 21
B Determining SAT Offsets vs TUS Offsets per SAE AMS 2750E AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 0
R Question on determining defective units - I am not recording fixture to part rejected Statistical Analysis Tools, Techniques and SPC 5
S Clause 8.2.2 Determining the requirements for products and services ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
D Determining Calibration Frequency schedule for items used in production Manufacturing and Related Processes 2
C Determining if Maintenance Contractor is an External Service subject to ISO 9001 Clause 8.4 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 43
S AS9100D PEAR - Examples for organization's method for determining process results? AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 5
I Determining Calibration Tolerance of a Measurement Device General Measurement Device and Calibration Topics 2
J ISO 17025 Documented Procedure for 6.2.5 - Determining competency ISO 17025 related Discussions 4
V Determining FDA 820 (registration) vs ISO 13485 - Supplier gives us the kit ISO 13485:2016 - Medical Device Quality Management Systems 1

Similar threads

Top Bottom