Determining Sample Size for FDA Verification and Validation Activities

Mark Meer

Trusted Information Resource
#21
Thanks again, Bev D! Let's see if I've got this straight....
Continuing with the hypothetical "back-rest" example, here is the steps I'd take:

  1. Create a survey where subjects rank their "sitting comfort" on a scale of 1 to 5. I will consider 4-5 "good" and 1-3 "bad" so I can use Binomial formula.
  2. I give it to "X" people and find the average "goods" is p=0.45 (the actual mean is 3.2).
  3. I choose a delta of d=0.2 and confidence of 95%, and use the formula n = 4p(1-p) / d^2 and get a sample size of approximately 25 subjects.
  4. Conclusion: with 25 subjects I can be 95% confident that their survey results will be within +/- 0.2 from the average of the whole population.

Is this valid? If so, a couple of questions:
- How do you choose a quantity for "X" to establish your initial average (assuming no historical data available)? The file you posted recommends "take a small sample" or "logical guess"... No sure what this means in practice. Given we're talking about the general population here, what is a "small sample"? (I arbitrarily chose 20)
- If the survey values are discrete integers (1 to 5), does a delta value of 0.2 make sense, or should it also be an integer?

...of course, please let me know if I'm totally out to lunch here. Having little statistics background I could have this completely wrong! :eek:
 
Elsmar Forum Sponsor

Bev D

Heretical Statistician
Staff member
Super Moderator
#22
[*]I choose a delta of d=0.2 and confidence of 95%, and use the formula n = 4p(1-p) / d^2 and get a sample size of approximately 25 subjects.
[*]Conclusion: with 25 subjects I can be 95% confident that their survey results will be within +/- 0.2 from the average of the whole population.
[/LIST]

Is this valid?
close. you can say that your estimate will be not more than .2 from the true proportion 5% fo the time. (I know the whole confidence/alpha risk thing is confusing)


- How do you choose a quantity for "X" to establish your initial average (assuming no historical data available)? The file you posted recommends "take a small sample" or "logical guess"... No sure what this means in practice. Given we're talking about the general population here, what is a "small sample"? (I arbitrarily chose 20)
This really is a guess if you want to pre-establsih the 'current' rate. which isn't as horrible as you think. you need to start somewhere. Even with a lot of reliabel data the sampel size you really need wil be driven by the results you actually get from the sample. Teh frustrating part about this is that you can calculate a precise sample size according to the math if all of your assumptions are correct. but it will never be more than an estimate and a fairly rough one at that. While there are a thousand wrong sampel sizes (usually far too small) there are many decent sample sizes htat will tell you what you need to know. (somone once said; its better to have an approximate answer to the right question than a precise answer to the wrogn question). Once you accept this - all is good.

On the other hand if determine what 'good' rate you want as your target than there is no guessing or pre-sampling to determine how much better you want to be. This is the approach I usually take.

- If the survey values are discrete integers (1 to 5), does a delta value of 0.2 make sense, or should it also be an integer?
Absolutely. the delta is a proportion. an integer divided into another integer. so you are good there. remember the binomial deals with proportions that range from 0-1. or none to 100%. It can't deal with an actual integer....
 

Mark Meer

Trusted Information Resource
#23
Once again, thanks. Becoming clearer..

Absolutely. the delta is a proportion. an integer divided into another integer. so you are good there. remember the binomial deals with proportions that range from 0-1. or none to 100%. It can't deal with an actual integer....
Hmmm... I'm afraid I'm still confused here. The document you posted earlier in this thread states:

2. (delta) = the amount of accuracy you want; this is the minimum amount of detectable difference from the true mean.
(delta) is always expressed as a number, not a percent.
Note that +/- (delta) = the Confidence Interval.


I'm confused: you're saying "delta is a proportion", but the document says "[delta is] not a percent".
Is the source of my confusion equating "proportion" with "percent"?
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#24
2. (delta) = the amount of accuracy you want; this is the minimum amount of detectable difference from the true mean.
(delta) is always expressed as a number, not a percent.
Note that +/- (delta) = the Confidence Interval.


I'm confused: you're saying "delta is a proportion", but the document says "[delta is] not a percent".
Is the source of my confusion equating "proportion" with "percent"?
yes it is. let me give you an example:

I often hear people say they want the estimate to be accurate (technically its precision, not accuracy) to within plus or minus 5%. in this case the delta is not 5, and it is not .05. say I'm measuring a length (in mm) that typically can vary from 20-40mm. If i specify a '5% delta' the actual delta number is diffrent if my estimate comes out as 20 (5%=1 mm) or 40 (5%= 2 mm) so we must specify how many measurement units we want as the delta.

now when we deal with categorical/binomial data it gets a bit more confusing becuase the unit of measure for binomial data is a rate expressed as a proportion. X defects/Y units tested. this proportion is what is used in the formula. so the delta must also be expressed as a proportion.

does that help?
 

Marc

Hunkered Down for the Duration with a Mask on...
Staff member
Admin
#26
:topic: If one of the people here provides you with a lot of help, give the person "Karma" from time to time (see the
button at the top of each person's post).

If you also want to Thank the person using the Thanks button in the post, you have to give "Karma" first and then use the
button on the bottom of the post.
 

Michael Malis

Quite Involved in Discussions
#27
Hi Bev,

How do you differentiate sampling between simple disposable parts and complicated equipment?

For example, it is easy to justify sampling 60 pieces per lot when we mold 10,000 pieces. However, if we only build 15 units per week ($50,000/each), what is statistically valid sampling would you recommend if we want 95/90% confidence?

Regards,
Michael
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#28
Are you referring to an acceptance sampling plan or validation plan?
My organization manufactures both consumables (high volume) and instruments (low volume). So I have faced the same dilemma.

ACCEPTANCE SAMPLING (aka RELEASE TESTING)
We test 100% of our instruments. the sample size applies to the number of runs and is often only 1. If you are asking about acceptance sampling I can elaborate...but since this is a validation thread:

VALIDATION TESTING:
In general, for validation we test smaller numbers of instruments, but will test many runs on those instruments. We apply the 'sample size' calculations to the number of runs (the patient sample is the sampling unit) than the number of instruments. We will typically use only a few instruments. The thought is that each instrument at new product launch is as varied as the other given the complex nature of the assembly and components. We will specify that 3 lots of consumables need to be used (in keeping with PQ principles).

When we validate a changed component we will typically use 6 instruments, randomly assigning 3 of the new components and 3 of the current version to each of 6 instruments. We will then perform a number of runs on each instrument (statistically based on the largest component of variation of the instrument system; this could be a fairly small number as we can often use continuous data statistics, but has been quite large when testing for a rather rare error code - categorical data) and then we will change out the components so that an instrument that started with a new component gets a current version component and rerun the testing. The analysis is done as a paired t-test (block = instrument) within the instruments. This provides us a lot of power while minimizing the sample size...

If you are servicing the Human medical market, your FDA reviewer and/or statistician can provide more specific guidance on the acceptable sample sizes for validation...I can only provide non-binding advice. :notme:
 

joemar

Involved In Discussions
#30
Since this thread is still active, I have a similar question as much of this thread. I've read the previous, but wanted some clarification. My situation is:

I have been tasked to perform a OQ and PQ to validate the seals made by a new bar sealer. I am reading the template I was given, and I dont understand how they determined the sample size. Once in production, this sealer will produce something like 400,000 seals in a year, and do lot sizes of 3000. There is inprocess testing as well, but the frequency of the inprocess testing has always been somewhat gray. My template gives the following rationale for the sample size used: "[FONT=&quot][upper and lower limit] [/FONT][FONT=&quot]operational qualification runs will include a total of 59 samples per the handbook of statistics. With no test failures, the sample size will provide at least a 95% confidence and a 95% reliability that no problems will occur.[/FONT]"

Now, im reviewing the handbook of statistics, specifically the "Zero Acceptance Number Sampling Plans" book, 4th edition, by Nicholas Squeglia, which is our reference book, and I dont find a single reference to the number 59 anywhere in the entire book. Seals are made 3 pouches at a time, so I thought maybe it was a permutation of 3x20=60 (obviously 60 does not equal 59. but the book states that an AQL of 2.5 for a lot of 1201 to 3200 is 23 units sampled... kinda close...)

Anyway, point is, i am confused how these numbers were created and wondering if anyone here (1) could explain how 59 units was chosen as a sample size (or if not, provide evidence that this seems like a fishy number), and (2) could give me their opinions on what a proper sample size would be under the conditions defined above?

I really appreciate the help.

Joe
 
Thread starter Similar threads Forum Replies Date
B Determining sample size for device sterility Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
D Determining of sample size for 'Operational Qualification' AQL - Acceptable Quality Level 5
R Determining Sample Size for Medical Device Component Validation Inspection, Prints (Drawings), Testing, Sampling and Related Topics 0
S Surveillance Sampling Test - Determining Sample Size Inspection, Prints (Drawings), Testing, Sampling and Related Topics 5
S Determining sample size for inspection to achieve x% confidence re defects Misc. Quality Assurance and Business Systems Related Topics 10
S Determining Sample Size - AQL & LQ AQL - Acceptable Quality Level 10
B Bayes Success-Run on Attribute Data - Determining Sample Size Statistical Analysis Tools, Techniques and SPC 2
A Determining Sample Size in Design V&V activities 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 44
G Determining Sample Size for Equipment Qualification Studies Qualification and Validation (including 21 CFR Part 11) 10
D Determining Sample Size for Capability in Six Sigma Project Six Sigma 5
H Determining the Correct Sample Size when AQL points to two Sample Sizes AQL - Acceptable Quality Level 7
L Determining Cpk based on a given Sample Size, Confidence Level, & Reliability Inspection, Prints (Drawings), Testing, Sampling and Related Topics 2
M Determining Sample Size needed for Operational Qualification (OQ) Inspection, Prints (Drawings), Testing, Sampling and Related Topics 24
T Determining the appropriate Sample Size using an AQL Table Inspection, Prints (Drawings), Testing, Sampling and Related Topics 29
J Functional Test Sampling - Determining Sample Size to eliminate 100% Testing Inspection, Prints (Drawings), Testing, Sampling and Related Topics 2
W Sampling Plan problem - Determining Sample Size for both Attribute and Variables Data ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
S Determining the Total Sample Size using AQL - AQL Lot Quantity question AQL - Acceptable Quality Level 8
1 Determining AQL and Sample Plan - Batch size is from 1-500 AQL - Acceptable Quality Level 24
E Determining Sample Size for Design Verification and Design Validation Other Medical Device and Orthopedic Related Topics 20
N Auditing Proof of Delivery signed by the customer - Determining Sample Size General Auditing Discussions 6
P Determining Sample Size - Filling a bag with three different types of candy Inspection, Prints (Drawings), Testing, Sampling and Related Topics 31
D Determining sample size required based on a 95% confidence level Reliability Analysis - Predictions, Testing and Standards 7
B Determining sample size for drop test and waterproof testing Inspection, Prints (Drawings), Testing, Sampling and Related Topics 6
T Determining Receiving Inspection Sample Size Inspection, Prints (Drawings), Testing, Sampling and Related Topics 1
N Determining Sample Size for Attribute Data Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
C Internal Audit Sampling Plans - Determining Internal Audit Sample Size Internal Auditing 45
Q Determining the Appropriate Sample Size for Process Validation/Qualification Inspection, Prints (Drawings), Testing, Sampling and Related Topics 13
T Using Cpk for the purpose of determining sample size/frequencies Capability, Accuracy and Stability - Processes, Machines, etc. 2
G Determining Sample Size and Frequency for Features on Control Plans FMEA and Control Plans 8
S Determining sample sizes for PQ Qualification and Validation (including 21 CFR Part 11) 4
R Determining Sample Sizes for Testing (Chemistry or Microbiology) Inspection, Prints (Drawings), Testing, Sampling and Related Topics 6
B Selecting a Sample and Determining the Reference Value for a Bias Study Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 2
S Determining the requirements for the products and services ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
Q Determining Adverse Effects of Corrective/Preventive Actions ISO 13485:2016 - Medical Device Quality Management Systems 2
M Determining if an Insulin Pen Testing Machine is a Medical Device? EU Medical Device Regulations 4
M Determining a tolerance value for Measuring devices in-house inspection General Measurement Device and Calibration Topics 12
D Determining the the maximum number of reprocessing cycles of attachments CE Marking (Conformité Européene) / CB Scheme 2
R Determining Uncertainty from Gage R&R Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 1
A IATF 16949 4.3.1 - Determining the scope of the quality management system - supplemental IATF 16949 - Automotive Quality Systems Standard 9
D ISO 9001:2015 4.3 Determining the Scope of the QMS ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
J Determining SPC tolerance Statistical Analysis Tools, Techniques and SPC 21
B Determining SAT Offsets vs TUS Offsets per SAE AMS 2750E AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 0
R Question on determining defective units - I am not recording fixture to part rejected Statistical Analysis Tools, Techniques and SPC 5
S Clause 8.2.2 Determining the requirements for products and services ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
D Determining Calibration Frequency schedule for items used in production Manufacturing and Related Processes 2
C Determining if Maintenance Contractor is an External Service subject to ISO 9001 Clause 8.4 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 43
S AS9100D PEAR - Examples for organization's method for determining process results? AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 5
I Determining Calibration Tolerance of a Measurement Device General Measurement Device and Calibration Topics 2
J ISO 17025 Documented Procedure for 6.2.5 - Determining competency ISO 17025 related Discussions 4
V Determining FDA 820 (registration) vs ISO 13485 - Supplier gives us the kit ISO 13485:2016 - Medical Device Quality Management Systems 1

Similar threads

Top Bottom