Determining Sample Size for FDA Verification and Validation Activities

[Mark Meer]

Thanks again, Bev D! Let's see if I've got this straight....
Continuing with the hypothetical "back-rest" example, here is the steps I'd take:

1. Create a survey where subjects rank their "sitting comfort" on a scale of 1 to 5. I will consider 4-5 "good" and 1-3 "bad" so I can use Binomial formula.
2. I give it to "X" people and find the average "goods" is p=0.45 (the actual mean is 3.2).
3. I choose a delta of d=0.2 and confidence of 95%, and use the formula n = 4p(1-p) / d^2 and get a sample size of approximately 25 subjects.
4. Conclusion: with 25 subjects I can be 95% confident that their survey results will be within +/- 0.2 from the average of the whole population.

Is this valid? If so, a couple of questions:
- How do you choose a quantity for "X" to establish your initial average (assuming no historical data available)? The file you posted recommends "take a small sample" or "logical guess"... No sure what this means in practice. Given we're talking about the general population here, what is a "small sample"? (I arbitrarily chose 20)
- If the survey values are discrete integers (1 to 5), does a delta value of 0.2 make sense, or should it also be an integer?

...of course, please let me know if I'm totally out to lunch here. Having little statistics background I could have this completely wrong!

 

[Bev D]

[*]I choose a delta of d=0.2 and confidence of 95%, and use the formula n = 4p(1-p) / d^2 and get a sample size of approximately 25 subjects.
[*]Conclusion: with 25 subjects I can be 95% confident that their survey results will be within +/- 0.2 from the average of the whole population.
[/LIST]

Is this valid?

close. you can say that your estimate will be not more than .2 from the true proportion 5% fo the time. (I know the whole confidence/alpha risk thing is confusing)

- How do you choose a quantity for "X" to establish your initial average (assuming no historical data available)? The file you posted recommends "take a small sample" or "logical guess"... No sure what this means in practice. Given we're talking about the general population here, what is a "small sample"? (I arbitrarily chose 20)

This really is a guess if you want to pre-establsih the 'current' rate. which isn't as horrible as you think. you need to start somewhere. Even with a lot of reliabel data the sampel size you really need wil be driven by the results you actually get from the sample. Teh frustrating part about this is that you can calculate a precise sample size according to the math if all of your assumptions are correct. but it will never be more than an estimate and a fairly rough one at that. While there are a thousand wrong sampel sizes (usually far too small) there are many decent sample sizes htat will tell you what you need to know. (somone once said; its better to have an approximate answer to the right question than a precise answer to the wrogn question). Once you accept this - all is good.

On the other hand if determine what 'good' rate you want as your target than there is no guessing or pre-sampling to determine how much better you want to be. This is the approach I usually take.

- If the survey values are discrete integers (1 to 5), does a delta value of 0.2 make sense, or should it also be an integer?

Absolutely. the delta is a proportion. an integer divided into another integer. so you are good there. remember the binomial deals with proportions that range from 0-1. or none to 100%. It can't deal with an actual integer....

 

[Mark Meer]

Once again, thanks. Becoming clearer..

Absolutely. the delta is a proportion. an integer divided into another integer. so you are good there. remember the binomial deals with proportions that range from 0-1. or none to 100%. It can't deal with an actual integer....

Hmmm... I'm afraid I'm still confused here. The document you posted earlier in this thread states:

2. (delta) = the amount of accuracy you want; this is the minimum amount of detectable difference from the true mean.
(delta) is always expressed as a number, not a percent.
Note that +/- (delta) = the Confidence Interval.


I'm confused: you're saying "delta is a proportion", but the document says "[delta is] not a percent".
Is the source of my confusion equating "proportion" with "percent"?

 

[Bev D]

2. (delta) = the amount of accuracy you want; this is the minimum amount of detectable difference from the true mean.
(delta) is always expressed as a number, not a percent.
Note that +/- (delta) = the Confidence Interval.


I'm confused: you're saying "delta is a proportion", but the document says "[delta is] not a percent".
Is the source of my confusion equating "proportion" with "percent"?

yes it is. let me give you an example:

I often hear people say they want the estimate to be accurate (technically its precision, not accuracy) to within plus or minus 5%. in this case the delta is not 5, and it is not .05. say I'm measuring a length (in mm) that typically can vary from 20-40mm. If i specify a '5% delta' the actual delta number is diffrent if my estimate comes out as 20 (5%=1 mm) or 40 (5%= 2 mm) so we must specify how many measurement units we want as the delta.

now when we deal with categorical/binomial data it gets a bit more confusing becuase the unit of measure for binomial data is a rate expressed as a proportion. X defects/Y units tested. this proportion is what is used in the formula. so the delta must also be expressed as a proportion.

does that help?

 

[Mark Meer]

does that help?

Totally!
If I could click the thanks button more than once, I would!

And, I'm sure I speak for other stat-challenged forum goers too..

 

[Marc]

If one of the people here provides you with a lot of help, give the person "Karma" from time to time (see the

button at the top of each person's post).

If you also want to Thank the person using the Thanks button in the post, you have to give "Karma" first and then use the

button on the bottom of the post.

 

[Michael Malis]

Hi Bev,

How do you differentiate sampling between simple disposable parts and complicated equipment?

For example, it is easy to justify sampling 60 pieces per lot when we mold 10,000 pieces. However, if we only build 15 units per week ($50,000/each), what is statistically valid sampling would you recommend if we want 95/90% confidence?

Regards,
Michael

 

[Bev D]

Are you referring to an acceptance sampling plan or validation plan?
My organization manufactures both consumables (high volume) and instruments (low volume). So I have faced the same dilemma.

ACCEPTANCE SAMPLING (aka RELEASE TESTING)
We test 100% of our instruments. the sample size applies to the number of runs and is often only 1. If you are asking about acceptance sampling I can elaborate...but since this is a validation thread:

VALIDATION TESTING:
In general, for validation we test smaller numbers of instruments, but will test many runs on those instruments. We apply the 'sample size' calculations to the number of runs (the patient sample is the sampling unit) than the number of instruments. We will typically use only a few instruments. The thought is that each instrument at new product launch is as varied as the other given the complex nature of the assembly and components. We will specify that 3 lots of consumables need to be used (in keeping with PQ principles).

When we validate a changed component we will typically use 6 instruments, randomly assigning 3 of the new components and 3 of the current version to each of 6 instruments. We will then perform a number of runs on each instrument (statistically based on the largest component of variation of the instrument system; this could be a fairly small number as we can often use continuous data statistics, but has been quite large when testing for a rather rare error code - categorical data) and then we will change out the components so that an instrument that started with a new component gets a current version component and rerun the testing. The analysis is done as a paired t-test (block = instrument) within the instruments. This provides us a lot of power while minimizing the sample size...

If you are servicing the Human medical market, your FDA reviewer and/or statistician can provide more specific guidance on the acceptable sample sizes for validation...I can only provide non-binding advice.

 

[Michael Malis]

Thank you Bev!

I was interested in Validation testing rationale.

Regards,
Michael

 

[joemar]

Since this thread is still active, I have a similar question as much of this thread. I've read the previous, but wanted some clarification. My situation is:

I have been tasked to perform a OQ and PQ to validate the seals made by a new bar sealer. I am reading the template I was given, and I dont understand how they determined the sample size. Once in production, this sealer will produce something like 400,000 seals in a year, and do lot sizes of 3000. There is inprocess testing as well, but the frequency of the inprocess testing has always been somewhat gray. My template gives the following rationale for the sample size used: "[FONT=&quot][upper and lower limit] [/FONT][FONT=&quot]operational qualification runs will include a total of 59 samples per the handbook of statistics. With no test failures, the sample size will provide at least a 95% confidence and a 95% reliability that no problems will occur.[/FONT]"

Now, im reviewing the handbook of statistics, specifically the "Zero Acceptance Number Sampling Plans" book, 4th edition, by Nicholas Squeglia, which is our reference book, and I dont find a single reference to the number 59 anywhere in the entire book. Seals are made 3 pouches at a time, so I thought maybe it was a permutation of 3x20=60 (obviously 60 does not equal 59. but the book states that an AQL of 2.5 for a lot of 1201 to 3200 is 23 units sampled... kinda close...)

Anyway, point is, i am confused how these numbers were created and wondering if anyone here (1) could explain how 59 units was chosen as a sample size (or if not, provide evidence that this seems like a fishy number), and (2) could give me their opinions on what a proper sample size would be under the conditions defined above?

I really appreciate the help.

Joe