Sample Size for Design Verification and Validation

[gxzn008]

Recently, FDA auidted our site, they raised one question on sample size for design verification as follows:

Specifically, procedures for identifying valid statistical techniques required for verifying the acceptability of product characteristics of a new design during design verification and design validation testing have not been adequately established. For example, for the design validation testing of the new product design project, only one unit was tested. No valid statistical rationale was documented for the testing of only one unit.

So according to our practice, we only finish one prototype from Lab for design verification, so how to solve this problem, do we need manufacture more prototypes from Lab for design verification?

Can anyone help me clarify this?

Thanks.

 

[somashekar]

I have a question.

Samples scientifically represents a lot when the lot is happening in and under controlled conditions.
As you are talking about design verification and validation, of what use is the rest of the products if you take only one for the purpose ?
Hence I do not think you have any lot quantity at this design process, and I guess you are not making hundreds of products at this stage.

Kindly explain more.

 

[Ronen E]

Recently, FDA auidted our site, they raised one question on sample size for design verification as follows:

Specifically, procedures for identifying valid statistical techniques required for verifying the acceptability of product characteristics of a new design during design verification and design validation testing have not been adequately established. For example, for the design validation testing of the new product design project, only one unit was tested. No valid statistical rationale was documented for the testing of only one unit.

So according to our practice, we only finish one prototype from Lab for design verification, so how to solve this problem, do we need manufacture more prototypes from Lab for design verification?

Can anyone help me clarify this?

Thanks.

Hello,

Reading your post carefully, I noticed that the FDA's comment mostly related to VALIDATION, whilst your question talks about VERIFICATION. These are typically 2 separate stages. During design verification you are required to DEMONSTRATE, so a small sample may suffice (with the right justification); however, in validation you are required to PROVE, and hence the expectation for statistical rigour.

Further, please note that the FDA didn't focus on the sample size to itself, but on the lack of statistical justification thereof and/or lack of procedure(s) for identification of valid statistical techniques for determining sample size.

Had I been in your place, I would have considered the following measures:

(1) Establish a procedure(s) for identification of valid statistical techniques for determining sample size.

(2) Go through all open projects and recently completed ones and make sure all testing sample sizes are accounted for in terms of valid statistical techniques.

(3) Make sure that validation batch size and/or sampling and/or test sample size (as applicable) are representative of the expected / planned commercial production. Capture and document.

If one expects to commercially manufacture 1 machine (device) per year across 10 years, I guess sampling and statistical considerations would be irrelevant. BTW, you could use Bootstrap and other relevant techniques if you only practically have a very small sample available.

Cheers,
Ronen.

 

[gxzn008]

Ronen, thanks for your post.

Actually for design validation, we can use confidence/reliability (95/90) as a basis to get the sample size (30) using the link http://elsmar.com/Forums/showthread.php?t=51211; so from your meaning, for design verfiication, we can provide the rationale for test.

Can anyone share the a procedure(s) for identification of valid statistical techniques for determining sample size?

Mike

 

[Statistical Steven]

Recently, FDA auidted our site, they raised one question on sample size for design verification as follows:

Specifically, procedures for identifying valid statistical techniques required for verifying the acceptability of product characteristics of a new design during design verification and design validation testing have not been adequately established. For example, for the design validation testing of the new product design project, only one unit was tested. No valid statistical rationale was documented for the testing of only one unit.

So according to our practice, we only finish one prototype from Lab for design verification, so how to solve this problem, do we need manufacture more prototypes from Lab for design verification?

Can anyone help me clarify this?

Thanks.

This is a HUGE misconception...sample size does NOT imply unique devices. If the variability within a device is equal to or greater than what I expect to see between devices I can do multiple tests on a single unit. If there is NO variability between tests (for example software) you can justify a sample size of 1 since there is NO variability between tests. Does this make things clearer?

 

[Ronen E]

This is a HUGE misconception...sample size does NOT imply unique devices. If the variability within a device is equal to or greater than what I expect to see between devices I can do multiple tests on a single unit. If there is NO variability between tests (for example software) you can justify a sample size of 1 since there is NO variability between tests. Does this make things clearer?

How do you PROVE the level of variability between non-software device units (e.g. machines), when you have only a few or just one?

 

[Statistical Steven]

How do you PROVE the level of variability between non-software device units (e.g. machines), when you have only a few or just one?

You have component level testing and other sources of testing to estimate variability. Typically it is a scientific judgement and good engineering assessment. Typically for large capital equipment many of the features are verified prior to release so the "risk" at V&V is lower.