Determining Sample Size in Design V&V activities

Mark Meer

Trusted Information Resource
#21
Hi Bev, I take your point, but I don't think there's any confusion.
(except for the confusion caused by inconsistent expectations/policies from regulators ;))

The original poster was clearly expressing the FDA was concerned with their design verification sample sizes and was seeking advice with respect to "electrical safety tests, software verification, mechanical durablility and loadability tests"...most of which are covered in standards like IEC 60601.

So my question stands: Why does the FDA accept IEC sample size of 1, and yet requires justification for internal design verification sample sizes? Does IEC have a "statistically valid" rationale for only requiring one representative sample?

It seems to me that if the FDA will recognize an IEC sample size of 1, then any comparable internal verification activities should also be able to use 1. If more than 1 is used, great...but this shouldn't have to be justified statistically.
 
Elsmar Forum Sponsor

Statistical Steven

Statistician
Staff member
Super Moderator
#22
Mark

Any sample size specified in a international standard or compendium is recognized by the FDA as acceptable without justification needed by the company IF and ONLY IF the test method specified by the standard or compendium is followed as written and is applicable. A classic example is the use of ANSI Z1.4. People use it for other applications besides incoming inspection and without the switching rules and get asked to justify their use of the standard. If used correctly, the FDA would accept it for incoming inspection.



Hi Bev, I take your point, but I don't think there's any confusion.
(except for the confusing caused by inconsistent expectations/policies from regulators ;))

The original poster was clearly expressing the FDA was concerned with their design verification sample sizes and was seeking advice with respect to "electrical safety tests, software verification, mechanical durablility and loadability tests"...most of which are covered in standards like IEC 60601.

So my question stands: Why does the FDA accept IEC sample size of 1, and yet requires justification for internal design verification sample sizes. Does IEC have a "statistically valid" rationale for only requiring one representative sample?

It seems to me that if the FDA will recognize an IEC sample size of 1, then any comparable internal verification activities should also be able to use 1. If more than 1 is used, great...but this shouldn't have to be justified statistically.
 
Q

QA-Man

#23
I typical try to use Hypothesis testing when possible with a minimal sample size of 30. It could be 30 units or 30 tests. However the more the better. I also will state in the QMS or V&V documents that all definitions and methods are in accordance with ISO 3534 series of standards. Compliance wise, it has never been an issue.

I am not a statistician so I'd be curious to get Statistical Steven and Bev D's take on this.
 

Ronen E

Problem Solver
Staff member
Moderator
#24
Hi Mark,

60601-1 explicitly states that the test are conducted on one unit.

5.2 TYPE TESTS are performed on a representative sample of the item being tested.
Note, also, that the tests are expected to be conducted in a specific order (see Annex B) and still pass all requirements after any damage that may result from tests. So the markings legibility test is the last test after all the rubbing, cleaning simulations, heat, humidity, etc etc.
Hi Bev, I take your point, but I don't think there's any confusion.
(except for the confusion caused by inconsistent expectations/policies from regulators ;))

The original poster was clearly expressing the FDA was concerned with their design verification sample sizes and was seeking advice with respect to "electrical safety tests, software verification, mechanical durablility and loadability tests"...most of which are covered in standards like IEC 60601.

So my question stands: Why does the FDA accept IEC sample size of 1, and yet requires justification for internal design verification sample sizes? Does IEC have a "statistically valid" rationale for only requiring one representative sample?

It seems to me that if the FDA will recognize an IEC sample size of 1, then any comparable internal verification activities should also be able to use 1. If more than 1 is used, great...but this shouldn't have to be justified statistically.
I think that there's a misunderstanding here.

"A representative sample" might be read as "a unit", but I think that in the broad statistical sense (as commonly used in the literature, including published standards) it means A SET that represents the variability in the entire population. It could be a set of 1 sometimes - depends on the circumstances and essentially on the question whether that set of 1 can accurately and reliably represent the entire population.

For example, when you injection-mold in a 32-cavity tool you wouldn't normally regard 1 piece as a "representative sample". Rather, the minimum for testing would be a single shot - 32 pieces. And then, there's also shot-to-shot variability, day-and-night variability (sometimes), seasonal variability, raw material batch variability etc. etc... The concept of "representative sample" means that testing or inspecting that sample (set) can be regarded as representing the whole lot.

It all becomes less obvious when the number of units produced falls (eg MRI machines) but the principles still hold (sometimes units are substituted with uses or runs).

When it comes to stand-alone software it gets more fuzzy and I admit that I'm no software expert so I'll leave that aspect to more knowledgable people.
 

Statistical Steven

Statistician
Staff member
Super Moderator
#25
I typical try to use Hypothesis testing when possible with a minimal sample size of 30. It could be 30 units or 30 tests. However the more the better. I also will state in the QMS or V&V documents that all definitions and methods are in accordance with ISO 3534 series of standards. Compliance wise, it has never been an issue.

I am not a statistician so I'd be curious to get Statistical Steven and Bev D's take on this.
Can be very dangerous to have a one size fits all sample size. Sample size needs to be related to risk. A sample size of 30 might be too large for low risk and too small for higher risk issues. The 3534 set of standards are just statistical definitions, not sure if you meant 2859 or 3951. Anyway, relying on a standard alone has to be grounded in the technical use of the standard and must be applied correctly.
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#26
Steven is correct. Risk - or probability of detection, confidence, and the standard deviation of the process - has a lot to do with it. Also the nature of the causal system must be considered. Some functions are deterministic some are subject to variation. Remember that sample size is related directly to the amount of variation (SD).

Remember too that the homogeneity of the underlying process is even more critical than sample size. A random sample cannot overcome processes that are vary non-homogenous.

And we can never get around the mathematical fact that low occurrences rates of severe categorical effects will require large sample sizes.

The calculation of mathematical formulas is no substitute for thinking...
 

Marcelo

Inactive Registered Visitor
#27
Something that seems missing here is the fact that things like type testing and the like are related, nowadays, to conformity assessment procedures, in particular product certification schemes, as defined in ISO/IEC 17067:2013, Conformity assessment - Fundamentals of product certification and guidelines for product certification schemes.

Why is this important? Because the scheme would need to more clearly define what the requirements are. For example, in Brazil, the certification scheme for medical equipment clearly defines that only one sample shall be used unless there's need and justification for another (and this one sample is blindly sampled from production units). Is this statistically correct? Probably not, but the point is this is the agreed requirements of the scheme (on the other hand, in the case of equipment, it seems that there's an expectation that most, if not all, characteristics, can be evaluated by only one sample).

On a related matter, we did include requirement that

The organization shall document verification / validation plans that include methods, acceptance criteria and, as appropriate, statistical techniques with rationale for sample size

Because the FDA wanted the standard to require them.
 
Q

QA-Man

#29
Steven is correct. Risk - or probability of detection, confidence, and the standard deviation of the process - has a lot to do with it. Also the nature of the causal system must be considered. Some functions are deterministic some are subject to variation. Remember that sample size is related directly to the amount of variation (SD).

Remember too that the homogeneity of the underlying process is even more critical than sample size. A random sample cannot overcome processes that are vary non-homogenous.

And we can never get around the mathematical fact that low occurrences rates of severe categorical effects will require large sample sizes.

The calculation of mathematical formulas is no substitute for thinking...
As a mathematician, I agree with you. When I was a quality person years ago, I would adamantly agree with you.

When I became a regulatory guy I learned to agree with whatever can be interpreted within the regulations that is favorable for my client or employer. So I use the same technique over and over as long as it continues to be accepted by whatever regulatory authority that reviews it.
 

Mark Meer

Trusted Information Resource
#30
When I became a regulatory guy I learned to agree with whatever can be interpreted within the regulations that is favorable for my client or employer. So I use the same technique over and over as long as it continues to be accepted by whatever regulatory authority that reviews it.
:lol: ...Sorry, but my cynical side has to chuckle at this...

I typical try to use Hypothesis testing when possible with a minimal sample size of 30.
Curious: I presume that none of your clients produce low-volume devices? We produce our devices in low quantities, and so to procure 30 samples exclusively for testing purposes would be totally impractical.

For us, sample sizes have to be justified by risk, and despite the FDA, this isn't always tied to statistics.
 
Thread starter Similar threads Forum Replies Date
B Determining sample size for device sterility Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
D Determining of sample size for 'Operational Qualification' AQL - Acceptable Quality Level 5
R Determining Sample Size for Medical Device Component Validation Inspection, Prints (Drawings), Testing, Sampling and Related Topics 0
S Surveillance Sampling Test - Determining Sample Size Inspection, Prints (Drawings), Testing, Sampling and Related Topics 5
S Determining sample size for inspection to achieve x% confidence re defects Misc. Quality Assurance and Business Systems Related Topics 10
S Determining Sample Size - AQL & LQ AQL - Acceptable Quality Level 10
B Bayes Success-Run on Attribute Data - Determining Sample Size Statistical Analysis Tools, Techniques and SPC 2
G Determining Sample Size for Equipment Qualification Studies Qualification and Validation (including 21 CFR Part 11) 10
D Determining Sample Size for Capability in Six Sigma Project Six Sigma 5
H Determining the Correct Sample Size when AQL points to two Sample Sizes AQL - Acceptable Quality Level 7
L Determining Cpk based on a given Sample Size, Confidence Level, & Reliability Inspection, Prints (Drawings), Testing, Sampling and Related Topics 2
M Determining Sample Size needed for Operational Qualification (OQ) Inspection, Prints (Drawings), Testing, Sampling and Related Topics 24
T Determining the appropriate Sample Size using an AQL Table Inspection, Prints (Drawings), Testing, Sampling and Related Topics 29
J Functional Test Sampling - Determining Sample Size to eliminate 100% Testing Inspection, Prints (Drawings), Testing, Sampling and Related Topics 2
W Sampling Plan problem - Determining Sample Size for both Attribute and Variables Data ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
S Determining the Total Sample Size using AQL - AQL Lot Quantity question AQL - Acceptable Quality Level 8
1 Determining AQL and Sample Plan - Batch size is from 1-500 AQL - Acceptable Quality Level 24
E Determining Sample Size for Design Verification and Design Validation Other Medical Device and Orthopedic Related Topics 20
N Auditing Proof of Delivery signed by the customer - Determining Sample Size General Auditing Discussions 6
P Determining Sample Size - Filling a bag with three different types of candy Inspection, Prints (Drawings), Testing, Sampling and Related Topics 31
Q Determining Sample Size for FDA Verification and Validation Activities Inspection, Prints (Drawings), Testing, Sampling and Related Topics 41
D Determining sample size required based on a 95% confidence level Reliability Analysis - Predictions, Testing and Standards 7
B Determining sample size for drop test and waterproof testing Inspection, Prints (Drawings), Testing, Sampling and Related Topics 6
T Determining Receiving Inspection Sample Size Inspection, Prints (Drawings), Testing, Sampling and Related Topics 1
N Determining Sample Size for Attribute Data Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
C Internal Audit Sampling Plans - Determining Internal Audit Sample Size Internal Auditing 45
Q Determining the Appropriate Sample Size for Process Validation/Qualification Inspection, Prints (Drawings), Testing, Sampling and Related Topics 13
T Using Cpk for the purpose of determining sample size/frequencies Capability, Accuracy and Stability - Processes, Machines, etc. 2
G Determining Sample Size and Frequency for Features on Control Plans FMEA and Control Plans 8
S Determining sample sizes for PQ Qualification and Validation (including 21 CFR Part 11) 4
R Determining Sample Sizes for Testing (Chemistry or Microbiology) Inspection, Prints (Drawings), Testing, Sampling and Related Topics 6
B Selecting a Sample and Determining the Reference Value for a Bias Study Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 2
S Determining the requirements for the products and services ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
Q Determining Adverse Effects of Corrective/Preventive Actions ISO 13485:2016 - Medical Device Quality Management Systems 2
M Determining if an Insulin Pen Testing Machine is a Medical Device? EU Medical Device Regulations 4
M Determining a tolerance value for Measuring devices in-house inspection General Measurement Device and Calibration Topics 12
D Determining the the maximum number of reprocessing cycles of attachments CE Marking (Conformité Européene) / CB Scheme 2
R Determining Uncertainty from Gage R&R Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 1
A IATF 16949 4.3.1 - Determining the scope of the quality management system - supplemental IATF 16949 - Automotive Quality Systems Standard 9
D ISO 9001:2015 4.3 Determining the Scope of the QMS ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
J Determining SPC tolerance Statistical Analysis Tools, Techniques and SPC 21
B Determining SAT Offsets vs TUS Offsets per SAE AMS 2750E AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 0
R Question on determining defective units - I am not recording fixture to part rejected Statistical Analysis Tools, Techniques and SPC 5
S Clause 8.2.2 Determining the requirements for products and services ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
D Determining Calibration Frequency schedule for items used in production Manufacturing and Related Processes 2
C Determining if Maintenance Contractor is an External Service subject to ISO 9001 Clause 8.4 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 43
S AS9100D PEAR - Examples for organization's method for determining process results? AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 5
I Determining Calibration Tolerance of a Measurement Device General Measurement Device and Calibration Topics 2
J ISO 17025 Documented Procedure for 6.2.5 - Determining competency ISO 17025 related Discussions 4
V Determining FDA 820 (registration) vs ISO 13485 - Supplier gives us the kit ISO 13485:2016 - Medical Device Quality Management Systems 1

Similar threads

Top Bottom