Determining Sample Size in Design V&V activities

G

guysta

#11
Hi All,
First I'd like to thank you all for the insightful comments which help clear things up, but still, I'm not sure I'm there yet.
I have a couple of questions which some of you perhaps could help with:
1. For software Verification and Validation testing, would you believe the rational of SW being a deterministic process, and thus having a 0 standard deviation would justify why pure SW testing is done on a sample size of 1 device?
2. When considering Design Verification and Validation, for an expensive piece of equipment, for example, an MRI scanner or a CT Scanner, it would probably not make sense to test 20-30 systems? how would you tackle this?
If a test protocol tests many different Variables, how would we go about explaining the sample size for it? for each variable a different sample size would be needed, and also, it should be defined while planning, before you have actual results.

I hope any of you could help clear things up.

Thanks:confused:
 
Elsmar Forum Sponsor

Bev D

Heretical Statistician
Staff member
Super Moderator
#12
We must remember that sample size is not controlled only by the number of units when we have an instrument or device that will be used more than once. This is the case with software and medical devices such as CT or MRI scanners. (Examples of single use things are pills, vaccines, blood collection tubes etc.)

For multiple use things, sample size includes the number of things and the number of uses - or runs - of each thing.

Statistically speaking I would use at least 3 of the things regardless of the cost. There may be exceptions and only your reviewer/statistician can answer that question.

As for software, it is deterministic but only to a point. Remember that software doesn't exist in isolation of the device, the user or the conditions of use. Conditions and varying inputs will effect the results. So your sample size for software will be based on both number of devices (the physics of the different device builds dictate that they will not be the same) and the number of runs or uses...
 
G

guysta

#13
Thank you for the reply, that did clear up the issue on repetitions (runs) accounting towards the sample size.
In your experience, would using AQL tables from ANSI Z1.4 although not optimal for V&V Practice (As I read in the article Statistical Sampling Plan for Design V&V of Medical Devices by Liem Ferryanto, Ph.D.)
suffice as a sample size selection rationale for an FDA audit as a statistical rationale?
As I did notice that for a lot/batch size of 2-8 units, 6.5% non-conforming would accept the hypothesis with 0 failures for a sample size of 2 units.

Thanks in advance!
 
J

Julie O

#14
my company has to define a procedure for defining sample sizes regarding design verification.
It sounds like you are trying figure out what methods to use to determine sample sizes for design verification. Like Ronen, I'd want to see exactly what the inspector wrote, but I think the procedure the inspector was looking for was a high-level procedure that describes the process you follow in order to figure out what statistical methods to use to determine sample sizes for design verification.

I wouldn't expect to see any statistical methods cited in this procedure. The procedure should be applicable to any and all design you verify, and there is no way of knowing what types of verification you might need for future designs, much less which statistical methods might be appropriate to use to determine the sample sizes.
 

kreid

Involved In Discussions
#15
For software design V&V you test the design to see if it works, full stop. How much testing you do, i.e. to what depth is up for debate (see the FDA guideline General Principles of Software Validation).

You might choose to do usability testing and for that you will have to justify the size of the user/test group, and again the is and FDA guideline on usability to help you there.

But you only need to exerise the software once.
 

Mark Meer

Trusted Information Resource
#16
I've always been confused by this concept of statistically determining sample sizes for design verification activities.

As was mentioned in this thread, it really depends on the product and characteristics, no?

I'd like to pose a related question: when using standards (e.g. IEC 60601) to verify design requirements, how do these justify sampling?

For example, we had a design requirement "shall be unaffected by a drop from 1m onto hard surface", which was verified through the IEC 60601 drop test. The 60601 testing, however, used (if I recall correctly) just 2 units, without justifying this number. The FDA didn't seem to have any problem with this...
 
J

Julie O

#17
it really depends on the product and characteristics, no?
I agree. That's why I think the inspector might not be looking for a procedure that identifies specific approaches to determining sample sizes for specific types of verification testing, but a high-level procedure that describes the process the company will follow to decide all that for whatever type of verification it might do.
 
#18
Hi Mark,

60601-1 explicitly states that the test are conducted on one unit.

5.2 TYPE TESTS are performed on a representative sample of the item being tested.
Note, also, that the tests are expected to be conducted in a specific order (see Annex B) and still pass all requirements after any damage that may result from tests. So the markings legibility test is the last test after all the rubbing, cleaning simulations, heat, humidity, etc etc.
 

Mark Meer

Trusted Information Resource
#19
60601-1 explicitly states that the test are conducted on one unit.
Thanks Pads. Only one representative sample then.

So while the FDA is harping on the OP to justify his/her sampling for their design verification activities, industry-standards (that the FDA recognizes) use a sample of 1 for verifying requirements?
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#20
an element of confusion here: guysta was asking about software validation and Mark you invoked testing for specific characteristics covered by industry test standards. they are markedly different beasts. Where an accepted standard exists, the FDA - or other regulatory body that accepts that standard - will accept the sample size (or the auditor/reviewer should accept it).
In the case of medical devices in general there is no set sample size for function and/or general hazard. the sample size should be statistically justified in light of the risk. In the case of software there is considerable 'discussion' regarding the aspect of the deterministic nature of software and the conditions and inputs it uses form the device it is running on...
 
Thread starter Similar threads Forum Replies Date
B Determining sample size for device sterility Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
D Determining of sample size for 'Operational Qualification' AQL - Acceptable Quality Level 5
R Determining Sample Size for Medical Device Component Validation Inspection, Prints (Drawings), Testing, Sampling and Related Topics 0
S Surveillance Sampling Test - Determining Sample Size Inspection, Prints (Drawings), Testing, Sampling and Related Topics 5
S Determining sample size for inspection to achieve x% confidence re defects Misc. Quality Assurance and Business Systems Related Topics 10
S Determining Sample Size - AQL & LQ AQL - Acceptable Quality Level 10
B Bayes Success-Run on Attribute Data - Determining Sample Size Statistical Analysis Tools, Techniques and SPC 2
G Determining Sample Size for Equipment Qualification Studies Qualification and Validation (including 21 CFR Part 11) 10
D Determining Sample Size for Capability in Six Sigma Project Six Sigma 5
H Determining the Correct Sample Size when AQL points to two Sample Sizes AQL - Acceptable Quality Level 7
L Determining Cpk based on a given Sample Size, Confidence Level, & Reliability Inspection, Prints (Drawings), Testing, Sampling and Related Topics 2
M Determining Sample Size needed for Operational Qualification (OQ) Inspection, Prints (Drawings), Testing, Sampling and Related Topics 24
T Determining the appropriate Sample Size using an AQL Table Inspection, Prints (Drawings), Testing, Sampling and Related Topics 29
J Functional Test Sampling - Determining Sample Size to eliminate 100% Testing Inspection, Prints (Drawings), Testing, Sampling and Related Topics 2
W Sampling Plan problem - Determining Sample Size for both Attribute and Variables Data ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
S Determining the Total Sample Size using AQL - AQL Lot Quantity question AQL - Acceptable Quality Level 8
1 Determining AQL and Sample Plan - Batch size is from 1-500 AQL - Acceptable Quality Level 24
E Determining Sample Size for Design Verification and Design Validation Other Medical Device and Orthopedic Related Topics 20
N Auditing Proof of Delivery signed by the customer - Determining Sample Size General Auditing Discussions 6
P Determining Sample Size - Filling a bag with three different types of candy Inspection, Prints (Drawings), Testing, Sampling and Related Topics 31
Q Determining Sample Size for FDA Verification and Validation Activities Inspection, Prints (Drawings), Testing, Sampling and Related Topics 41
D Determining sample size required based on a 95% confidence level Reliability Analysis - Predictions, Testing and Standards 7
B Determining sample size for drop test and waterproof testing Inspection, Prints (Drawings), Testing, Sampling and Related Topics 6
T Determining Receiving Inspection Sample Size Inspection, Prints (Drawings), Testing, Sampling and Related Topics 1
N Determining Sample Size for Attribute Data Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
C Internal Audit Sampling Plans - Determining Internal Audit Sample Size Internal Auditing 45
Q Determining the Appropriate Sample Size for Process Validation/Qualification Inspection, Prints (Drawings), Testing, Sampling and Related Topics 13
T Using Cpk for the purpose of determining sample size/frequencies Capability, Accuracy and Stability - Processes, Machines, etc. 2
G Determining Sample Size and Frequency for Features on Control Plans FMEA and Control Plans 8
S Determining sample sizes for PQ Qualification and Validation (including 21 CFR Part 11) 4
R Determining Sample Sizes for Testing (Chemistry or Microbiology) Inspection, Prints (Drawings), Testing, Sampling and Related Topics 6
B Selecting a Sample and Determining the Reference Value for a Bias Study Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 2
S Determining the requirements for the products and services ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
Q Determining Adverse Effects of Corrective/Preventive Actions ISO 13485:2016 - Medical Device Quality Management Systems 2
M Determining if an Insulin Pen Testing Machine is a Medical Device? EU Medical Device Regulations 4
M Determining a tolerance value for Measuring devices in-house inspection General Measurement Device and Calibration Topics 12
D Determining the the maximum number of reprocessing cycles of attachments CE Marking (Conformité Européene) / CB Scheme 2
R Determining Uncertainty from Gage R&R Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 1
A IATF 16949 4.3.1 - Determining the scope of the quality management system - supplemental IATF 16949 - Automotive Quality Systems Standard 9
D ISO 9001:2015 4.3 Determining the Scope of the QMS ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
J Determining SPC tolerance Statistical Analysis Tools, Techniques and SPC 21
B Determining SAT Offsets vs TUS Offsets per SAE AMS 2750E AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 0
R Question on determining defective units - I am not recording fixture to part rejected Statistical Analysis Tools, Techniques and SPC 5
S Clause 8.2.2 Determining the requirements for products and services ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
D Determining Calibration Frequency schedule for items used in production Manufacturing and Related Processes 2
C Determining if Maintenance Contractor is an External Service subject to ISO 9001 Clause 8.4 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 43
S AS9100D PEAR - Examples for organization's method for determining process results? AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 5
I Determining Calibration Tolerance of a Measurement Device General Measurement Device and Calibration Topics 2
J ISO 17025 Documented Procedure for 6.2.5 - Determining competency ISO 17025 related Discussions 4
V Determining FDA 820 (registration) vs ISO 13485 - Supplier gives us the kit ISO 13485:2016 - Medical Device Quality Management Systems 1

Similar threads

Top Bottom