Firmware Verification Testing & Sample Size

V

v1cph1rth

#1
Hi All,

We are developing custom firmware for our medical device (class II), and it is only intended to run on two versions of hardware. We are ISO 13485 and there is some leeway with statistical sample sizes/justification. I am struggling trying to understand if statistical sample size justification is required or not. Or if we can provide a justification for much smaller testing regime.

The firmware developer is trying to claim that they perform exhaustive testing of all features on all hardwares (n=2) . Testing more than 1 sample per hardware version is silly, because you are testing that the "bits" are transferring properly and working. The firmware also verifies itself on startup.

I am used to looking at a statistical plan for testing (ex: identifying confidence level and probability of finding a bug). I was balked at when throwing out a 90% CI with 15% bug finding resulting in a n=15 sample size.

Now, I do Usability testing with larger sample size that validates a lot of these firmware functionality. But it isn't officially verifying the input/output at the firmware "language" level. Does anyone have some insight on proper sampling method for this situation?
 
Elsmar Forum Sponsor

yodon

Leader
Super Moderator
#2
The reason for a sample size is to account for variability. Usability testing has quite a bit of variability through the users.

Software, though, given the same inputs and conditions, is (supposed to be) deterministic and so doing more that one test given the same setup will give the same result. So, yes, a 'sample size' of 1 is appropriate for software tests. Given that you have, in your example, 2 hardware platforms, it is appropriate to test on both platforms.

The point about software verifying itself at startup is outside the scope of formal firmware verification (although, of course, the verification process itself would be verified).
 
Thread starter Similar threads Forum Replies Date
F Firmware as SOUP - Sensor with third party produced firmware IEC 62304 - Medical Device Software Life Cycle Processes 2
H Question about implications of performing Firmware upgrade via MDDS - Medical Device Data Systems 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 2
D Documentation Requirements for Firmware in a 510(k) Submission Other US Medical Device Regulations 1
T Firmware version in EMC report / DoC CE Marking (Conformité Européene) / CB Scheme 1
C Is the firmware in a component OTS if it is customised? Other US Medical Device Regulations 2
A Dental Laser Certification IEC 62304 for a Firmware IEC 62304 - Medical Device Software Life Cycle Processes 1
T Information and Structure for Technical Documentation for Firmware ISO 13485:2016 - Medical Device Quality Management Systems 1
T Necessary test documents for Class II Medical Device Hardware and Firmware ISO 13485:2016 - Medical Device Quality Management Systems 9
T 510k Submission - Where to describe firmware and firmware design documentation? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 6
S Medical Device Software (Firmware) Validation with using IEC62304 IEC 62304 - Medical Device Software Life Cycle Processes 21
B Requirement for Updating Medical Device Software (Firmware) in the Field 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 11
S Predicate Device Requirements - Firmware vs. Software Updates 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 3
M Firmware Validation Protocol/template - FDA regulated Medical Devices industry 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 5
S Is firmware or embedded software subject to the same standards as software? Qualification and Validation (including 21 CFR Part 11) 9
Le Chiffre Anyone with experience in open-source firmware in medical devices ISO 13485:2016 - Medical Device Quality Management Systems 5
L Software control program to handle firmware we develop - Seeking Procedure Software Quality Assurance 1
Q Help With Calibration of Software - Microcontroller firmware General Measurement Device and Calibration Topics 2
K ISO 17025 Method Validation and Verification for Test Lab ISO 17025 related Discussions 4
C IATF 16949:2016 Major NC pertaining to 7.1.5.2.1 Calibration/verification records IATF 16949 - Automotive Quality Systems Standard 15
T Who's responsible for verification of customer specifications? IATF 16949 - Automotive Quality Systems Standard 12
J Design Input Verification 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 3
G Injection Molded Parts in Verification & Validation Other Medical Device and Orthopedic Related Topics 3
M Root Cause and Corrective Action for CAPA's lacking validation/verification ISO 13485:2016 - Medical Device Quality Management Systems 19
B Use of Statistical Techniques in Design Verification Design and Development of Products and Processes 18
G Software verification vs. system verification IEC 62304 - Medical Device Software Life Cycle Processes 3
S Nadcap - Heat Treatment Proof of Verification question for Digital results AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 5
placebo_master What is the difference between "Fitness for Purpose" and "Calibration/Verification"? General Measurement Device and Calibration Topics 6
G Pad Printing Validation OR Verification ISO 13485:2016 - Medical Device Quality Management Systems 4
G Seeking reference guides/ documentation /tips on verification best practices Other Medical Device Related Standards 6
G Verification of Expected Service Life: Max # Reprocessing Cycles IEC 60601 - Medical Electrical Equipment Safety Standards Series 4
G IATF 7.1.5.2.1 Calibration/verification records :Program/software verification IATF 16949 - Automotive Quality Systems Standard 7
G Calculating Ppk for Design Verification - Variable Sampling Design and Development of Products and Processes 15
A Formel Q Konkret : § 4.3.1 D/TLD verification IATF 16949 - Automotive Quality Systems Standard 1
J Design Verification Testing and Statistics Reliability Analysis - Predictions, Testing and Standards 3
G How to Record Informal Testing (Not Verification/Validation) Other Medical Device and Orthopedic Related Topics 15
C Items used for Design Verification Design and Development of Products and Processes 7
I User ID verification for in-house e-signature compliance Qualification and Validation (including 21 CFR Part 11) 2
R Validation of Software used in Verification Testing ISO 13485:2016 - Medical Device Quality Management Systems 2
R SaMD Verification & Validation IEC 62304 - Medical Device Software Life Cycle Processes 6
H CAPA effectiveness verification methods for different types of CA Nonconformance and Corrective Action 10
J Managing design verification regression testing of design changes Design and Development of Products and Processes 1
M Validation of Data verification tool per 21 CFR 820 Quality Assurance and Compliance Software Tools and Solutions 1
placebo_master Risks of executing a verification protocol against existing data ISO 13485:2016 - Medical Device Quality Management Systems 4
placebo_master Verification Protocols - when to implement them (and when to skip in lieu of a report) 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 4
I Software (SaMD) mobile application verification testing: objective evidence Medical Information Technology, Medical Software and Health Informatics 2
M Reference part calibration or verification General Measurement Device and Calibration Topics 12
J Verification of purchased product / supplier questionnaires ISO 13485:2016 - Medical Device Quality Management Systems 2
H Verification of the purchased products which are services, like continuous IT services ISO 13485:2016 - Medical Device Quality Management Systems 7
K Why does load cell supplier requires force verification General Measurement Device and Calibration Topics 3
D Use Error Risk Controls and Control Verification ISO 14971 - Medical Device Risk Management 6

Similar threads

Top Bottom