Firmware Verification Testing & Sample Size

[v1cph1rth]

Hi All,

We are developing custom firmware for our medical device (class II), and it is only intended to run on two versions of hardware. We are ISO 13485 and there is some leeway with statistical sample sizes/justification. I am struggling trying to understand if statistical sample size justification is required or not. Or if we can provide a justification for much smaller testing regime.

The firmware developer is trying to claim that they perform exhaustive testing of all features on all hardwares (n=2) . Testing more than 1 sample per hardware version is silly, because you are testing that the "bits" are transferring properly and working. The firmware also verifies itself on startup.

I am used to looking at a statistical plan for testing (ex: identifying confidence level and probability of finding a bug). I was balked at when throwing out a 90% CI with 15% bug finding resulting in a n=15 sample size.

Now, I do Usability testing with larger sample size that validates a lot of these firmware functionality. But it isn't officially verifying the input/output at the firmware "language" level. Does anyone have some insight on proper sampling method for this situation?

 

[yodon]

The reason for a sample size is to account for variability. Usability testing has quite a bit of variability through the users.

Software, though, given the same inputs and conditions, is (supposed to be) deterministic and so doing more that one test given the same setup will give the same result. So, yes, a 'sample size' of 1 is appropriate for software tests. Given that you have, in your example, 2 hardware platforms, it is appropriate to test on both platforms.

The point about software verifying itself at startup is outside the scope of formal firmware verification (although, of course, the verification process itself would be verified).