The following is my advice. You will not find support for this is any published consensus authority. I recommend application of testing commensurate with the level of risk. For medical device manufacturers, risk three groups should be considered.
- Risk to patients, users ("14971 risk")
- Regulatory, compliance risk ("13485" or "21 CFR 820" risk)
- Business risk ("C-suite risk")
- Protocols approved prior to execution
- Protocols with well defined and documented test conditions
- Protocols with appropriately and specifically trained test executors
- Protocols with complete and explicit traceability to requirements
- Protocols with well-established, predicted, and necessary results
Now that we've established what the most thorough end-point is, work backwards.
Unscripted testing is more relaxed. Individual organizations are free to relax any combination of the bullet points above. I have my own recommendations, but I will spare the forum any more polemics (at least in this post).
Ad hoc testing is literally just that: Testing done that isn't done per plan or procedure.
In the recent past (and present), medical device manufacturers (as well as pharmaceutical manufacturers) have been rightly concerned with the validation of medical device designs and manufacturing processes for pharmaceuticals... and those require scripted testing in order to satisfy regulatory requirements... which is why when people were trying to establish methodologies for "proving" that software systems met their business needs, they naively defaulted to the most stringent approach... which is entirely derived from '14971 risk' (for medical device manufacturers).
The FDA draft guidance will make it explicitly clear that if there is no risk to patients/users, that they will under no circumstances expect that the default NPS process for medical device manufacturers will require scripted testing, as is done for medical devices.
