Verification and Validation as a Solo Engineer

[Jean_B]

I'm gathering from the examples given by @Peter Selvey the presence of any characteristics critical to quality (critical to performance, critical to safety) to be reason to infer the presence of or designate essential performance.

Essential performance is also related by @d_addams to an absence of benefit meaning the benefit-risk ratio becomes unacceptable by default.

Both of these can be academically true, but given limited knowledge/resources are hard to be ensured (and thus can only be assured).
Given that, my mind was that essential performance deals with those characteristics critical to quality that are so important they must be ensured for the device to make sense: life-saving or vitality-supporting, or vitality-monitoring for when deviations in monitored physiological conditions results in a need for immediate action to prevent death; it's hard for me to find the words to deal with permanent impairment so that's probably where my threshold for it lies. Note: this means I don't put the bar at serious injury which can be recovered from, but higher.

Note: this is turning tangential from the thread topic. I'm aware and won't continue a discussion (here).

 

[d_addams]

I'm gathering from the examples given by @Peter Selvey the presence of any characteristics critical to quality (critical to performance, critical to safety) to be reason to infer the presence of or designate essential performance.

Essential performance is also related by @d_addams to an absence of benefit meaning the benefit-risk ratio becomes unacceptable by default.

Both of these can be academically true, but given limited knowledge/resources are hard to be ensured (and thus can only be assured).
Given that, my mind was that essential performance deals with those characteristics critical to quality that are so important they must be ensured for the device to make sense: life-saving or vitality-supporting, or vitality-monitoring for when deviations in monitored physiological conditions results in a need for immediate action to prevent death; it's hard for me to find the words to deal with permanent impairment so that's probably where my threshold for it lies. Note: this means I don't put the bar at serious injury which can be recovered from, but higher.

Note: this is turning tangential from the thread topic. I'm aware and won't continue a discussion (here).

yes its getting tangential regarding the definition and use of Essential Performance, but in the context of 'advice for a solo practitioner', ones misapplication of the standards is a risk and something a solo practitioner should have mitigations in place to protect against (methods and ways to get 2nd opinions or independent review).

 

[mr9000]

I think it would have been best would I have not mentioned the essential performance bit. But I always cherish a discussion in this forum, so we got that at least.
While I all get your points for declaring essential performance, I don't know what else to say than I personally have seen medical devices claiming no essential performance being sold in the field, cleared by TÜV. And for these kind of products it makes sense. They aid therapy and healing, but are not part of standard procedures. So from a risk-based approach, them not working (e.g. their performance) does not produce any harm. They still have basic safety.
1/3 of devices failing or using cheap components IMO relates to business risks, not patient risks. Applying common sense to the development process, I don't think it should be necessary to bring in the medical perspective into that sort of problem. So although it may be fine from the patient side, your approach of making money and satisfying customers should step in. That includes the claimed performance of the device. They are tested, perform self-tests and inform the user if they are not working anymore.

If the product is supposed to do X to derive medical benefit (measuring the patients temperature), not measuring the temperature does not cause harm (the patients state is not changed by not measuring their temperature), but the product has now become useless and there is no benefit to placing it in their mouth, only risk. Exposing the patient to no benefit, only to risk is UNACCEPTABLE. This is not the same as saying the benefit/risk of the thermometer product is unacceptable. In this case the essential performance of the thermometer is to report the patients temperature. If the product can not do that, it only creates risk and there is no available benefit to offset the risk.

In that case, how do you tackle that risk? Include a back-up battery? Implementing an alarm system? Or do you expect to know the patient does not insert a thermometer if the display stays off? There has to be a cut-off to risk minimizing steps. There are medical devices requiring backups of important components for single fault conditions. But if you can demonstrate for your intended use and your risk management that it is all ok, why bother? We're not flying to the moon.

your example of people justifying the failure because 'according to 60601 this isn't essential performance because the failure condition is acceptable thus not essential performance' is the exact thought experiment to demonstrate this is NOT how assigning a function as essential performance should be used.

I have learned to refer to normative standards EXACTLY AS STATED. And not being afraid of discussions. In a perfect world, there should be no room for two opinions, but as we can see in this forum there is still a lot to discuss ;-) See the discussion in another thread about using alarm signals / information signals and the need of implementing 60601-1-8. I did not study law because I'm not a big friend of choosing words carefully and precisely while every addressat clearly gets your intention, but that's the way it is in this field.

Explain the above as part of your QMS for being a solo engineer.

I do not have my own QMS, I'll be working under the clients QMS as sort of temporary expert. They need to update their QMS to MDR alongside this product development so "we" get the chance to define the processes etc. now from the start. So here's me asking if anyone knows some approaches that can be done with a single developer as effective as possible ;-) In the end all documents will be signed by me and the client. My main concern is not the main documentation (requirement specifications, design documents, ...) but especially the meta validation of the verification methods. Are there objective ways I can do the verification (especially of software) mostly by myself and how? Or is it impossible to develop a medical device with only one developer?

Thank you all for taking part in the discussion!

 

[Loekje]

I respectfully disagree that there can be no medical device without Esential Performance.
The company I worked for before my current employment had a whole range of diagnostic devices without essential performance.

You have to read IEC-60601-1 clause 3.27 litterally:
* ESSENTIAL PERFORMANCE performance of a clinical function, other than that related to BASIC SAFETY, where loss or
degradation beyond the limits specified by the MANUFACTURER results in an unacceptable RISK.

RISK is a direct unacceptable patient risk, not a business risk or risk that a diagnose does not come in a timely matter or whatever.
And unacceptable is to a degree to the manufacturer, where the test house/ notified body / FDA must agree upon.

There is no risk-benefit analyis in 60601 (except for 11.1.2), that is a different ISO-14791 ball game.

Annex A:
During the initial RISK ANALYSIS, the MANUFACTURER identifies the performance of the clinical
fuction(s) of the ME EQUIPMENT or ME SYSTEM that is necessary for achieving the INTENDED USE.
The MANUFACTURER also identifies other qualitative and quantitative characteristics that could
affect the safety of the ME EQUIPMENT or ME SYSTEM.
The performance limits specified by the MANUFACTURER could be anywhere within the full range
of intended performance in NORMAL CONDITION and SINGLE FAULT CONDITION, to no performance.

 

[Tidge]

I want to quote something written above:

I'd agree with above, the "no essential performance" is a slight of hand to keep things simple...

My experience with NRTLs (primarily UL, but also the associated circle of midwestern US test labs) has been that the guy doing the tests will shortcut discussions around certain testing, for many medical devices, as saying "this device has no essential performance".

It is practically impossible to get them to explain what they mean, or to put such a statement in writing... but in the 3rd edition era I've got the following sense:

0) The NRTLs are (for understandable reasons) holding to a "just test against the standard, as written" mentality... with a side of "don't bring your weasel clauses about risk management into my lab". [clause 4.5 was literally called "the weasel clause" in my training at UL]

1) The NRTLs do recognize that the 2nd edition approach to safety of medical devices was incomplete. The original example given (by UL, during 3rd edition training) was (simplified) "in 2nd edition, as long as the ventilator didn't catch on fire during testing it was safe. This meant that a tipped-over ventilator that stopped breathing for a patient was also safe. In 3rd edition it has to keep breathing."

2) NRTLs don't want to think to hard about risk management, like their clients have to. All their clients do things differently, and so besides checking that there exists a risk management process, they really don't want to see any outputs from it (my experience), with one exception: They will ask for a 'essential performance statement' so that they can add some product-specific test to be done after they do their standard tests. (*1)

3) When a medical device is covered by a particular standard, I've seen them just say "there is no essential performance, I'm just going to test against the particular (as well as the standard and the collaterals)." I *think* this is basically the tester not wanting to think to hard about any specific device, with the rational that the particular covers all possible elements of risk... but a NRTL can take this approach based on the informative annex of 60601-1... it's just that the test lab doesn't say "essential performance is defined in 60601-2-XX", instead they say "there is no essential performance."

Outside of "essential performance" being in the title of, and defined in, 60601-1 3rd... it isn't a term that (to my knowledge) appears in regulations or standards like 14971... so outside of the 60601-1 context, "essential performance" doesn't have a consensus definition. I can explain what I think it should mean, but as a practical matter it can't go beyond: performance of a clinical function, other than that related to basic safety, where loss or degradation beyond the limits specified by the manufacturer results in unacceptable risk.

(*1) There is a potential problem for NRTL clients when testers (who generally apply some common sense) get a little too imaginative about what they think a medical device should do under some testing... and occasionally a medical device won't do what the tester expects, but has been examined by the client's risk management process to be "acceptable" (in the risk acceptability framework). At this point it often comes down to the NRTL (which doesn't want to think about risk management) not believing the client's risk management process or artifacts... despite having "inspected it" per 60601-1.

 

[EmiliaBedelia]

1/3 of devices failing or using cheap components IMO relates to business risks, not patient risks. Applying common sense to the development process, I don't think it should be necessary to bring in the medical perspective into that sort of problem. So although it may be fine from the patient side, your approach of making money and satisfying customers should step in. That includes the claimed performance of the device. They are tested, perform self-tests and inform the user if they are not working anymore.

sidestepping 60601.... this isn't fulfilling design controls, GSPR 1 (Devices shall achieve the performance intended by their manufacturer), or GSPR 4 (mfgs. must eliminate risks and reduce as far as possible through safe design and manufacture).

You need acceptance criteria and appropriate sample size (risk based) for design verification. If you have designed your device to have a function (regardless of whether it is "essential" or not), you need to verify/validate its performance. If you are indeed accepting a 33% failure rate (or 10%, or 5%, or whatever) you need a clinically relevant/risk based justification for why that requirement is appropriate and why you can't reduce that risk any farther. If it's not important whether a feature works or not, then what's the point?

If the purpose of the device isn't defined as a device per the Medical Devices Regulation, it isn't a device. So your issue should be, how do I market this item without making claims about its medical purpose?

If it is a medical device, it needs an intended purpose, and it needs to meet it according to your specifications. The specification is up to you, but you have to set it based on the clinical intended use and the state of the art. There's a difference between "This device sometimes doesn't produce a clinical benefit even if it's used correctly" and "This device sometimes doesn't work at all". It's not appropriate to rationalize design/manufacturing failure based on a weak clinical benefit.

Quite honestly, MDR is a nightmare for even companies that have millions to spend on MDR compliance so you are in for a lot of work to get ready. It's a very different ballgame from any experience you have with MDD so please do not assume that you know what you are in for based on that experience. To get certified you need!!!!!!! to connect risk management, design and development, clinical evaluation, and postmarket surveillance. From an engineering perspective, you can certainly do stuff on your own, but you absolutely need to align with the clinical side to ensure that your risk assessment is clinically appropriate, and your verification and validation will appropriately address all the risks. What will happen to your client when you are no longer freelancing for them? You should get with those people now and get aligned.

Anyway, going back to V&V: the more objectively you define your requirements, the easier it is to be objective in how you verify them. If there are applicable standards or white papers or anything available to justify how you translated your user needs into your specifications, use them. Calibration/qualification for any standard test equipment should cover the technical aspect of whether your measurements are appropriate.