IEC 62366 - Summative Evaluation

[Koskis]

Thanks for your help and quick responses.

I know the HF&UE Guidance + IEC62366 and that the case is not fully complying with especially the FDA guidance. Open for me is the wiggle room: How FDA adapts these in practice for small company, low risk, SAMD product.

Iam not concerned about the actual use error based risks on the device based on the existing evidence, but based on your comments it seems there is a potential audit risk.

 

[yodon]

I wouldn't try to play the "small company" card. I've never seen that a successful approach for not doing something. The risk-based justification, as mentioned, might be the only reason FDA would accept the approach. It's not possible for any of us to say whether it would be successful or not. That's very much going to be just the opinion of the reviewer. And don't forget that the reviewer's conclusions are also reviewed.

 

[Tidge]

"Wiggle room" is not a good strategy for dealing with submissions. My experience with the FDA has been that at first order they go through the list of required/expected deliverables, and if something is missing (or if they don't recognize it being present) it gets flagged. The next amount of effort by the reviewer is dedicated to the quality of the submitted material. They won't waste their time in the initial submission reading justifications for what they see as missing elements. The best outcomes I've seen are when a reviewer genuinely doesn't recognize that work they expected to see was included in the submission. In my case, this has only happened with relatively green reviewers of submissions that include software elements of greater systems; we never had a real issue with software it's just that I think the reviewer's eyes glazed over when confronted with the 62304 appendices and didn't even open them. The "buddy supervisor" reviewer at the agency was very patient with the peer and us, with no hard feelings.

This is simply my opinion: Arguing (in the submission packet) that you don't have to do something is a red flag. Not including a required part of a submission and trying to rationalize the deficiency (either by some "clever" wording in the initial submission, or during the follow-up question period) is a losing strategy. Do the work such that the arguments are about the quality of the work done, instead of trying to make the argument be about the quality of the arguments.

 

[Koskis]

....and now for something completely different

About summative testing scope: In addition to critical tasks (or hazardous use scenarios with defined selection criteria) related originating from use errors do you include also the user facing mitigations which are traced from system failure modes?

Examples below (Severity of related harm in both situation higher than severe):
1) As a part of risk assessment, potential software failure mode was found which would lead to wrong patient's information being shown in rare situations in SaMD solution. (Bad) Mitigation could be a in addition to SW related mitigations an indication / warning in the system about ensuring right patient is selected, or a caution in the manual for actual end user.
2) Symbols related to electrical safety, originating from standards and are placed to the device to avoid electrical shock.

Would the examples in 1 or 2 be in the scope of summative evaluation (task based - or knowledge test)?

 

[Tidge]

In addition to critical tasks (or hazardous use scenarios with defined selection criteria) related originating from use errors do you include also the user facing mitigations which are traced from system failure modes?

Yes. "How much" summative testing goes into this effort depends on the nature of the risk and the specifics of the line of risk analysis. The user classes affect the nature of the summative tests as well; for example: I don't know if the issue with unsterilized endoscopes infecting patients could be partially blamed on poor summative testing of the users (a highly trained user class), even though I know that "new instructions to users" were proposed as part of the risk controls. I'm not close enough to those product families to know if design changes were also implemented to control those risks.

For electrical safety symbols, those originate in a consensus standard so I personally would not dedicate much (if any) summative testing. Off the top-of-my-head the most I would do during a summative test is ask a participant if they notice certain symbols and ask them what reference they would use to understand them. If they answered "the IFU", I would consider this a passing result.

Active indicators typically get more summative testing. I motivate this because of the medical electrical devices I work with include alarm systems, and I try to make it clear in the risk management files about the distinction between elements of the alarm system and any features which indicate something but don't specifically originate/participate in a line of risk analysis.

 

[yodon]

Risk management (14971) expects you to demonstrate that the risk controls are effective. I often use a summative study to support my assertions that risk controls are effective.

 

[Solyneu]

Hello, I have a question that has been more or less adressed on this topic but I would benefit from recent advice

In this situation:

• None of hazard-related use scenarios identified before usability studies lead to a severe harm caused by a use error
• All these use-related risks are evaluated as acceptable, even BEFORE risk control
• 3 rounds of formative evaluation were conducted, 2 of which were composed of usability tests
• All previously identified hazard-related use scenarios were tested
• New hazard-related use scenarios were identified, but none lead to a severe harm caused by a use error. These newly identified use-related risks are evaluated as acceptable, even BEFORE risk control

Is a summative evaluation required? The standard mentions that:

OBJECTIVE EVIDENCE to support the determination that use-related RESIDUAL RISK has been reduced to acceptable levels is generated by conducting SUMMATIVE EVALUATION of the USABILITY of the USER INTERFAC

So I guess my question can be summarized as: is objective evidence needed, when risks were evaluated as acceptable level even before risk control?

Thank you,

 

[yodon]

Is a summative evaluation required?

That's a difficult one to answer and should be driven more by your regulatory folks. Although you provided good information, it may not be sufficient for anyone here to make a definitive statement.

If it were me, though, I would probably lean towards doing some form of a summative study. Your formative studies may not have considered all relevant use scenarios or user groups. Having users actually operate the system provides really good intel. While 62366 is focused on safety, you still have an adoption factor to consider. If the system frustrates users, you may have difficulty getting good market traction.

 

[DanMann]

I think it would be very difficult to justify having no summative evaluation to a regulatory agency or notified body; even though in principle I would be supportive of this approach where it is legitimate that no serious harm could be caused by the device, I think it would be a serious regulatory risk not to have something.

 

[Vicky Santos]

This is an old thread, but I think my question is quite close to the topic.
I would like to get some idea or advise on Usability Engineering evaluation per EN 62366-1:2015 . Our medical device under evaluation is a Single Use Injector System, intended for use in the insertion of intraocular lens into the eye during cataract surgery. It is a class IIa device, with contact duration =less than 24 hrs. It is also a WET-established device. For the Usability Engineering verification we performed, a summative test, participants are our technical staff and not the intended user (Surgeon). Our NB questioned us and requested us to justify this.
Can someone help me to provide reference guidelines for justification? Appreciate so much your advise. Thanks!