IEC 62366 - Summative Evaluation

[Julie_]

Hi ,

I was wondering if anyone has experience with medical device usability activities, in which none of your identified critical tasks have an unacceptable risk.

In this case, would you still need to perform Summative Evaluation? I assume you still need to do it, but on which criteria do you select your scenarios in this case?

Anyone who can provide me with some advise regarding this?
Would be very much appreciated!

 

[yodon]

If you follow the 62366 standard, you select (for the summative studies) either:
– all HAZARD-RELATED USE SCENARIOS; or
– the subset of the HAZARD-RELATED USE SCENARIOS based on the SEVERITY of the potential HARM that could be caused by USE ERROR (e.g. for which medical intervention would be needed).

So not just those that have unacceptable risk.

 

[Julie_]

Thank you for your reply!

Does this mean that if you could not identify hazard-related use scenarios with serious potential harm that could be caused by use error, you should select all hazard-related use scenarios?

Or can you then still choose a subset of hazard-related use scenarios? For example those that have the most severe potential harm, even if that harm is identified as minor or negligible?

 

[yodon]

The standard allows you to select which scenarios. You have to provide rationale for your selection (or exclusion).

Interestingly, the companion Technical Report (62366-2 - application of the standard) adds another option to those listed above: include a subset of the HAZARD-RELATED USE SCENARIOS based on the SEVERITY and additional circumstances specific to the MEDICAL DEVICE and the MANUFACTURER.

The Report goes on to say that:

For more complicated MEDICAL DEVICES, the MANUFACTURER might limit the HAZARD-RELATED USE SCENARIOS included in the SUMMATIVE EVALUATION to those with a SEVERITY for which medical intervention would be needed (option 2). Sometimes, the MANUFACTURER might also want to consider other circumstances specific to the MEDICAL DEVICE and the MANUFACTURER when selecting the most critical HAZARD-RELATED USE-SCENARIOS (option 3).

So you may be able to justify focusing on just the higher-risk scenarios. I would avoid trying to use time or budget as a justification, though.

 

[HenkP]

The thread is 'old', yet the question remains an interesting one.

All standards and regional guidelines, like the FDA's, on Usability / HFE state that Human Factors engineering 'focuses on usability as it relates to safety.'

62366-1:2019 This International Standard strictly focuses on applying the USABILITY ENGINEERING PROCESS to optimize MEDICAL DEVICE USABILITY as it relates to SAFETY.
62366-2 brings TASK accuracy, completeness and EFFICIENCY, and USER satisfaction into scope for the design, not for the Summative testing.

FDA guidance: FDA is primarily concerned that devices are safe and effective for the intended users, uses, and use environments. The goal is to ensure that the device user interface has been designed such that use errors that occur during use of the device that could cause harm or degrade medical treatment are either eliminated or reduced to the extent possible.......
CDRH believes that for those devices where analysis of risk indicates that users performing tasks incorrectly or failing to perform tasks could result in serious harm, manufacturers should submit human factors data in premarket submissions (i.e., PMA, 510(k)).

So technically, If you are a brave man, or woman, you could argue that if your risk management process was thorough and did not reveal any residual risks that use error can result in serious harm, there is no need for summative usability testing at all. I'm not that brave though...

 

[Tidge]

So technically, If you are a brave man, or woman, you could argue that if your risk management process was thorough and did not reveal any residual risks that use error can result in serious harm, there is no need for summative usability testing at all. I'm not that brave though...

There is an applicable adage about the arms race between folks who make "idiot-proof" devices and the folks who make "idiots"....

I can think of two reasons why medical device designers (interested in controlling risks) would perform summative testing independent of specific usability standards:

1. A summative test will (at least) provide some measure of objective analysis to support the (does "bravery" imply "unsupported"?) qualitative analysis provided by an otherwise "thorough" risk analysis.
2. A summative test exposes the user classes directly to the finished (not prototype) device, in a way that is somewhat free of the design team who may have particular blind spots.

 

[HenkP]

There is an applicable adage about the arms race between folks who make "idiot-proof" devices and the folks who make "idiots"....

I can think of two reasons why medical device designers (interested in controlling risks) would perform summative testing independent of specific usability standards:

1. A summative test will (at least) provide some measure of objective analysis to support the (does "bravery" imply "unsupported"?) qualitative analysis provided by an otherwise "thorough" risk analysis.
2. A summative test exposes the user classes directly to the finished (not prototype) device, in a way that is somewhat free of the design team who may have particular blind spots.

To be clear, I was being sarcastic I'm not at all insinuating not to perform summative tests....

Having said that, if your need is 'only' to challenge and then improve your design with intended users, in the absence of R&D bias, that can be accomplished by formative tests with real users. That is easier if you have, or are, a dedicated Human Factors Engineer and have the 'luxury' of conducting these studies without too strong ties to the design team. But even as one of the designers it is possible to do objective testing with real users as long as you're able to zip your mouth and sit on your hands and are willing to let testers struggle for a bit to figure your design out, and find flaws or potential for improvement.

Your first bullet is right on the mark, regulators require objective proof based on observational data. The summative test (or usability validation in FDA talk) is really the official validation of your final product s' user interface to be reported and submitted as part of your regulatory submission. It presents, compared to the formative evaluations, a much larger workload in terms of plans, protocols, acceptance criteria, tester recruitment, observation studies, information for safety (Labeling) testing and reporting.

 

[Koskis]

I have related question to the scoping of usability activities in US market / FDA. Iam working with a class II therapeutical SaMD product for long term illness with 2 extensive clinical trials already done with over 50 end user patients and summative tests with 5 users (in EU). Only critical task risks are related to the device (non-medical device) rather than to the medical software. Their mitigation in SW was validated in the summative.

My question is: based on your experience on the FDA expectations, whats your guestimate: for US market and 510k approval would another 15 user summative usability test be needed or whether the its possible to justify the work done for the 510K report? Background is that even though 5 person summative test is soft proof for direct and traced critical task usability, the clinical trial already suggests stronger evidence.

Thanks

 

[Tidge]

If I were in your position, I would consult wit 62366-2 Annex E to see if you can leverage the usability data you have (possibly including what you have from the clinical trial), but based on Annex K I would say that 5 users seems like too small a sample size. If I were an independent reviewer of the larger results (outside of the formal summative testing) I would wan to know that those studies were established to make a sincere effort to collect usability data, rather than a post hoc rationalization of "the device has good usability!"

Frankly, I think usability in the SaMD area is one where you may not be likely to get much pushback, providing you can put together a coherent analysis, even based on 5 users.

 

[yodon]

FDA will tend to fall back on their HF & UE guidance which calls for a minimum of 15 participants in EACH user group. And with the original participants being in the EU, they may question the applicability to US users. You might be able to make a risk-based justification for the existing data, but I would think it would be a difficult case.