IEC 62366 - Summative Evaluation

J

Julie_

#1
Hi ,

I was wondering if anyone has experience with medical device usability activities, in which none of your identified critical tasks have an unacceptable risk.

In this case, would you still need to perform Summative Evaluation? I assume you still need to do it, but on which criteria do you select your scenarios in this case?

Anyone who can provide me with some advise regarding this?
Would be very much appreciated!
 
Elsmar Forum Sponsor

yodon

Leader
Super Moderator
#2
If you follow the 62366 standard, you select (for the summative studies) either:
– all HAZARD-RELATED USE SCENARIOS; or
– the subset of the HAZARD-RELATED USE SCENARIOS based on the SEVERITY of the potential HARM that could be caused by USE ERROR (e.g. for which medical intervention would be needed).

So not just those that have unacceptable risk.
 
J

Julie_

#3
Thank you for your reply!

Does this mean that if you could not identify hazard-related use scenarios with serious potential harm that could be caused by use error, you should select all hazard-related use scenarios?

Or can you then still choose a subset of hazard-related use scenarios? For example those that have the most severe potential harm, even if that harm is identified as minor or negligible?
 

yodon

Leader
Super Moderator
#4
The standard allows you to select which scenarios. You have to provide rationale for your selection (or exclusion).

Interestingly, the companion Technical Report (62366-2 - application of the standard) adds another option to those listed above: include a subset of the HAZARD-RELATED USE SCENARIOS based on the SEVERITY and additional circumstances specific to the MEDICAL DEVICE and the MANUFACTURER.

The Report goes on to say that:

For more complicated MEDICAL DEVICES, the MANUFACTURER might limit the HAZARD-RELATED USE SCENARIOS included in the SUMMATIVE EVALUATION to those with a SEVERITY for which medical intervention would be needed (option 2). Sometimes, the MANUFACTURER might also want to consider other circumstances specific to the MEDICAL DEVICE and the MANUFACTURER when selecting the most critical HAZARD-RELATED USE-SCENARIOS (option 3).

So you may be able to justify focusing on just the higher-risk scenarios. I would avoid trying to use time or budget as a justification, though. :)
 
#5
The thread is 'old', yet the question remains an interesting one.

All standards and regional guidelines, like the FDA's, on Usability / HFE state that Human Factors engineering 'focuses on usability as it relates to safety.'

62366-1:2019 This International Standard strictly focuses on applying the USABILITY ENGINEERING PROCESS to optimize MEDICAL DEVICE USABILITY as it relates to SAFETY.
62366-2 brings TASK accuracy, completeness and EFFICIENCY, and USER satisfaction into scope for the design, not for the Summative testing.

FDA guidance: FDA is primarily concerned that devices are safe and effective for the intended users, uses, and use environments. The goal is to ensure that the device user interface has been designed such that use errors that occur during use of the device that could cause harm or degrade medical treatment are either eliminated or reduced to the extent possible.......
CDRH believes that for those devices where analysis of risk indicates that users performing tasks incorrectly or failing to perform tasks could result in serious harm, manufacturers should submit human factors data in premarket submissions (i.e., PMA, 510(k)).

So technically, If you are a brave man, or woman, you could argue that if your risk management process was thorough and did not reveal any residual risks that use error can result in serious harm, there is no need for summative usability testing at all. I'm not that brave though...
 

Tidge

Trusted Information Resource
#6
So technically, If you are a brave man, or woman, you could argue that if your risk management process was thorough and did not reveal any residual risks that use error can result in serious harm, there is no need for summative usability testing at all. I'm not that brave though...
There is an applicable adage about the arms race between folks who make "idiot-proof" devices and the folks who make "idiots"....

I can think of two reasons why medical device designers (interested in controlling risks) would perform summative testing independent of specific usability standards:
  1. A summative test will (at least) provide some measure of objective analysis to support the (does "bravery" imply "unsupported"?) qualitative analysis provided by an otherwise "thorough" risk analysis.
  2. A summative test exposes the user classes directly to the finished (not prototype) device, in a way that is somewhat free of the design team who may have particular blind spots.
 
#7
There is an applicable adage about the arms race between folks who make "idiot-proof" devices and the folks who make "idiots"....

I can think of two reasons why medical device designers (interested in controlling risks) would perform summative testing independent of specific usability standards:
  1. A summative test will (at least) provide some measure of objective analysis to support the (does "bravery" imply "unsupported"?) qualitative analysis provided by an otherwise "thorough" risk analysis.
  2. A summative test exposes the user classes directly to the finished (not prototype) device, in a way that is somewhat free of the design team who may have particular blind spots.
To be clear, I was being sarcastic I'm not at all insinuating not to perform summative tests....

Having said that, if your need is 'only' to challenge and then improve your design with intended users, in the absence of R&D bias, that can be accomplished by formative tests with real users. That is easier if you have, or are, a dedicated Human Factors Engineer and have the 'luxury' of conducting these studies without too strong ties to the design team. But even as one of the designers it is possible to do objective testing with real users as long as you're able to zip your mouth and sit on your hands and are willing to let testers struggle for a bit to figure your design out, and find flaws or potential for improvement.

Your first bullet is right on the mark, regulators require objective proof based on observational data. The summative test (or usability validation in FDA talk) is really the official validation of your final product s' user interface to be reported and submitted as part of your regulatory submission. It presents, compared to the formative evaluations, a much larger workload in terms of plans, protocols, acceptance criteria, tester recruitment, observation studies, information for safety (Labeling) testing and reporting.
 

Koskis

Starting to get Involved
#8
I have related question to the scoping of usability activities in US market / FDA. Iam working with a class II therapeutical SaMD product for long term illness with 2 extensive clinical trials already done with over 50 end user patients and summative tests with 5 users (in EU). Only critical task risks are related to the device (non-medical device) rather than to the medical software. Their mitigation in SW was validated in the summative.

My question is: based on your experience on the FDA expectations, whats your guestimate: for US market and 510k approval would another 15 user summative usability test be needed or whether the its possible to justify the work done for the 510K report? Background is that even though 5 person summative test is soft proof for direct and traced critical task usability, the clinical trial already suggests stronger evidence.

Thanks
 

Tidge

Trusted Information Resource
#9
If I were in your position, I would consult wit 62366-2 Annex E to see if you can leverage the usability data you have (possibly including what you have from the clinical trial), but based on Annex K I would say that 5 users seems like too small a sample size. If I were an independent reviewer of the larger results (outside of the formal summative testing) I would wan to know that those studies were established to make a sincere effort to collect usability data, rather than a post hoc rationalization of "the device has good usability!"

Frankly, I think usability in the SaMD area is one where you may not be likely to get much pushback, providing you can put together a coherent analysis, even based on 5 users.
 

yodon

Leader
Super Moderator
#10
FDA will tend to fall back on their HF & UE guidance which calls for a minimum of 15 participants in EACH user group. And with the original participants being in the EU, they may question the applicability to US users. You might be able to make a risk-based justification for the existing data, but I would think it would be a difficult case.
 
Thread starter Similar threads Forum Replies Date
M IEC 62366 Hazard-related Use Scenario vs FDA Critical Task EU Medical Device Regulations 3
H Regarding Usability Engineering File according to IEC 62366-1 CE Marking (Conformité Européene) / CB Scheme 2
adir88 Amendment 1 of IEC 62366-1 - Details yet? IEC 62366 - Medical Device Usability Engineering 2
K Primary Operating Functions (POF) in IEC 62366-1:2015. IEC 62366 - Medical Device Usability Engineering 7
V Applicability of IEC 62366-1 usability to resorbable bone substitutes IEC 62366 - Medical Device Usability Engineering 9
pashah Looking for Usability File Template acc. IEC 62366-1 and IEC 60601-1-6 IEC 62366 - Medical Device Usability Engineering 4
C Usability IEC 62366-1:2015 and MDR 2017/745 - Risk based approach IEC 62366 - Medical Device Usability Engineering 1
E Usability Process - Looking for a template (IEC 62366) IEC 62366 - Medical Device Usability Engineering 1
J IEC 62366 - Ethical approval IEC 62366 - Medical Device Usability Engineering 5
T I need to make test reports according IEC 62304 & IEC 62366 IEC 62366 - Medical Device Usability Engineering 2
C IEC 60601-1-6 2010 maps to IEC 62366:2007 - What about 62366-1:2015? IEC 60601 - Medical Electrical Equipment Safety Standards Series 4
K Usability of ME Equipment - IEC-62366-1:2015 IEC 62366 - Medical Device Usability Engineering 3
T IEC 62366 Test Report Form IEC 62366 - Medical Device Usability Engineering 20
M IEC 62366-1:2015 Usability Verification / Validation Plan example wanted IEC 62366 - Medical Device Usability Engineering 20
T IEC 62366:2015 Annex C - changed parts from 5.1-5.8 --> but what about 5.9? IEC 62366 - Medical Device Usability Engineering 3
S IEC 62366 - Usability Test Protocol format Imported Legacy Blogs 0
M IEC 62366-1 published - March 2015 IEC 62366 - Medical Device Usability Engineering 7
S IEC 62366 vs. FDA Human Factors Requirements Human Factors and Ergonomics in Engineering 2
Z Final Validation Unit Requirements - Annex D of IEC 62366 IEC 62366 - Medical Device Usability Engineering 6
Z IEC 62366 and Medical Device Usability Validation IEC 62366 - Medical Device Usability Engineering 10
C Application of IEC 62366 - Usability of Non-Active Medical Devices IEC 62366 - Medical Device Usability Engineering 15
T IEC 62366 included in 3rd ed IEC 60601? IEC 62366 - Medical Device Usability Engineering 1
M IEC 62366:2007 - Is FDA Usability Engineering Standard a Requirement in Canada? Canada Medical Device Regulations 5
S IEC 62366 - Embedded Software in separate Usability Engineering File? IEC 62366 - Medical Device Usability Engineering 1
J Where to buy standards such as IEC 62366:2007 Other Medical Device Related Standards 2
S IEC 62366 for Stand-Alone Medical Device Software and Usability Engineering IEC 62366 - Medical Device Usability Engineering 16
D Training on IEC 62366 (Medical Device Usability Engineering) for CE Mark? IEC 62366 - Medical Device Usability Engineering 5
M What do you think the revision of IEC 62366 should bring? IEC 62366 - Medical Device Usability Engineering 16
A IEC 60601-1-6 last release or IEC 62366? IEC 62366 - Medical Device Usability Engineering 17
K How to apply IEC 62366 and what shall be changed in establishing a Technical File ISO 13485:2016 - Medical Device Quality Management Systems 2
D IEC EN 62366 Usability Retroactive Compliance Other Medical Device Related Standards 1
R ISO IEC EN 62366 Usability Engineering Essential Requirements IEC 62366 - Medical Device Usability Engineering 8
P IEC 62366 vs. IEC 60601-1-6 - Has IEC 62366 now replaced IEC 60601-1-6? IEC 62366 - Medical Device Usability Engineering 19
H Regarding Double Insulation & Caution Symbol on Product according to IEC 61010-1 CE Marking (Conformité Européene) / CB Scheme 6
S PDF version of IEC 60601-1 IEC 60601 - Medical Electrical Equipment Safety Standards Series 2
P IEC 60601-2-25 requirements IEC 60601 - Medical Electrical Equipment Safety Standards Series 0
R Brexit IEC 61000-4-2 test point consideration IEC 60601 - Medical Electrical Equipment Safety Standards Series 2
MaHoDie Is it possible to assign medical software to security class A (according to IEC 62304)? EU Medical Device Regulations 5
Richard Regalado Informational ISO/IEC 27001:2022 has been published IEC 27001 - Information Security Management Systems (ISMS) 0
F IEC 60601-2-54:2009 Accuracy of CURRENT TIME PRODUCT IEC 60601 - Medical Electrical Equipment Safety Standards Series 2
C Sample qualification to be IEC 60601-1 series tested IEC 60601 - Medical Electrical Equipment Safety Standards Series 2
H IEC 62368-1 Scope of Testing CE Marking (Conformité Européene) / CB Scheme 3
A IEC 18562-1 report EU Medical Device Regulations 5
H IEC 61010-1 Clause No. 10.5 Resistance to Heat - Sub Clause 10.5.3 Insulating Material CE Marking (Conformité Européene) / CB Scheme 6
H Is testing according to IEC 60601-1 & IEC 60601-1-2 mandatory for a device being registered under MDR? IEC 60601 - Medical Electrical Equipment Safety Standards Series 4
I IEC 60812 or ISO 14971 for PFMEA? What should we use? ISO 14971 - Medical Device Risk Management 3
H Required explanation of sub clause 7.2.101.2 (Exception) of IEC 61010-2-020:2006 CE Marking (Conformité Européene) / CB Scheme 5
XRAY_3121 Class II heating pad - does the power supply have to have IEC 60601-1 testing?? US Medical Device Regulations 5
G History of IEC 60601-1:2005 IEC 60601 - Medical Electrical Equipment Safety Standards Series 3
H Pollution Degree (IEC 60664-1) CE Marking (Conformité Européene) / CB Scheme 2

Similar threads

Top Bottom