Summative evaluation: how many sample per participant?

Mhalik

Starting to get Involved
#1
Hello,
I am currently working in a company who has products on the market for a lot of decades, so all their usability dossiers were based on the annex C (uoup as they did not have usability process when the device were designed) : there was no summative evaluation ever.
The standard mentions that in case of user interface change, the new user interface (or part of user interface) shall be part of the usability process. There is currently a change of user interface (hardware device) and we are going to need a summative evaluation for the validation. I am in charge of the protocol.

I justified the number of participants based on the risk level VS. complexity of the interface and I am happy with the rationale. However I don't really know how many times the participants shall repeat the tested scenario. How do you usually decide this?
I would go for something like "the user can stop testing, if the user still obtain the same results/the same feeling after X repetitions."
But still... How to define the number X? Just an arbitrary number such as 4 or 5 ?

Thank you in advance for you thoughts!

Mhalik
 
Elsmar Forum Sponsor

yodon

Leader
Super Moderator
#2
Normally your sample size would be the number of study participants (and at least FDA guidance indicates minimum 15 of each user group).

Maybe I'm missing something but why would a user do anything different in successive attempts?
 
#3
Additional thought. The evaluation should mimic the real-world use as much as possible. Would repetitions happen in a real use environment? If not, it does not make sense to include a repetition in your summative evaluation.
 

Tidge

Trusted Information Resource
#4
With summative validation, our (complex) devices do not require multiple runs with individual users. I don't want to pass myself off as an expert in usability, but it seems like common sense that for general users there are two foreseeable circumstances:
  • Users get better at the tasks the more they do them, so rather than a first-order test of usability you are getting second-order information about the mental models of individual users in their execution of tasks... and it would be complicated to plan a study design to account for such things.
  • Users may not be fully engaged with the usability study after the third time you have them execute the same tasks. In operations, we put controls in for this (including task-specific training and evaluation), but such things would not be appropriate for user validations.
 

Mhalik

Starting to get Involved
#5
Thank you for all your thoughts! It is very interesting.

@yodon : I was mainly wondering because we seek also for difficulties, in case they could lead to use error, and for some of our scenarios the difficulty of the task is only related to the surgeon "feelings". Thewe though that for these scenarios, 1 time may not be enough for the user to conclude on the difficulty (note that the device does not require training) However I think Tidge is right here, the first use is the worst case, as the user is not yet used to the manipulation so more likely to make it wrong.

Have a good day!
 
#6
Hello,
I am currently working in a company who has products on the market for a lot of decades, so all their usability dossiers were based on the annex C (uoup as they did not have usability process when the device were designed) : there was no summative evaluation ever.
The standard mentions that in case of user interface change, the new user interface (or part of user interface) shall be part of the usability process. There is currently a change of user interface (hardware device) and we are going to need a summative evaluation for the validation. I am in charge of the protocol.

I justified the number of participants based on the risk level VS. complexity of the interface and I am happy with the rationale. However I don't really know how many times the participants shall repeat the tested scenario. How do you usually decide this?
I would go for something like "the user can stop testing, if the user still obtain the same results/the same feeling after X repetitions."
But still... How to define the number X? Just an arbitrary number such as 4 or 5 ?

Thank you in advance for you thoughts!

Mhalik
Hi Mhalik @Mhalik , that is a very good question. See below for my answers:

I would not advise to determine the sample size based on the risk level or complexity of the user interface. The sample size is typically based on either; 1) the requirements set forth by notified bodies, and 2) the minimum number of participants that are advised to yield a sufficient and valid amount of test performance data.

Regarding the participant sample size, there are a few considerations that need to be made:

1. If the device is intended for an FDA submission, you should include a minimum of 15 participants per distinct user group.
2. If the device is solely intended for European MDR submission, I typically advise a minimum of 6 participants as this reveals most of the major usability problems that might occur. Whereas, 8 participants has been shown to yield 90% of the major usability issues that might occur.

Regarding the repetition of a use scenario / task, consider the following:

1. Are you also evaluating accompanying instructional documentation? If yes, then consider repeating the use scenario and require participants to read the instructional document step-by-step during the second scenario (such that you can evaluate the documentation's clarity and effectiveness as well)

In any other case, I would not consider repeating scenarios as during a summative test you want to validate whether the device is safe and effective for first-time use. Please feel free to ask any questions if anything is unclear.
 
Thread starter Similar threads Forum Replies Date
MaHoDie Summative Evaluation with Post-Market Data? Human Factors and Ergonomics in Engineering 2
Q Summative Usability Evaluation Testing: prior or during Clinical Investigation? Human Factors and Ergonomics in Engineering 10
T HF testing / Summative evaluation for MDDS class I necessary? Human Factors and Ergonomics in Engineering 2
MrTetris When a follow-up summative evaluation is not required? IEC 62366 - Medical Device Usability Engineering 1
J IEC 62366 - Summative Evaluation IEC 62366 - Medical Device Usability Engineering 6
MaHoDie Summative Usability test Human Factors and Ergonomics in Engineering 2
G Devices from IQ, OQ or PQ process to be used for verification, validation and summative? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 5
V 510(k) vs. CE Marking Summative Usability Testing Expectations IEC 62366 - Medical Device Usability Engineering 4
S Human Factor Summative Study Class III Device RA Checklist wanted Human Factors and Ergonomics in Engineering 2
R MDR Clinical Evaluation Plan EU Medical Device Regulations 4
R Clinical Evaluation - state of the art literature search EU Medical Device Regulations 1
J Clinical Evaluation For Instruments - Class I, Class Ir, Class Im, and Class IIa - EU MDR EU Medical Device Regulations 4
M Clinical evaluation interface with the risk management process EU Medical Device Regulations 9
L Clinical Evaluation MDR EU Medical Device Regulations 5
L Supplier performance evaluation Supplier Quality Assurance and other Supplier Issues 8
J Revamping Supplier Qualification, Re-evaluation, and Monitoring Requirements ISO 13485:2016 - Medical Device Quality Management Systems 6
J ISO 9001:2015 Is a Project Evaluation form necessary? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
I Clinical evaluation for legacy device EU Medical Device Regulations 16
M Clinical evaluation report training EU Medical Device Regulations 3
K Continue to do clinical evaluation? or transfer to RA? or focus on clinical studies as MA? Career and Occupation Discussions 1
O CLINICAL EVALUATION - VACCUM PLANTS Other Medical Device Related Standards 4
B Evaluation of suppliers API Spec Q1 5.6.1.2c Oil and Gas Industry Standards and Regulations 2
B Creation and evaluation of new jobs Oil and Gas Industry Standards and Regulations 0
E DESIGN VALIDATION, USABILITY AND CLINICAL EVALUATION request Medical Device and FDA Regulations and Standards News 0
SANTHSH Consumables, Paints, End-covers, NDE Chemicals & Tools suppliers classification and evaluation Oil and Gas Industry Standards and Regulations 6
Aymaneh IVDR and Performance Evaluation Plan EU Medical Device Regulations 2
S Clinical Evaluation of equivalent device EU Medical Device Regulations 6
L Clinical evaluation plan EU Medical Device Regulations 17
C Biologic Evaluation based on ISO 10993-1 EU Medical Device Regulations 2
M Selling non-CE marked devices for evaluation EU Medical Device Regulations 4
M Procedure for clinical evaluation according to new MDR EU Medical Device Regulations 11
Q Process map Evaluation and Analysis Method Process Maps, Process Mapping and Turtle Diagrams 5
M Supplier evaluation Supplier Quality Assurance and other Supplier Issues 5
I QMS monitoring, measurement, analysis and evaluation requirement - Template ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
T Biological Evaluation (10993) & Risk Management ISO 14971 - Medical Device Risk Management 9
shimonv Clinical evaluation report for class I device EU Medical Device Regulations 18
A Applicability of Photobiological Safety Evaluation for LED used in medical devices Reliability Analysis - Predictions, Testing and Standards 2
P GSPRs / Clinical Evaluation EU Medical Device Regulations 3
silentmonkey Overall Benefit/Risk Analysis - Risk Management VS Clinical Evaluation ISO 14971 - Medical Device Risk Management 3
M NICE Medical Technology Evaluation Programme - Recommendations Service Industry Specific Topics 0
S Regular updates of clinical evaluation report EU Medical Device Regulations 6
K Surface finish (Evaluation Length) Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
M Clinical Evaluation Benchmark vs. Equivalent EU Medical Device Regulations 2
MaHoDie IVDR and Performance Evaluation Plan CE Marking (Conformité Européene) / CB Scheme 2
S User evaluation for self monitoring blood glucose test systems US Food and Drug Administration (FDA) 4
S Australia TGA Clinical Evaluation Report (CER) Other Medical Device Regulations World-Wide 0
SANTHSH API Spec. Q1 clause 5.6.1.2 On site evaluation Oil and Gas Industry Standards and Regulations 12
B Biological evaluation plan and report Other Medical Device Related Standards 7
F Biocompatibility evaluation for Hardware/Interface Components? Other Medical Device Related Standards 13
B Clinical Evaluation Expert Panels - MDR EU Medical Device Regulations 1

Similar threads

Top Bottom