Summative evaluation: how many sample per participant?

Mhalik

Starting to get Involved
Hello,
I am currently working in a company who has products on the market for a lot of decades, so all their usability dossiers were based on the annex C (uoup as they did not have usability process when the device were designed) : there was no summative evaluation ever.
The standard mentions that in case of user interface change, the new user interface (or part of user interface) shall be part of the usability process. There is currently a change of user interface (hardware device) and we are going to need a summative evaluation for the validation. I am in charge of the protocol.

I justified the number of participants based on the risk level VS. complexity of the interface and I am happy with the rationale. However I don't really know how many times the participants shall repeat the tested scenario. How do you usually decide this?
I would go for something like "the user can stop testing, if the user still obtain the same results/the same feeling after X repetitions."
But still... How to define the number X? Just an arbitrary number such as 4 or 5 ?

Thank you in advance for you thoughts!

Mhalik
 

yodon

Leader
Super Moderator
Normally your sample size would be the number of study participants (and at least FDA guidance indicates minimum 15 of each user group).

Maybe I'm missing something but why would a user do anything different in successive attempts?
 
Additional thought. The evaluation should mimic the real-world use as much as possible. Would repetitions happen in a real use environment? If not, it does not make sense to include a repetition in your summative evaluation.
 

Tidge

Trusted Information Resource
With summative validation, our (complex) devices do not require multiple runs with individual users. I don't want to pass myself off as an expert in usability, but it seems like common sense that for general users there are two foreseeable circumstances:
  • Users get better at the tasks the more they do them, so rather than a first-order test of usability you are getting second-order information about the mental models of individual users in their execution of tasks... and it would be complicated to plan a study design to account for such things.
  • Users may not be fully engaged with the usability study after the third time you have them execute the same tasks. In operations, we put controls in for this (including task-specific training and evaluation), but such things would not be appropriate for user validations.
 

Mhalik

Starting to get Involved
Thank you for all your thoughts! It is very interesting.

@yodon : I was mainly wondering because we seek also for difficulties, in case they could lead to use error, and for some of our scenarios the difficulty of the task is only related to the surgeon "feelings". Thewe though that for these scenarios, 1 time may not be enough for the user to conclude on the difficulty (note that the device does not require training) However I think Tidge is right here, the first use is the worst case, as the user is not yet used to the manipulation so more likely to make it wrong.

Have a good day!
 

shpponer

Registered
Hello,
I am currently working in a company who has products on the market for a lot of decades, so all their usability dossiers were based on the annex C (uoup as they did not have usability process when the device were designed) : there was no summative evaluation ever.
The standard mentions that in case of user interface change, the new user interface (or part of user interface) shall be part of the usability process. There is currently a change of user interface (hardware device) and we are going to need a summative evaluation for the validation. I am in charge of the protocol.

I justified the number of participants based on the risk level VS. complexity of the interface and I am happy with the rationale. However I don't really know how many times the participants shall repeat the tested scenario. How do you usually decide this?
I would go for something like "the user can stop testing, if the user still obtain the same results/the same feeling after X repetitions."
But still... How to define the number X? Just an arbitrary number such as 4 or 5 ?

Thank you in advance for you thoughts!

Mhalik

Hi Mhalik @Mhalik , that is a very good question. See below for my answers:

I would not advise to determine the sample size based on the risk level or complexity of the user interface. The sample size is typically based on either; 1) the requirements set forth by notified bodies, and 2) the minimum number of participants that are advised to yield a sufficient and valid amount of test performance data.

Regarding the participant sample size, there are a few considerations that need to be made:

1. If the device is intended for an FDA submission, you should include a minimum of 15 participants per distinct user group.
2. If the device is solely intended for European MDR submission, I typically advise a minimum of 6 participants as this reveals most of the major usability problems that might occur. Whereas, 8 participants has been shown to yield 90% of the major usability issues that might occur.

Regarding the repetition of a use scenario / task, consider the following:

1. Are you also evaluating accompanying instructional documentation? If yes, then consider repeating the use scenario and require participants to read the instructional document step-by-step during the second scenario (such that you can evaluate the documentation's clarity and effectiveness as well)

In any other case, I would not consider repeating scenarios as during a summative test you want to validate whether the device is safe and effective for first-time use. Please feel free to ask any questions if anything is unclear.
 
Top Bottom