Usability Objectives and Verification & Validation Plan

F

frivolas

#1
Hi all!

I'm the Usability Engineer of my team, brand new to the industry, the company and the position, and we're currently designing a hand held and operated device. It has a simple interface with just a couple of buttons and two light bars, but it has some nice potential hazardous situations.

I've performed a function and task analysis and with the aid of a couple more analytical studies I've come with a comprehensive UFMEA that complies with our internal standards.

I'm starting to write the V&V plan for this project but I'm kind of lost on the process. The 62366 only says that a plan should be presented and then it starts talking about verification tests.

I've written usability test plans before, but as I understand this, the standard means that I need to write a project plan, which will also include the individual test plan or plans depending of how many tests I make. I've also written project plans before but for something completely different, so I don't know if that'd be the proper thing to do.

I hope you guys can help me here to learn more about what info or sections such plan should have.

Then, as part of such planning, I'm starting to write the Usability Specification, in which I understand I need to put the usability evaluation criteria. My problem here is that even though the standard (and other people in this forum) indicate that one should write Usability Goals, the FDA does not recognize such a thing as "Usability Goals". It doesn't even recognize the example shown in annex G (I'm new to the forum so I can't post links, but if you go to the FDA Recognized Consensus Standards Database and search for 62366 you'll see what I'm talking about).

So I'd really appreciate your advice on how to write my usability evaluation criteria, added to your kind advice on how to write the V&V plan.

Many thanks!!
Oscar
 
Last edited by a moderator:
Elsmar Forum Sponsor

Mark Meer

Trusted Information Resource
#2
Re: Usability Objectives and V&V Plan

I can advise on verification:

Documentation of verification activities needs to stem from requirement specifications. You can maintain a single specification document (one for the whole device/system), or keep several (one for software requirements, one for useability requirements, one for labelling requirements...etc.).... whatever makes most sense to keep things organized.

Regardless, however, the point is that there is a plan to verify each stated requirement. You might, for example, have the following documents:

Requirements Specification
Verification Protocol
Verification Report

As a super-simplified example, the above documents might contain the following line-items for a tennis-ball:

Requirement Specification
| ID | Requirement | Verification Plan |
| Rq2.3 | Shall bounce at least 1ft when dropped from 4ft onto concrete | Test Tst2.3 of Verification Protocol |

Verification Protocol
| ID | Test Description | Acceptance Criteria |
| Tst2.3 | Drop the ball from 4ft onto concrete. Measure the bounce height using (calibrated) measuring device XX. | Height measured is >= 1ft. |

Verification Report
| Date | Test ID | Pass/Fail | Notes |
| YY-MM-DD | Tst2.3 | Pass | Measured 1'2" bounce. |

That's it. The specification provides traceability to a particular verification test, the protocol details the tests and acceptance criteria, and the report confirms that the requirement was verified according to the protocol.

...hope this helps some... :)
 
Last edited:
F

frivolas

#3
Re: Usability Objectives and V&V Plan

...hope this helps some... :)
Hi Mark,

I love it! Makes a lot of sense and it also aligns with the type of documentation we're looking for in the team. Thanks so much!

My QA Engineer will own most of the, let's call them, functional Design Verification tests, so I'll check with her if she has a specific format she will follow. I'll then proceed to create a plan like the one you describe for testing the usability specification.

Do you know if the V&V Plan should also include items like a timeline or any other piece of information?

Thanks again!
Oscar
 

Mark Meer

Trusted Information Resource
#4
Re: Usability Objectives and V&V Plan

Do you know if the V&V Plan should also include items like a timeline or any other piece of information?
Typically no. This type of information would typically be part of a development plan document.

There's a bunch of other information you might include in a protocol, such as specific equipment requirements, training requirements, environment/conditioning requirements... Bottom line, however, is to make a protocol that is a) repeatable/replicatable and b) results in a high degree of confidence in the results. For example:

Test Protocol:
* "Use a calibrated torque meter with a tolerance of xxx."
* "Test shall be performed by personnel trained in yyy."

Test Report:
* "Equipment Used: (reference your equipment record, which will include specifications and calibration schedule/records)"
* "Tested By: Joe Bloggs (reference credentials/training record if necessary/applicable)"

I'll then proceed to create a plan like the one you describe for testing the usability specification.
Just a note of caution to distinguish between verifiable requirements, and requirements that must be validated. For example, the following design requirements must be validated (again, using the simplified tennis-ball example):

Requirements:
"Ball can fit comfortably into the palm of the hand."
"Ball shall aid users in achieving better serves."

Validation Plan:
Ball is given to 10 typical users and they are asked to use the ball for 2 weeks. Upon completion, they are given a survey asking:
"On a scale of 1-5 how comfortable is the ball to hold?"
"On a scale of 1-5 how much did you notice improvement in your serves?"
Acceptance criteria: average of greater than 3.0 on both survey items.

Best of luck!
 
F

frivolas

#5
Re: Usability Objectives and V&V Plan

Oh OK, so... is V&V plan actually talking of the collection of test protocols and results, rather one big project plan that outlines the evaluation strategy?

Sorry, I'm still a little bit confused.
 

Mark Meer

Trusted Information Resource
#6
Re: Usability Objectives and V&V Plan

...is V&V plan actually talking of the collection of test protocols and results, rather one big project plan that outlines the evaluation strategy?
How to best document is up to you to decide. Bottom line is that for each requirement you've specified, ask yourself: "we have specified requirement X, how are we going to demonstrate the requirement is met"? ...and then structure your documentation in such a way that the answer is obvious.

You might, for example, have the following requirement specification documents:
- Hardware Requirements Specification
- Software Requirements Specification
- Packaging and Labeling Requirements Specification
- IEC 60601-1:2005 (clauses x, y, z)
- ...

Then you might have a bunch of "test protocol" documents, which can include international standards where applicable:
- Design validation protocol
- Hardware testing protocol
- Software testing protocol
- ISO 10993-5:2009
- ...

The "plan" is then a matter of mapping each requirement in each specification to a particular test in a particular protocol. Some people do this in one big "Master Plan" document which lists all requirements from all specification documents, and gives a reference to a test for each one...
 
Thread starter Similar threads Forum Replies Date
Q Summative Usability Evaluation Testing: prior or during Clinical Investigation? Human Factors and Ergonomics in Engineering 6
K Usability for custom made medical devices IEC 62366 - Medical Device Usability Engineering 5
P Human Factors / Usability validation in the time of COVID Human Factors and Ergonomics in Engineering 17
N Usability testing required for FDA IDE (investigational device exemption)? Human Factors and Ergonomics in Engineering 8
V Applicability of IEC 62366-1 usability to resorbable bone substitutes IEC 62366 - Medical Device Usability Engineering 9
pashah Looking for Usability File Template acc. IEC 62366-1 and IEC 60601-1-6 IEC 62366 - Medical Device Usability Engineering 4
C Usability IEC 62366-1:2015 and MDR 2017/745 - Risk based approach IEC 62366 - Medical Device Usability Engineering 1
T MDR article 22 - System of 2 CE products and validation / usability? EU Medical Device Regulations 10
M Informational Understanding Costs And Risks For HFE Usability Studies — Part 1: Testing In-House Medical Device and FDA Regulations and Standards News 0
P Usability/human factors engineering requirement (standard) for IVD medical device Other Medical Device Related Standards 2
P Global medical device human factors/usability requirement IEC 62366 - Medical Device Usability Engineering 3
E Usability Process - Looking for a template (IEC 62366) IEC 62366 - Medical Device Usability Engineering 1
S Usability procedure - looking for a template IEC 62366 - Medical Device Usability Engineering 3
K Usability of ME Equipment - IEC-62366-1:2015 IEC 62366 - Medical Device Usability Engineering 3
M Usability Standard - The difference between IEC 60601-1-6 to IEC 62336 Human Factors and Ergonomics in Engineering 1
V 510(k) vs. CE Marking Summative Usability Testing Expectations IEC 62366 - Medical Device Usability Engineering 4
M IEC 62366-1:2015 Usability Verification / Validation Plan example wanted IEC 62366 - Medical Device Usability Engineering 20
Q EU ROHS and GUI Usability Trial Requirements EU Medical Device Regulations 10
M New MHRA guidance on Human Factors - Usability Engineering IEC 62366 - Medical Device Usability Engineering 1
S IEC 62366 - Usability Test Protocol format Imported Legacy Blogs 0
M Does Medical Device Usability Study = Clinical Trial? Other Medical Device and Orthopedic Related Topics 26
L BSI White Paper on Human Factors/Usability Engineering IEC 62366 - Medical Device Usability Engineering 12
R What is the typical industry standard for documenting Usability Risk? Human Factors and Ergonomics in Engineering 16
Z IEC 62366 and Medical Device Usability Validation IEC 62366 - Medical Device Usability Engineering 10
P Number of Participants - Usability Validation IEC 62366 - Medical Device Usability Engineering 3
C Application of IEC 62366 - Usability of Non-Active Medical Devices IEC 62366 - Medical Device Usability Engineering 15
P Usability Plan for Existing Equipment IEC 62366 - Medical Device Usability Engineering 5
Ronen E Can Usability Studies save lives? Other Medical Device and Orthopedic Related Topics 1
L Patient Monitor Viewing Distance Usability Requirements IEC 60601 - Medical Electrical Equipment Safety Standards Series 11
M IEC 62366:2007 - Is FDA Usability Engineering Standard a Requirement in Canada? Canada Medical Device Regulations 5
S IEC 62366 - Embedded Software in separate Usability Engineering File? IEC 62366 - Medical Device Usability Engineering 1
S IEC 62366 for Stand-Alone Medical Device Software and Usability Engineering IEC 62366 - Medical Device Usability Engineering 16
D Training on IEC 62366 (Medical Device Usability Engineering) for CE Mark? IEC 62366 - Medical Device Usability Engineering 5
Y Applying Human Factors and Usability Engineering - FDA Guidance Document 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 1
S New Draft FDA Guidance - Human Factors & Usability Engineering for Med Device Design US Food and Drug Administration (FDA) 0
C Understanding Design Decisions relating to Safety and Usability Human Factors and Ergonomics in Engineering 3
M Employees Awareness, Usability & Effectiveness of the ISO QMS - Survey needed ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 17
D IEC EN 62366 Usability Retroactive Compliance Other Medical Device Related Standards 1
N Usability Formula to Predict Use Error Reduction IEC 62366 - Medical Device Usability Engineering 6
R ISO IEC EN 62366 Usability Engineering Essential Requirements IEC 62366 - Medical Device Usability Engineering 8
M Link between Risk Management, Usability Engineering and R&D in Medical Devices ISO 14971 - Medical Device Risk Management 2
W Usability Engineering ISO 62366:2008 - Medical Devices IEC 62366 - Medical Device Usability Engineering 124
C Usability Engineering File - Looking for an example/template IEC 62366 - Medical Device Usability Engineering 3
S Usability standard IEC 60601-1-6 in the Medical Device Industry IEC 60601 - Medical Electrical Equipment Safety Standards Series 4
J Quality Objectives and resources ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 11
M FSMS Targets and Objectives in SMART planning Food Safety - ISO 22000, HACCP (21 CFR 120) 4
U Examples of Quality Objectives for a Medtech start up ISO 13485:2016 - Medical Device Quality Management Systems 4
V Quality Objectives - ISO 9001 2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 26
I Quality Policy and Objectives examples Elsmar Cove Forum Suggestions, Complaints, Problems and Bug Reports 5
M Specific Values of Quality Objectives Benchmarking 7

Similar threads

Top Bottom