How to write requirements for Medical App Validation and Verification

[elmagnifico]

Hi all,

I am working with a documentation of a medical app I wonder what is a better way to write requirements and how to put them in sentences properly for verification and validation. Here is a small part of the requirements;

a) Launching the mobile application
- Launch icon should be visible
- Launch icon should bring the splash screen on
- Splash screen should not stay more than 1 second

b) Logging in to the mobile application
- USER have a master pin code
- Pin code should consist four numerical character
- USER should see how many characters being entered
- The numbers entered are replaced by circles and remains hidden
- If the pin code is right, USER should reach to the main menu
- If the pin code is wrong, USER will see an option to receive a new pin code by email

First of all, I wonder if there is a better way to write requirements. For example, should it be like this;

1) Launching the mobile application

1.1 Launch icon should be visible
1.2 Launch icon should bring the splash screen on
1.3 Splash screen should not stay more than 1 second

After listing them correctly, how should I write the validation plan. Here is an example that I plan to continue with

Requirement
Req1.1 Launch icon should be visible

Validation Plan
The application is used by 5 typical users and they are asked to launch the application on the main window. Upon completion, they are given a survey asking: ?On a scale of 1-5, how visible is the launch icon of the app?

Acceptance Criteria
Average of greater than 3.0

Notes

But i wonder now should do it that way and validate all sub-levels or should I create a more general validation plan and skip the details of it such as;

80% of users should be able to launch the app within 10 seconds on their first try

And for the verification, I feel like some of the requirements don't look verifiable or I don't know how to put them in words in a good way. For example, how can I verify the requirement that saying "launch icon should be visible"? It looks validatable but not verifiable. Should I create my requirement in a way that they should be both verifiable and validatable?

I hope I managed to explain my problem. Please ask if it is not clear.

Cheers,

 

[maaquilino]

Re: How to write requirements for validation and verification

If you're under FDA standards you must have traceability; the traceability matrix is the first document every FDA auditor I've dealt with has asked for. The document traces requirements to design to testing. It's also a great document to help you ensure all your requirements were tested.

You should always have each part of the requirement numbered for traceability, so the second example is correct:
1) Launching the mobile application

1.1 Launch icon should be visible
1.2 Launch icon should bring the splash screen on
1.3 Splash screen should not stay more than 1 second

If you’re having the software built in-house or by an outside company specifically for you, you first should have a high level requirements document from the users specifying what they want. You would then write a detailed requirements document that fleshes out those high level requirements so that a developer can design and build your system from this document. You want your requirements to be as detailed as necessary so that you get what you’re asking for.

That also means your testing needs to be as detailed as necessary so you ensure your requirements have been met. You should have a test group that’s independent of the developers doing the testing to the detailed requirements. Once that testing passes, then you should have a user group made up of your actual users test the requirements to the higher level requirements. Launching the mobile app would be the high level requirements; the three requirements listed below it are the detailed requirements.

Verification and Validation generally consists of several phases: unit testing done by the developer of the code; requirements testing by Software testers to the detailed requirements; and user testing to the high level requirements. Your Validation Plan should outline what verification and validation will be done, and on what software modules/components including the version number(s) of the software. It should cover unit testing, requirements testing, and user testing, and detail what will be done, how it will be done, and who will be doing it. I have separate protocols for the requirements and user testing with the relevant test scripts attached, but you can also include the detailed testing information it in your validation plan. I don’t do it that way because the requirements and user testing isn’t generally known or documented yet by the time I need to have a validation plan done and approved. There are numerous templates on the Cove and on the internet for validation plans that will help you see what needs to be included in them.

Your test cases or test scripts should list what the requirement is, the steps to test it, and the expected result. I use a template with columns (for some reason I can’t post attachments on here or I’d attach the template for you). For example, for your first requirement the I would have the following:
(X.X denotes the test script number that traces to the requirement number; the first line is the heading of the column, the second line is what would go under that column for the first requirement)

Requirement
1.1 Launch icon should be visible

Steps
X.X Verify launch icon is visible

Expected Result
Launch icon is visible

Pass/Fail
(I usually put a check box in this column with the words ‘Actual results as expected’ that the tester can check, rather than having them rewrite the actual results when they match the expected ones.

This test would be validated using visual verification with a screen print showing the launch icon visible as your objective evidence of the test passing. I make screen prints of every test result, whether it’s a visual verification or not, as that’s my objective evidence the test passed instead of just relying on the testers Pass or Fail.

I don’t know what you mean by your Acceptance Criteria, as testing passes based on the results of the tests, not the testers’ opinion or scale rating. A test either passes or fails. A good tester will notice things during requirements testing and should bring it to the attention of whoever is overseeing the testing. But this acceptance criteria would be good for feedback during the user testing of the system.

 

[yodon]

Nice reply from maaquilino. I'll try to briefly add to that.

From the way you wrote the post, one of the challenges you may be having is how you state the requirements. Using "should" often implies 'wiggle room.' If you re-state those as "shall" does it clarify?

Validation is from the user's perspective: does it meet their needs. So your approach using 5 users and averaging the score is probably going to be acceptable. (Although you may want to say that any individual score at 2 or below requires at least some additional evaluation).

Verification is from the perspective of what the system is required to do. Show proof that you have met your functional requirements. (Another confounding factor may be that you're focusing on the UI more than the functionality?) You can still do some level of verification of the UI but is it really proving the system functionality?

If you look at a typical 'V' model, you'll see user requirements (what do the users need) at the top which flow down to functional requirements (what engineering has to do to 'answer' those user needs). Validation shows that the user requirements are met and verification shows the functional requirements are met.

 

[elmagnifico]

Thanks for the replies. They were very helpful!

I will try to summarize what I understood from your explanations

There are 3 main tasks for validation&verification;

- unit testing + requirement testing (for verification)
- user testing (for validation)

Now, my questions are;
1) When I create the table with columns, can I use the same table for both validation and verification steps? Or should I create different tables in different files?
2) When I create this validation plan which includes all the steps and definitions for validation and verification tasks, only thing that i need to do for the validation and verification steps is to fill the empty columns on the tables, right? Will I need other documents to create?


maaquilino, Could you send that verification &validation template to me directly as a message? or you can also host it somewhere on the net (dropbox public folder, etc.) and send the link here, so others can see it as well.

And can i find a complete usability engineering file for a medical device (mobile app) which is already on the market or for something is not being produced anymore? Are they top secret files?

@yodon, I think I am not sure about either "shall" or "should" fits to my requirements. I think it in this way;

if the user has a freedom i should use "shall" and
if the system needs a function i put there "should".

Is that wrong? English is not my mother tongue, so i would like to have your opinion on this.

Cheers,

 

[maaquilino]

Thanks, Yodon; nice reply from you also; great catch on the ‘shall’ versus ‘should’!

Elmagnifico, I use one table that has separate columns for unit testing, OQ (independent testing) and PQ (user testing). I sent you a private message with my email address as for some reason I can’t put attachments on here, and today is my last day on my current contract so I’m not sure when I’ll be on the Cove again in the near future. I can send you the test script form and the V&V Plan and V&V Final Report templates I use.

There are several documents you should create. The verification and validation plan I use lays out the tasks that need to be done for verification and validation, how it will be done, and who is responsible for which tasks. I then have protocols and final reports for both OQ and PQ testing. Each protocol defines the detailed testing that will take place for that phase of the testing. And each protocol has test scripts written and attached to the protocol; these test scripts will be executed and filled out by the tester as to whether or not the test passed or failed. The final reports detail what happened during the testing, how many discrepancies were found and their resolution, and regression testing, etc.

When you use the term ‘shall’ that means something must be done; ‘should’ means they should do it but they don’t have to. When writing requirements, I always write them as ‘shall’ because those requirements are not negotiable; they are what is being asked for and have to be done. Same thing when writing scripts, though my scripts are written as steps so I don’t use the term ‘shall’:
User logs on,
User enters valid user ID,
User enters valid password
System accepts logon and displays X screen.

 

[ET_Marley]

I think an important point here when writing the usability requirements is to consider the consequences when a requirement is not met. I am a bit concerned that some of these requirements are not safety related (e.g. "a splash screen should not stay on for more than 1 second" - what is the importance of "1 second" in terms of safety?). I think you have to be careful to focus on safety-related usability issues for compliance (the FDA are not interested in general usability issues, only safety related ones).

For example, what is the purpose of the splash screen in terms of safety and what happens if the user forgets their pin code? If the user can't access the app then is there a risk that some harm comes to the user??

In a hypothetical case where one manufacturer develops 2 medical apps, for example, one to measure blood pressure and the other to measure heart rate. Because they are developed by the same manufacturer they may look the same and have similar launch icons.

In this case there is a risk that the two applications may be confused with each other resulting in a user looking at a heart rate measurement rather than a blood pressure measurement. This may have safety implications for the user and hence the need to have a splash screen.

If this is the case, the time it is displayed is irrelevant. What is important is that the user can confirm the purpose of the application and confirm it is the one that is appropriate for their medical purpose and not confuse the intended app with another one. The success criteria should be measured accordingly.

Maybe the requirements should read something like:

1.1: The user shall be able to open the application and determine it as the correct one


Regarding scales of acceptance (and the the use of shall vs should for that matter), I have heard discussions about the FDA's adverse views on setting usability goals that state 80% success rate which then implies an acceptable 20% of use errors to occur. My understanding is that the FDA is more concerned with the devices not being used in a way that may cause harm. If any safety related use-errors do occur during testing then they need to be understood, explained and mitigated for in the design as far as is possible. As a result a test either passes or fails, as maaquilino said, and not passes 80% of the time.

Rather than setting usability objectives (e.g. 80% success rate), it is more useful to understand the potential hazardous situations of the device, implement strategies to reduce/eliminate their occurrence/severity. These can be phrased as pass or fail usability requirements that can then be tested to ensure that the proposed design is sufficient to prevent use-errors from occurring or to enable them to be safely recovered from by monitoring them in a tracability matrix.

Any use-errors should be examined and documented within the test reports.

 

[elmagnifico]

Hi ET_Marley,

Thanks for your input. I realized that I need to ask better questions to get the answers I need.

What does our app do?
? Connects via Bluetooth to the medical device(patients use it themselves at home)
? Gets the treatment data from the medical device
? Enables user to compare-comment-rate treatments
? Sends to healthcare professional information about treatments details and your subjective evaluations, so they can see the patient?s progress and maybe change the treatment

My questions

1) This app will compliment a medical device but it has no role with the treatment that the medical device provides. Do you think there should be one big usability engineering file includes the requirements related with the app inside or the medical device and the mobile app should have different files and applications for FDA? By the way while our client develops the medical device, we develop the mobile app as a consultant company.

2) How should I write the requirements? Should I create different sections such as functional requirements, non-functional requirements, safety related requirements, etc??

Or should I use our apps workflow and write everything down under each title such as;

a) Logging in to the app
1. USER shall enter four numerical characters to log in
2. USER should see how many characters being entered
3. The numbers entered are replaced by circles and remains hidden
4. If the pin code is right, USER should reach to the main menu
5. If the pin code is wrong, USER will see an option to receive a new pin code by email

b) Rating treatment
1. USER should slide his/her finger on the stars to rate irrigation
2. Selected stars should be filled-in
3. Unselected stars should remain outlined
4. Stars represent the level of satisfaction;
- one star / terrible,
- two stars / not good
- three stars is ok
- four stars /very good
- five stars/perfect
5. Selected stars will change their appearance
6. USER shall go back to irrigation details by using back button on the top left corner
7. USER shall navigate between rating questions by using ?prev? and ?next? buttons
8. USER shall skip rating questions


If I do it this way where should I mention the design requirements on the login page such as colors, font sizes, etc.? (Please also evaluate the quality of the requirements above)



3) Are usability goals requirements also?
On the net sometimes I see that requirements are written this way;

80% of the users should able to login to the app in 15 seconds.

However, 62366 mentions those as usability goals. What do you think? Are they also requirements to validate and verify? If they are, it is easy to validate I think. Find 5 users to test and see if they are able to login but how can you verify this then? With which method?

Also, where do this %80 or 15 seconds come from? How do you set this kind of numbers as a limit?


3) Will I verify and validate the same requirements with the same methods?
I thought that we can use cognitive walkthrough method for verification and user tests for validation. However I have seen some examples that mention methods such as IQ, OQ and PQ which I don?t get. What are they for? Is there any standards to follow?


4)Do we need to have a worst case use scenario for validation test?

Our app is not used during the treatment and it is just for keeping track of the treatments. So, how should we set the worst case use scenario? What could be the conditions related? And where will be use this scenario? Will the user test for validation be based on this scenario? Should we set the test environment and equipments according to this?

5) Can I find an example or template documentation for mobile apps related to 62366 standards?

Thanks already everyone who read this and spend some time to help me.

 

[Chromehounds]

yes, very detailed reply.