Location of Verification Test Procedures


Oscar Wang

hi, guys. here is a discussion in our company about the location of verification procedure. Current situation is as below:
1. We have a system requirement specification (DHF) as Design Input
2. first version Test cases are compiled in verification procedures (DHF)
3. during the product verification, there are a lot of software defect.
4. defects are recorded and handled in defect management system (Bugzilla).
5. Some of these defects are not mapped to system requirement specification. As you known, it is impossible to enumerate all detail requirements in design input, we just choose critical ones. There are a lot of software bugs which don't map to any design input requirements.
6. In order to verify these bugs, some test case should be revised and others shall be newly created.
Here is the argument point: shall we add every newly created test case into our verification procedure? If the answer is no, what kind of test case and their test record can be written on the defect management system (Bugzilla). how to define these criteria?


Re: help needed for location of verification test procedures

It's not only possible to define all requirements in detail, it's vital. (If the requirements are not specific, how do the engineers know what to make?)

Sometimes requirements specification is an iterative process of trial and error, with requirements specified in increasing detail over time as they are elicited, for example in discussions with the client , as a result of requirements analysis and draft architectural design, or as a result of lab experiments.

Poor requirements management is one common reason for software failure. In telecom, requirements specifications can include thousands of specific requirements.

And yes, all the test cases should accumulate into one or several test specs.

This isn't specific to the medical sector, but good software engineering practice as described, for example, in CMMI, Agile and TickIT.

Hope this helps

Oscar Wang

Re: help needed for location of verification test procedures

Thanks for pldey42's comment. I didn't say the requirement is not specific and how to define specific may be variance in different situation. for example, is the following requirement specific enough?
$ The system shall have a function that allow to create a new patient.
$ The patient table shall be able to move vertically from XX cm to XX cm.
In medical device development, we have requirements in system level and requirements in subsystem/component level. I am confusing about which level should be more specific. who can help me clarify this issue? when FDA audit the design control, which level do they care about?


Super Moderator
Re: help needed for location of verification test procedures

Indeed, it would be impossible to specify every behavior required of software. So you do what you can... and it sounds like you're doing that.

Test failures, though are indicators of something gone wrong so you should take an introspective look and see if improvements can be made. This could be as wide-ranging as (yes) improving your requirements development process, improving design reviews, adding scenario reviews, improving (or implementing) coding standards, instituting peer reviews at the code level, peer programming, etc.

But to address your question, I generally take the tack of separating design change verification from system verification. At the design change verification level, you ensure that the changes really do fix the bug. You can do this through a variety of means; e.g., code inspections, unit testing, and up through (focused) system level tests. All this can be captured (with proper discipline) as verification evidence (possibly even as attachments in Bugzilla).

You can review the formal verification (confirmation that requirements are met) test procedures to see if they can be updated to test either specifically for the condition or more generically. It's not a bad idea, in my mind, to do both - have the formal test specifically exercise the condition that revealed the bug in the first place AND test generically 'around' the condition. As all studies indicate, changing code increases the probability of other failures. So if you have the specific condition tested, a change in the future that re-introduces the original error would be caught. It's a judgment call, though. A good test team should be able to strike the right balance.

Oscar Wang

Re: help needed for location of verification test procedures

Thanks yodon for your good suggestion. It is also the preferred solution in our internal discussion. But i still have a big concern on the bugzillar issue. because bugzillar is not customerized tool, it is difficult to ensure that the test case and its expected result is properly reviewed and confirmed before the verification starts and often test results are not eleborated properly. Some evidence comments are just stating that test passes. Currently there are a lot of procedure and tool flaws in our team.
Top Bottom