7.6 for a Software Company - Calibrating Unit Testing

[Colleen kyle]

hello,

We have always had section 7.6 excluded from our Manual because we are a software company. Our last 9001:2000 audit we received an Observation saying we should be measuring Unit Testing. Anyone have any suggestions or thoughts?

-Colleen

 

[Claes Gefvenberg]

Hello Colleen, and welcome to the Cove

Er.... 7.6, you say? Yes, that sounds a bit strange when software is on the agenda. And you should be measuring Unit Testing? With what I wonder? No, I don't get it either.

Ok, let's look at it this way:

Tell us how you provide evidence of conformity of product to determined requirements (see 7.2.1).... Are you using any kind of equipment to do that? If not, I cannot for the life of me understand what 7.6 would have to do with it.

I wonder if your auditor was in fact going for 7.1c instead? Can you tell us more? What was the exact text in the observation?

/Claes

 

[Colleen kyle]

Hello Claes and thank you very much for responding.

The Observation read:

One of the exclusions listed is 7.6 (Calibration). It came to my attention during a discussion with management that the organization does use "facilities and techniques", in the conduct of a test verifying conformance of the software product to specified requirements. In addition, during a review of Design & Development it came to my attention that (software code) is subjected to a "Unit test" before acceptance. A review of this requirement may be helpful.


7.2.1 - We use a third party software to help us make sure we are meeting the customer requirements.

The first part of his Observation we were able to write some up on about using the third party software but I cant for the life of me figure out how we can do this for Unit Testing.

-Colleen

 

[howste]

Pardon my ignorance, but what is a Unit Test? At this point (without completely understanding it), my thoughts are that it's more design & development verification & validation (7.3.5 & 7.3.6) than calibration.

 

[edward.gibbs]

Based on the text of the observation and guessing, I think the auditor may be looking at calibration of software test methods.

It is virtually impossible to test complex code completely - so how much do you test, what functions under what stresses, etc.? Developing a test plan and following it is covered under 7.3 (7.3.1, 7.3.5, 7.3.6). But the measurement instruments used (benchmarks, scenarios, statistical techniques, etc.) may require calibration to ensure that they are accurately measuring what they are designed to measure.

I agree that it's a stretch, but I can see where he is going. Suppose for example that a scenario is developed to test a particular function in the code. How do you know that what the scenario is measuring is valid? Suppose the engineer who develops the code has a hard time making the scenario work, so he modifies the scenario to make his code pass. Is that an "Adjustment that would invalidate the measurement result" under 7.6.d? I tend to think so.

YMMV,

Ed Gibbs

 

[Graeme]

Colleen,

Like others, I am somewhat confused by the wording of this observation ...

Clearly, the auditor cannot be takling about "measuring equipment" since you are using a software tool instead of a voltmeter or a micrometer. However, what about the last paragraph of clause 7.6 (ISO 9001:2000) where it says "... the ability of computer software to satisfy the intended application shall be confirmed ... prior to use ..." It also refers to ISO 10012:2003 (that is the current version of what it actually says!) for guidance. 10012 has some requirements at 6.2.2: "Software ... shall be documented, identified and controlled ... tested and/or verified prior to initial use, approved and archived. Testing shall be to the extent necessary to ensure valid measurement results."

This seems to be saying that the commercial testing software that you use for testing your developed code must be validated somehow. Also, it must be placed under version control and re-tested any time it is patched or upgraded. As a guess, I think you would have to develop a test module that is verified by some other means, and contains examples that the test software should catch as good, questionable or bad (or whatever it does.) Then this test module becomes your "calibration standard" and is used to verify the testing software.

In my opinion, the intent should be that -- with respect to the commercial testing software you are using -- you can verify that the version in use operates as intended with samples of "known good" and "known bad" code from your development process. Both the test software and the code samples have to be documented and controlled.

I may have more suggestions tomorrow ... my copy of ISO 9000-3 (Guide to applying 9001:1994 to software development) is gathering dust at home since I don't do that in this job. By the way, that document is near the end of a revision, and may be released by the end of the year as ISO/IEC 90003.

 

[Claes Gefvenberg]

First off, I'll admit that I'm just as confused as everyone else...

Like others, I am somewhat confused by the wording of this observation ...

However, what about the last paragraph of clause 7.6 (ISO 9001:2000) where it says "... the ability of computer software to satisfy the intended application shall be confirmed ... prior to use ..."

Yes. That could fit in with: ...the organization does use "facilities and techniques", in the conduct of a test verifying conformance of the software product to specified requirements...

The second part: ...In addition, during a review of Design & Development it came to my attention that (software code) is subjected to a "Unit test" before acceptance. A review of this requirement may be helpful. may fit better under 7.3.5 or 7.3.6.

All things considered, I think the auditor could (should) have been a much more distinct when writing the observation. This kind of guessing game is in aid of nobody. Still, if we are interpreting this the way he meant it, he may have a point.

/Claes

 

[Graeme]

We have always had section 7.6 excluded from our Manual because we are a software company. Our last 9001:2000 audit we received an Observation saying we should be measuring Unit Testing. Anyone have any suggestions or thoughts?

Colleen,

Here is some more on your question, to add to yesterday’s comments.



Whatever tool is used, a major way to check its effectiveness in testing the product requirements is a test traceability matrix. This is used to track product/customer requirements to specific test cases. If possible, the test cases should simulate actual use as closely as possible, and use realistic data. Even if testing is done at the software unit or module level, overall system tests should be performed on the baseline product.



A similar matrix can be used to document your requirements for the testing tool, and the test cases used to validate pass/fail/indeterminate performance of the tool.



Measurements are used to assess the quality of product; and of the production process. However, the way most common metrics are used focuses only on the process and not on the product.

• Test coverage – important to show completeness of tests; percentage of code tested by tool.
• Number of defects – also type, severity, time to fix, etc. (This is a rework measure.)
• Rework due to bad fixes
• Rework due to errors in the original specification or requirements
• Rework due to requirements changes (in-house, or by customer)
• Documentation errors
• Customer satisfaction

Purchased product – including tools – must be validated to ensure they conform to specified requirements. This can be interpreted to fall under the calibration requirement of 7.6, especially the last paragraph. The performance of the tool must be validated on code and test cases from your development process. (Is there such a thing as a “calibration standard” software module or test case? I don’t think so.) The validation should have “known bad” as well as “known good” items. The validated tool has to be placed under version control, and re-validated any time an upgrade or patch is applied.



References:

• ISO 9000-3:1997, Quality management and quality assurance standards – Part 3: Guidelines for the application of ISO 9001:1994 to the development, supply, installation and maintenance of computer software.
• ISO 9000-3: A tool for software product and process improvement by R. Kehoe and A. Jarvis (Springer, 1996.)

Note: the standard referenced above is being re-written, and is currently in final draft status. It will become ISO/IEC 90003:200X Software and system engineering – Guidelines for the application of ISO 9001:2000 to computer software. (The X is because it is unknown when ISO will actually publish it. It could be this year, or maybe next year.)