7.6 for a Software Company - Calibrating Unit Testing

C

Colleen kyle

#1
hello,

We have always had section 7.6 excluded from our Manual because we are a software company. Our last 9001:2000 audit we received an Observation saying we should be measuring Unit Testing. Anyone have any suggestions or thoughts?

-Colleen :confused:
 
Elsmar Forum Sponsor
#2
Hello Colleen, and welcome to the Cove :bigwave:

Er.... 7.6, you say? Yes, that sounds a bit strange when software is on the agenda. And you should be measuring Unit Testing? With what I wonder? No, I don't get it either.

Ok, let's look at it this way:

Tell us how you provide evidence of conformity of product to determined requirements (see 7.2.1).... Are you using any kind of equipment to do that? If not, I cannot for the life of me understand what 7.6 would have to do with it.

I wonder if your auditor was in fact going for 7.1c instead? Can you tell us more? What was the exact text in the observation?

/Claes
 
C

Colleen kyle

#3
Hello Claes and thank you very much for responding. :D

The Observation read:

One of the exclusions listed is 7.6 (Calibration). It came to my attention during a discussion with management that the organization does use "facilities and techniques", in the conduct of a test verifying conformance of the software product to specified requirements. In addition, during a review of Design & Development it came to my attention that (software code) is subjected to a "Unit test" before acceptance. A review of this requirement may be helpful.


7.2.1 - We use a third party software to help us make sure we are meeting the customer requirements.

The first part of his Observation we were able to write some up on about using the third party software but I cant for the life of me figure out how we can do this for Unit Testing.

-Colleen :)
 

howste

Thaumaturge
Super Moderator
#4
Pardon my ignorance, but what is a Unit Test? At this point (without completely understanding it), my thoughts are that it's more design & development verification & validation (7.3.5 & 7.3.6) than calibration.
 
E

edward.gibbs

#5
Based on the text of the observation and guessing, I think the auditor may be looking at calibration of software test methods.

It is virtually impossible to test complex code completely - so how much do you test, what functions under what stresses, etc.? Developing a test plan and following it is covered under 7.3 (7.3.1, 7.3.5, 7.3.6). But the measurement instruments used (benchmarks, scenarios, statistical techniques, etc.) may require calibration to ensure that they are accurately measuring what they are designed to measure.

I agree that it's a stretch, but I can see where he is going. Suppose for example that a scenario is developed to test a particular function in the code. How do you know that what the scenario is measuring is valid? Suppose the engineer who develops the code has a hard time making the scenario work, so he modifies the scenario to make his code pass. Is that an "Adjustment that would invalidate the measurement result" under 7.6.d? I tend to think so.

YMMV,

Ed Gibbs
 
G

Graeme

#6
Colleen,

Like others, I am somewhat confused by the wording of this observation ...

Clearly, the auditor cannot be takling about "measuring equipment" since you are using a software tool instead of a voltmeter or a micrometer. However, what about the last paragraph of clause 7.6 (ISO 9001:2000) where it says "... the ability of computer software to satisfy the intended application shall be confirmed ... prior to use ..." It also refers to ISO 10012:2003 (that is the current version of what it actually says!) for guidance. 10012 has some requirements at 6.2.2: "Software ... shall be documented, identified and controlled ... tested and/or verified prior to initial use, approved and archived. Testing shall be to the extent necessary to ensure valid measurement results."

This seems to be saying that the commercial testing software that you use for testing your developed code must be validated somehow. Also, it must be placed under version control and re-tested any time it is patched or upgraded. As a guess, I think you would have to develop a test module that is verified by some other means, and contains examples that the test software should catch as good, questionable or bad (or whatever it does.) Then this test module becomes your "calibration standard" and is used to verify the testing software.

In my opinion, the intent should be that -- with respect to the commercial testing software you are using -- you can verify that the version in use operates as intended with samples of "known good" and "known bad" code from your development process. Both the test software and the code samples have to be documented and controlled.

I may have more suggestions tomorrow ... my copy of ISO 9000-3 (Guide to applying 9001:1994 to software development) is gathering dust at home since I don't do that in this job. By the way, that document is near the end of a revision, and may be released by the end of the year as ISO/IEC 90003.
 
#7
First off, I'll admit that I'm just as confused as everyone else...

Graeme said:
Like others, I am somewhat confused by the wording of this observation ...

However, what about the last paragraph of clause 7.6 (ISO 9001:2000) where it says "... the ability of computer software to satisfy the intended application shall be confirmed ... prior to use ..."
Yes. That could fit in with: ...the organization does use "facilities and techniques", in the conduct of a test verifying conformance of the software product to specified requirements...

The second part: ...In addition, during a review of Design & Development it came to my attention that (software code) is subjected to a "Unit test" before acceptance. A review of this requirement may be helpful. may fit better under 7.3.5 or 7.3.6.

All things considered, I think the auditor could (should) have been a much more distinct when writing the observation. This kind of guessing game is in aid of nobody. Still, if we are interpreting this the way he meant it, he may have a point.

/Claes
 
G

Graeme

#8
Colleen kyle said:
We have always had section 7.6 excluded from our Manual because we are a software company. Our last 9001:2000 audit we received an Observation saying we should be measuring Unit Testing. Anyone have any suggestions or thoughts?
Colleen,

Here is some more on your question, to add to yesterday’s comments.



Whatever tool is used, a major way to check its effectiveness in testing the product requirements is a test traceability matrix. This is used to track product/customer requirements to specific test cases. If possible, the test cases should simulate actual use as closely as possible, and use realistic data. Even if testing is done at the software unit or module level, overall system tests should be performed on the baseline product.



A similar matrix can be used to document your requirements for the testing tool, and the test cases used to validate pass/fail/indeterminate performance of the tool.



Measurements are used to assess the quality of product; and of the production process. However, the way most common metrics are used focuses only on the process and not on the product.

  • Test coverage – important to show completeness of tests; percentage of code tested by tool.
  • Number of defects – also type, severity, time to fix, etc. (This is a rework measure.)
  • Rework due to bad fixes
  • Rework due to errors in the original specification or requirements
  • Rework due to requirements changes (in-house, or by customer)
  • Documentation errors
  • Customer satisfaction

Purchased product – including tools – must be validated to ensure they conform to specified requirements. This can be interpreted to fall under the calibration requirement of 7.6, especially the last paragraph. The performance of the tool must be validated on code and test cases from your development process. (Is there such a thing as a “calibration standard” software module or test case? I don’t think so.) The validation should have “known bad” as well as “known good” items. The validated tool has to be placed under version control, and re-validated any time an upgrade or patch is applied.



References:

  • ISO 9000-3:1997, Quality management and quality assurance standards – Part 3: Guidelines for the application of ISO 9001:1994 to the development, supply, installation and maintenance of computer software.
  • ISO 9000-3: A tool for software product and process improvement by R. Kehoe and A. Jarvis (Springer, 1996.)
Note: the standard referenced above is being re-written, and is currently in final draft status. It will become ISO/IEC 90003:200X Software and system engineering – Guidelines for the application of ISO 9001:2000 to computer software. (The X is because it is unknown when ISO will actually publish it. It could be this year, or maybe next year.)
 
Thread starter Similar threads Forum Replies Date
U Document Approval - Software company ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
E In need of a new TGA sponsor - Small software company Other Medical Device Regulations World-Wide 4
M Software Development Company - Who would own the whole process and the certification afterwards? ISO 14001:2015 Specific Discussions 1
J Choosing QMS Software for Aerospace Company Quality Assurance and Compliance Software Tools and Solutions 5
R In a software development company: Is every bug reported by the customer a complaint? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 27
O ISO 13485 for software company - Selecting suppliers ISO 13485:2016 - Medical Device Quality Management Systems 3
P QMS software recommendations for a small ISO 13485 company Quality Manager and Management Related Issues 5
T ISO 13485 for software outsourcing company ISO 13485:2016 - Medical Device Quality Management Systems 7
Q QMS Software for Startup Medical Device Company Other Medical Device and Orthopedic Related Topics 7
D Looking for Calibration Software for small company General Measurement Device and Calibration Topics 20
Q Approving Software Suppliers in an AS9100 Company AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 5
D QMS Software Recommendations for a Small Company ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 12
T Purchasing SPC Software for Automotive Company Statistical Analysis Tools, Techniques and SPC 9
J ISO13485 Clause exclusions for a Medical Software Company ISO 13485:2016 - Medical Device Quality Management Systems 9
P Why does a SQA (Software Quality Assurance) standard be implemented in our company Software Quality Assurance 4
B ISO 9001 for a small Micro Software Company Software Quality Assurance 18
K ITIL or ITSM is applicable for Software product company Other ISO and International Standards and European Regulations 1
M Cheap/Free CAPA Software for small Pharmaceutical company Quality Assurance and Compliance Software Tools and Solutions 8
R Defining Interaction of Processes in a Software Company Software Quality Assurance 3
D FDA Compliant QMS Software Recommendation for Small Company Quality Assurance and Compliance Software Tools and Solutions 3
D FDA Establishment Registration for Software Company US Food and Drug Administration (FDA) 8
sagai ISO 9001 Clause 7.5 - Applicable Activities for a Software Company ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 10
S Software Company wants CMMI Software Quality Assurance 4
S SW (Software) - Customer vs. Company Installation 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 2
D Medical Devices Company QMS Software - Recommendation for an off the shelf software ISO 13485:2016 - Medical Device Quality Management Systems 13
J What SPC (Statistical Process Control) software is your company using? Statistical Analysis Tools, Techniques and SPC 15
E 'OnTime' for Management in small Medical Software company? Quality Assurance and Compliance Software Tools and Solutions 3
M CAPA Process for a small Software company - Ideas and recommendations wanted Nonconformance and Corrective Action 8
J SPC software Evaluation and Recommendations - Small Company Quality Assurance and Compliance Software Tools and Solutions 9
T Procedure for "Controlling Nonconformity" for a software company Document Control Systems, Procedures, Forms and Templates 2
M 3 types of Outsourcing - Software Company ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 4
A Design Review OR Design Change - Software development company ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
D Language Translation Requirements - Medical software company ISO 13485:2016 - Medical Device Quality Management Systems 3
G MRM (Management Review) Inputs Sheet Format for a Software Development Company Management Review Meetings and related Processes 1
M ISO 9001 and AS 9100 Registrations - AS9100 applicability to a software company AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 3
V What qualification is needed for working as QMS Engineer in a software company? Training - Internal, External, Online and Distance Learning 6
S ISO certification guidelines for Software Company ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 6
N Audit questions for software/hardware selling company Software Quality Assurance 17
N Internal Auditing - IT software hardware selling company Internal Auditing 5
W Software development company for Medical Device Diagnostic Equipment ISO 13485:2016 - Medical Device Quality Management Systems 5
A Internal Audits in a Software Company General Auditing Discussions 5
S Software Company Quality Manual For ISO 9001:2000 needed Quality Management System (QMS) Manuals 4
C Where can a small company find a automated ISO software package thats cheap Quality Assurance and Compliance Software Tools and Solutions 13
M Please suggest Registrar in Emeryville, CA - Software development company Registrars and Notified Bodies 5
S AS9100/AS9006 process documentation - Aerospace software consulting company AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 5
D AS9100 site specific certification question - Software company AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 2
K Software that will enable our company to make Quality Inspection Sheets Quality Assurance and Compliance Software Tools and Solutions 16
M Software requirement specification study vs. Predeployment study in software company Software Quality Assurance 2
M Seeking: Timesheet application for software company Quality Assurance and Compliance Software Tools and Solutions 2
M Control of nonconforming product - Software Company - ISO 9001, Clause 8.3 Nonconformance and Corrective Action 9

Similar threads

Top Bottom