posted 20 December 1998 03:25 AM               

-----snippo-----

From: [email protected] (Brian Charles Kohn)
Subject: RE: Q: Metrics for ISO Performance/Shugart/Gazley/Kohn

> One suggestion is to track your number of internal audit findings,
> grouping by "major", "minor".

I'd suggest that y'all NOT do that. Internal Audit Findings are not normalized, predictable metrics. As a matter of fact, many new companies will find that the number of findings increases for many reasons other than compliance going down:

- Internal auditors gaining a greater understanding of the requirements.
- Multi-part findings being split into separate findings for easier follow-up.
- Many minor findings found as a result of resolving a significant major finding.

I tried to develop some normalizing factors, about 10 years ago, but didn't feel they did the job. (Categorizing findings as either problems with the design of the quality system [S], problems with compliance with the defined system [C], or problems with the maintenance of evidence [E]; each were tracked separately and we were able to see as 'S' went down over time, 'C' had risen and then as 'C' subsided, 'E' went up and then down over time. At least that was the theory.) In the end, it didn't work, and wasn't very helpful.

First, I'd monitor metrics significant to the business. Beyond that, I'd monitored the following metrics to gauge "health of the quality system" when I was management rep:

- % of internal audits completed on time
- % audit findings with adequate evidence*
- value of the preventive actions initiated by management review
- % of CARs "on-plan"

* For each assertion of compliance, notes taken to show that objective evidence was assessed to establish compliance. For each finding, evidence indication of what evidence was observed to determine noncompliance. This was a critical measure to determine whether internal audit was effective, which in turn, made the first metric meaningful.

posted 20 December 1998 03:54 AM               

> - % of internal audits completed on time.
> *If your audit team spends an extra day or two on your internal
> audit but does it correctly and efficiently, how can you consider this
> a metric of your Quality Sytem performance.

This was indeed confusing. What I actually monitored was the % of
internal audits STARTED on time.

Sorry about the confusion.

posted 09 January 1999 08:53 PM               

quote:

---

Interesting to me to see the effort made to determine if there was a correlation between different factors. Could be a factoral design problem worth investigating with some spare time in the future.

---

Got Data? Send it. Spare time? Minimul, but will look into it.

posted 29 January 1999 09:00 AM               

I measure my own internal audit program by itâs efficiency. If I can get a team together, do the job and complile the report with the least impact on the day to day responsibilities of the auditors and the operation, and the least hassle and effort on my own part, then I think itâs going well. The quality of the audit depends on the quality of people at my disposal, and this can vary a lot, but the outcome varies little because the outcome depends on the value of the response.
I will never Îauditâ the company into compliance because this company, and any other company, is capable of straying off course faster than any fixing of non-compliances can pull them back on. The message from my audit is; ãwe are still finding N/cs and, as long as we can, the registrating auditor can. And, even more important, you are not developing and implementing the Documented System as you promised you would! That is the N/C that matters. The response to this should be proactive and preventive.ä

If I would let them, management review would be happy to use the number of Îmajorsâ as a measurement of my audit process. But I wonât let them and I keep pushing the results back onto their plate.

Anyway, if the number of Îmajorsâ rises, does that indicate that the audit
process is improving or that it has deteriorated? If the majors reduce, same question. If it stays the same then are we getting better at finding them but less good at avoiding them, or are we getting less good at finding them but better at avoiding them?

Safer, I think, to go back to Brian Charles Kohnâs advice and ãmonitor metrics significant to the businessä. Of course it is reasonable to expect to find a bunch of majors soon after a new quality system is put in place, and, having found them, they shouldnât happen again - for a few months anyway - but, in the long term, the majors are a function of management responsibility and not an indictor of the effectiveness or the lack of effectiveness of the audit process.

Audit is nothing on itâs own. Finding defects is dead easy. Getting the general trend is also easy. Finding all defects or finding the really important ones is a different matter. Itâs not an improvement process upon which one should bet the company.

posted 04 February 1999 01:19 PM               

Well stated! To steal a line from Crosby "Measurement of nonconformance is not Quality". A downward trend in majors/minors in auditing means little (nothing) if the process is not understood (inputs/outputs). It is management's responsibility to understand if things are as they appear. When inspecting, how often will the inspector reverify a measurement found within specification? How about when a measurement is found out (especially when it's close!)? Management must question the validity of information presented, especially in the case of Internal Quality Audits as it is a major source of information for the Quality System it represents. Folks tend to question less when things are as they are desired or expected, they accept information on face-value alone. Danger looms near if you aren't careful. I feel you are doing the right thing at Management Review. Don't let them sway your judgement and keep up the good work! Sorry I didn't respond sooner, but I did print it out to repond later as I feel you've touched on a significant issue.

posted 05 February 1999 08:39 PM               

As always, Thanks, fellas!