Internal Auditing Metrics - Findings are not normalized, predictable metrics

[Marc]

Comments anyone?

-----snippo-----

From: (Brian Charles Kohn)
Subject: RE: Q: Metrics for ISO Performance/Shugart/Gazley/Kohn

> One suggestion is to track your number of internal audit findings,
> grouping by "major", "minor".

I'd suggest that y'all NOT do that. Internal Audit Findings are not normalized, predictable metrics. As a matter of fact, many new companies will find that the number of findings increases for many reasons other than compliance going down:

- Internal auditors gaining a greater understanding of the requirements.
- Multi-part findings being split into separate findings for easier follow-up.
- Many minor findings found as a result of resolving a significant major finding.

I tried to develop some normalizing factors, about 10 years ago, but didn't feel they did the job. (Categorizing findings as either problems with the design of the quality system [S], problems with compliance with the defined system [C], or problems with the maintenance of evidence [E]; each were tracked separately and we were able to see as 'S' went down over time, 'C' had risen and then as 'C' subsided, 'E' went up and then down over time. At least that was the theory.) In the end, it didn't work, and wasn't very helpful.

First, I'd monitor metrics significant to the business. Beyond that, I'd monitored the following metrics to gauge "health of the quality system" when I was management rep:

- % of internal audits completed on time
- % audit findings with adequate evidence*
- value of the preventive actions initiated by management review
- % of CARs "on-plan"

* For each assertion of compliance, notes taken to show that objective evidence was assessed to establish compliance. For each finding, evidence indication of what evidence was observed to determine noncompliance. This was a critical measure to determine whether internal audit was effective, which in turn, made the first metric meaningful.

Brian Charles Kohn
http://www.synap.com

[Marc]

From: "John Gazley"
Subject: Re: Metrics for ISO performance/Shugart/Gazley/Kohn/Gazley

Brian-

First I would suggest that y'all think before you tell someone not to use any methods presented on this forum unless you are absolutely sure they have not been found to be valid at other companies.

>As a matter of fact many new companies will find that number of
>findings increase for many reasons other than compliance going
>down:
> - Internal auditors gaining a greater understanding of the
> requirements

"This should remain or possibly increase while your audit team gains experience, your system bugs are discovered, and overall awareness of the requirements grows."

>findings increase for many reasons other than compliance going down

We are all quality professionals, I doubt there are few among us who would not investigate any trends in this metric before simply labeling it as caused by non-compliance increasing/decreasing. Like any other process monitoring, investigation of causes is key.

I also stated that after the initial learning curve there should be a sharp decline in majors. I stand by this statement, and have witnessed it personnally at three other facilities I have helped gain QS9000 or ISO9000 certification. All three of which currently use this as an accepted metric. I have read countless articles and books written by experts in the field which reference this as a viable metric.

If a decrease in your "major" findings is not an item used to gage the functioning of your quality system, your missing the boat.

Please explain the viability as a metric:

- % of internal audits completed on time.
*If your audit team spends an extra day or two on your internal audit but does it correctly and efficiently, how can you consider this a metric of your Quality Sytem performance.

John

[Marc]

From: (Brian Charles Kohn)
Subject: Re: Metrics for ISO performance/.../Kohn/Gazley/Kohn

> - % of internal audits completed on time.
> *If your audit team spends an extra day or two on your internal
> audit but does it correctly and efficiently, how can you consider this
> a metric of your Quality Sytem performance.

This was indeed confusing. What I actually monitored was the % of
internal audits STARTED on time.

Sorry about the confusion.

Brian Charles Kohn

[Kevin Mader]

Marc,

Good postings. I use something similar to the method above. Interesting to me to see the effort made to determine if there was a correlation between different factors. Could be a factoral design problem worth investigating with some spare time in the future.

[Marc]

Spare time? You have spare time? Can you lend me some?

[Don Winton]

Interesting to me to see the effort made to determine if there was a correlation between different factors. Could be a factoral design problem worth investigating with some spare time in the future.

Got Data? Send it. Spare time? Minimul, but will look into it.

Regards,
Don

Internal audit metrics.

I measure my own internal audit program by it’s efficiency. If I can get a team together, do the job and complile the report with the least impact on the day to day responsibilities of the auditors and the operation, and the least hassle and effort on my own part, then I think it’s going well. The quality of the audit depends on the quality of people at my disposal, and this can vary a lot, but the outcome varies little because the outcome depends on the value of the response.
I will never ‘audit’ the company into compliance because this company, and any other company, is capable of straying off course faster than any fixing of non-compliances can pull them back on. The message from my audit is; “we are still finding N/cs and, as long as we can, the registrating auditor can. And, even more important, you are not developing and implementing the Documented System as you promised you would! That is the N/C that matters. The response to this should be proactive and preventive.”

If I would let them, management review would be happy to use the number of ‘majors’ as a measurement of my audit process. But I won’t let them and I keep pushing the results back onto their plate.

Anyway, if the number of ‘majors’ rises, does that indicate that the audit
process is improving or that it has deteriorated? If the majors reduce, same question. If it stays the same then are we getting better at finding them but less good at avoiding them, or are we getting less good at finding them but better at avoiding them?

Safer, I think, to go back to Brian Charles Kohn’s advice and “monitor metrics significant to the business”. Of course it is reasonable to expect to find a bunch of majors soon after a new quality system is put in place, and, having found them, they shouldn’t happen again - for a few months anyway - but, in the long term, the majors are a function of management responsibility and not an indictor of the effectiveness or the lack of effectiveness of the audit process.

Audit is nothing on it’s own. Finding defects is dead easy. Getting the general trend is also easy. Finding all defects or finding the really important ones is a different matter. It’s not an improvement process upon which one should bet the company.

rgds, John Cullen

[Kevin Mader]

John,

Well stated! To steal a line from Crosby "Measurement of nonconformance is not Quality". A downward trend in majors/minors in auditing means little (nothing) if the process is not understood (inputs/outputs). It is management's responsibility to understand if things are as they appear. When inspecting, how often will the inspector reverify a measurement found within specification? How about when a measurement is found out (especially when it's close!)? Management must question the validity of information presented, especially in the case of Internal Quality Audits as it is a major source of information for the Quality System it represents. Folks tend to question less when things are as they are desired or expected, they accept information on face-value alone. Danger looms near if you aren't careful. I feel you are doing the right thing at Management Review. Don't let them sway your judgement and keep up the good work! Sorry I didn't respond sooner, but I did print it out to repond later as I feel you've touched on a significant issue.

Back to the group...