The Elsmar Cove Business Standards Discussion Forums More Free Files Forum Discussion Thread Post Attachments Listing Elsmar Cove Discussion Forums Main Page
Welcome to what was The Original Cayman Cove Forums!
This thread is carried over and continued in the Current Elsmar Cove Forums

Search the Elsmar Cove!

Wooden Line
This is a "Frozen" Legacy Forum.
Most links on this page do NOT work.
Discussions since 2001 are HERE

Owl Line
The New Elsmar Cove Forums   The New Elsmar Cove Forums
  Auditing
  Internal Auditing Metrics

Post New Topic  Post A Reply
profile | register | preferences | faq | search

UBBFriend: Email This Page to Someone! next newest topic | next oldest topic
Author Topic:   Internal Auditing Metrics
Marc Smith
Cheech Wizard

Posts: 4119
From:West Chester, OH, USA
Registered:

posted 20 December 1998 03:25 AM     Click Here to See the Profile for Marc Smith   Click Here to Email Marc Smith     Edit/Delete Message   Reply w/Quote
Comments anyone?

-----snippo-----

From: bicker@nospam.no (Brian Charles Kohn)
Subject: RE: Q: Metrics for ISO Performance/Shugart/Gazley/Kohn

> One suggestion is to track your number of internal audit findings,
> grouping by "major", "minor".

I'd suggest that y'all NOT do that. Internal Audit Findings are not normalized, predictable metrics. As a matter of fact, many new companies will find that the number of findings increases for many reasons other than compliance going down:

- Internal auditors gaining a greater understanding of the requirements.
- Multi-part findings being split into separate findings for easier follow-up.
- Many minor findings found as a result of resolving a significant major finding.

I tried to develop some normalizing factors, about 10 years ago, but didn't feel they did the job. (Categorizing findings as either problems with the design of the quality system [S], problems with compliance with the defined system [C], or problems with the maintenance of evidence [E]; each were tracked separately and we were able to see as 'S' went down over time, 'C' had risen and then as 'C' subsided, 'E' went up and then down over time. At least that was the theory.) In the end, it didn't work, and wasn't very helpful.

First, I'd monitor metrics significant to the business. Beyond that, I'd monitored the following metrics to gauge "health of the quality system" when I was management rep:

- % of internal audits completed on time
- % audit findings with adequate evidence*
- value of the preventive actions initiated by management review
- % of CARs "on-plan"

* For each assertion of compliance, notes taken to show that objective evidence was assessed to establish compliance. For each finding, evidence indication of what evidence was observed to determine noncompliance. This was a critical measure to determine whether internal audit was effective, which in turn, made the first metric meaningful.

Brian Charles Kohn
http://www.synap.com

IP: Logged

Marc Smith
Cheech Wizard

Posts: 4119
From:West Chester, OH, USA
Registered:

posted 20 December 1998 03:47 AM     Click Here to See the Profile for Marc Smith   Click Here to Email Marc Smith     Edit/Delete Message   Reply w/Quote
From: "John Gazley"
Subject: Re: Metrics for ISO performance/Shugart/Gazley/Kohn/Gazley

Brian-

First I would suggest that y'all think before you tell someone not to use any methods presented on this forum unless you are absolutely sure they have not been found to be valid at other companies.

>As a matter of fact many new companies will find that number of
>findings increase for many reasons other than compliance going
>down:
> - Internal auditors gaining a greater understanding of the
> requirements

"This should remain or possibly increase while your audit team gains experience, your system bugs are discovered, and overall awareness of the requirements grows."

>findings increase for many reasons other than compliance going down

We are all quality professionals, I doubt there are few among us who would not investigate any trends in this metric before simply labeling it as caused by non-compliance increasing/decreasing. Like any other process monitoring, investigation of causes is key.

I also stated that after the initial learning curve there should be a sharp decline in majors. I stand by this statement, and have witnessed it personnally at three other facilities I have helped gain QS9000 or ISO9000 certification. All three of which currently use this as an accepted metric. I have read countless articles and books written by experts in the field which reference this as a viable metric.

If a decrease in your "major" findings is not an item used to gage the functioning of your quality system, your missing the boat.

Please explain the viability as a metric:

- % of internal audits completed on time.
*If your audit team spends an extra day or two on your internal audit but does it correctly and efficiently, how can you consider this a metric of your Quality Sytem performance.

John
jgazley@aarcorp.com

IP: Logged

Marc Smith
Cheech Wizard

Posts: 4119
From:West Chester, OH, USA
Registered:

posted 20 December 1998 03:54 AM     Click Here to See the Profile for Marc Smith   Click Here to Email Marc Smith     Edit/Delete Message   Reply w/Quote
From: bicker@nospam.no (Brian Charles Kohn)
Subject: Re: Metrics for ISO performance/.../Kohn/Gazley/Kohn

> - % of internal audits completed on time.
> *If your audit team spends an extra day or two on your internal
> audit but does it correctly and efficiently, how can you consider this
> a metric of your Quality Sytem performance.

This was indeed confusing. What I actually monitored was the % of
internal audits STARTED on time.

Sorry about the confusion.

Brian Charles Kohn
http://www.synap.com

IP: Logged

Kevin Mader
Forum Wizard

Posts: 575
From:Seymour, CT USA
Registered: Nov 98

posted 21 December 1998 10:07 AM     Click Here to See the Profile for Kevin Mader   Click Here to Email Kevin Mader     Edit/Delete Message   Reply w/Quote
Marc,

Good postings. I use something similar to the method above. Interesting to me to see the effort made to determine if there was a correlation between different factors. Could be a factoral design problem worth investigating with some spare time in the future.

IP: Logged

Marc Smith
Cheech Wizard

Posts: 4119
From:West Chester, OH, USA
Registered:

posted 21 December 1998 11:40 AM     Click Here to See the Profile for Marc Smith   Click Here to Email Marc Smith     Edit/Delete Message   Reply w/Quote
Spare time? You have spare time? Can you lend me some?

IP: Logged

Don Winton
Forum Contributor

Posts: 498
From:Tullahoma, TN
Registered:

posted 09 January 1999 08:53 PM     Click Here to See the Profile for Don Winton   Click Here to Email Don Winton     Edit/Delete Message   Reply w/Quote
quote:
Interesting to me to see the effort made to determine if there was a correlation between different factors. Could be a factoral design problem worth investigating with some spare time in the future.

Got Data? Send it. Spare time? Minimul, but will look into it.

Regards,
Don

IP: Logged

John C
Forum Contributor

Posts: 134
From:Cork City, Ireland
Registered: Nov 98

posted 29 January 1999 09:00 AM     Click Here to See the Profile for John C   Click Here to Email John C     Edit/Delete Message   Reply w/Quote
Internal audit metrics.

I measure my own internal audit program by itâs efficiency. If I can get a team together, do the job and complile the report with the least impact on the day to day responsibilities of the auditors and the operation, and the least hassle and effort on my own part, then I think itâs going well. The quality of the audit depends on the quality of people at my disposal, and this can vary a lot, but the outcome varies little because the outcome depends on the value of the response.
I will never Îauditâ the company into compliance because this company, and any other company, is capable of straying off course faster than any fixing of non-compliances can pull them back on. The message from my audit is; ăwe are still finding N/cs and, as long as we can, the registrating auditor can. And, even more important, you are not developing and implementing the Documented System as you promised you would! That is the N/C that matters. The response to this should be proactive and preventive.ä

If I would let them, management review would be happy to use the number of Îmajorsâ as a measurement of my audit process. But I wonât let them and I keep pushing the results back onto their plate.

Anyway, if the number of Îmajorsâ rises, does that indicate that the audit
process is improving or that it has deteriorated? If the majors reduce, same question. If it stays the same then are we getting better at finding them but less good at avoiding them, or are we getting less good at finding them but better at avoiding them?

Safer, I think, to go back to Brian Charles Kohnâs advice and ămonitor metrics significant to the businessä. Of course it is reasonable to expect to find a bunch of majors soon after a new quality system is put in place, and, having found them, they shouldnât happen again - for a few months anyway - but, in the long term, the majors are a function of management responsibility and not an indictor of the effectiveness or the lack of effectiveness of the audit process.

Audit is nothing on itâs own. Finding defects is dead easy. Getting the general trend is also easy. Finding all defects or finding the really important ones is a different matter. Itâs not an improvement process upon which one should bet the company.

rgds, John Cullen

IP: Logged

Kevin Mader
Forum Wizard

Posts: 575
From:Seymour, CT USA
Registered: Nov 98

posted 04 February 1999 01:19 PM     Click Here to See the Profile for Kevin Mader   Click Here to Email Kevin Mader     Edit/Delete Message   Reply w/Quote
John,

Well stated! To steal a line from Crosby "Measurement of nonconformance is not Quality". A downward trend in majors/minors in auditing means little (nothing) if the process is not understood (inputs/outputs). It is management's responsibility to understand if things are as they appear. When inspecting, how often will the inspector reverify a measurement found within specification? How about when a measurement is found out (especially when it's close!)? Management must question the validity of information presented, especially in the case of Internal Quality Audits as it is a major source of information for the Quality System it represents. Folks tend to question less when things are as they are desired or expected, they accept information on face-value alone. Danger looms near if you aren't careful. I feel you are doing the right thing at Management Review. Don't let them sway your judgement and keep up the good work! Sorry I didn't respond sooner, but I did print it out to repond later as I feel you've touched on a significant issue.

Back to the group...

IP: Logged

Marc Smith
Cheech Wizard

Posts: 4119
From:West Chester, OH, USA
Registered:

posted 05 February 1999 08:39 PM     Click Here to See the Profile for Marc Smith   Click Here to Email Marc Smith     Edit/Delete Message   Reply w/Quote
Not much I can say (as always) but I am going to print this thread out and give it to a client who is struggleing with this issue.

As always, Thanks, fellas!

Anyone else have any ideas?

IP: Logged

All times are Eastern Standard Time (USA)

next newest topic | next oldest topic

Administrative Options: Close Topic | Archive/Move | Delete Topic
Post New Topic  Post A Reply Hop to:

Contact Us | The Elsmar Cove Home Page

Your Input Into These Forums Is Appreciated! Thanks!


Main Site Search
Y'All Come Back Now, Ya Hear?
Powered by FreeBSD!Made With A Mac!Powered by Apache!