Internal Auditing Metrics - Findings are not normalized, predictable metrics

Marc

Hunkered Down for the Duration
Staff member
Admin
#1
Comments anyone?

-----snippo-----

From: (Brian Charles Kohn)
Subject: RE: Q: Metrics for ISO Performance/Shugart/Gazley/Kohn

> One suggestion is to track your number of internal audit findings,
> grouping by "major", "minor".

I'd suggest that y'all NOT do that. Internal Audit Findings are not normalized, predictable metrics. As a matter of fact, many new companies will find that the number of findings increases for many reasons other than compliance going down:

- Internal auditors gaining a greater understanding of the requirements.
- Multi-part findings being split into separate findings for easier follow-up.
- Many minor findings found as a result of resolving a significant major finding.

I tried to develop some normalizing factors, about 10 years ago, but didn't feel they did the job. (Categorizing findings as either problems with the design of the quality system , problems with compliance with the defined system [C], or problems with the maintenance of evidence [E]; each were tracked separately and we were able to see as 'S' went down over time, 'C' had risen and then as 'C' subsided, 'E' went up and then down over time. At least that was the theory.) In the end, it didn't work, and wasn't very helpful.

First, I'd monitor metrics significant to the business. Beyond that, I'd monitored the following metrics to gauge "health of the quality system" when I was management rep:

- % of internal audits completed on time
- % audit findings with adequate evidence*
- value of the preventive actions initiated by management review
- % of CARs "on-plan"

* For each assertion of compliance, notes taken to show that objective evidence was assessed to establish compliance. For each finding, evidence indication of what evidence was observed to determine noncompliance. This was a critical measure to determine whether internal audit was effective, which in turn, made the first metric meaningful.

Brian Charles Kohn
http://www.synap.com
 
Elsmar Forum Sponsor

Marc

Hunkered Down for the Duration
Staff member
Admin
#2
From: "John Gazley"
Subject: Re: Metrics for ISO performance/Shugart/Gazley/Kohn/Gazley

Brian-

First I would suggest that y'all think before you tell someone not to use any methods presented on this forum unless you are absolutely sure they have not been found to be valid at other companies.

>As a matter of fact many new companies will find that number of
>findings increase for many reasons other than compliance going
>down:
> - Internal auditors gaining a greater understanding of the
> requirements

"This should remain or possibly increase while your audit team gains experience, your system bugs are discovered, and overall awareness of the requirements grows."

>findings increase for many reasons other than compliance going down

We are all quality professionals, I doubt there are few among us who would not investigate any trends in this metric before simply labeling it as caused by non-compliance increasing/decreasing. Like any other process monitoring, investigation of causes is key.

I also stated that after the initial learning curve there should be a sharp decline in majors. I stand by this statement, and have witnessed it personnally at three other facilities I have helped gain QS9000 or ISO9000 certification. All three of which currently use this as an accepted metric. I have read countless articles and books written by experts in the field which reference this as a viable metric.

If a decrease in your "major" findings is not an item used to gage the functioning of your quality system, your missing the boat.

Please explain the viability as a metric:

- % of internal audits completed on time.
*If your audit team spends an extra day or two on your internal audit but does it correctly and efficiently, how can you consider this a metric of your Quality Sytem performance.

John
 

Marc

Hunkered Down for the Duration
Staff member
Admin
#3
From: (Brian Charles Kohn)
Subject: Re: Metrics for ISO performance/.../Kohn/Gazley/Kohn

> - % of internal audits completed on time.
> *If your audit team spends an extra day or two on your internal
> audit but does it correctly and efficiently, how can you consider this
> a metric of your Quality Sytem performance.

This was indeed confusing. What I actually monitored was the % of
internal audits STARTED on time.

Sorry about the confusion.

Brian Charles Kohn
 

Kevin Mader

One of THE Original Covers!
Staff member
Admin
#4
Marc,

Good postings. I use something similar to the method above. Interesting to me to see the effort made to determine if there was a correlation between different factors. Could be a factoral design problem worth investigating with some spare time in the future.
 
D

Don Winton

#6
Interesting to me to see the effort made to determine if there was a correlation between different factors. Could be a factoral design problem worth investigating with some spare time in the future.
Got Data? Send it. Spare time? Minimul, but will look into it.

Regards,
Don
 
J

John C

#7
Internal audit metrics.

I measure my own internal audit program by it’s efficiency. If I can get a team together, do the job and complile the report with the least impact on the day to day responsibilities of the auditors and the operation, and the least hassle and effort on my own part, then I think it’s going well. The quality of the audit depends on the quality of people at my disposal, and this can vary a lot, but the outcome varies little because the outcome depends on the value of the response.
I will never ‘audit’ the company into compliance because this company, and any other company, is capable of straying off course faster than any fixing of non-compliances can pull them back on. The message from my audit is; “we are still finding N/cs and, as long as we can, the registrating auditor can. And, even more important, you are not developing and implementing the Documented System as you promised you would! That is the N/C that matters. The response to this should be proactive and preventive.”

If I would let them, management review would be happy to use the number of ‘majors’ as a measurement of my audit process. But I won’t let them and I keep pushing the results back onto their plate.

Anyway, if the number of ‘majors’ rises, does that indicate that the audit
process is improving or that it has deteriorated? If the majors reduce, same question. If it stays the same then are we getting better at finding them but less good at avoiding them, or are we getting less good at finding them but better at avoiding them?

Safer, I think, to go back to Brian Charles Kohn’s advice and “monitor metrics significant to the business”. Of course it is reasonable to expect to find a bunch of majors soon after a new quality system is put in place, and, having found them, they shouldn’t happen again - for a few months anyway - but, in the long term, the majors are a function of management responsibility and not an indictor of the effectiveness or the lack of effectiveness of the audit process.

Audit is nothing on it’s own. Finding defects is dead easy. Getting the general trend is also easy. Finding all defects or finding the really important ones is a different matter. It’s not an improvement process upon which one should bet the company.

rgds, John Cullen
 

Kevin Mader

One of THE Original Covers!
Staff member
Admin
#8
John,

Well stated! To steal a line from Crosby "Measurement of nonconformance is not Quality". A downward trend in majors/minors in auditing means little (nothing) if the process is not understood (inputs/outputs). It is management's responsibility to understand if things are as they appear. When inspecting, how often will the inspector reverify a measurement found within specification? How about when a measurement is found out (especially when it's close!)? Management must question the validity of information presented, especially in the case of Internal Quality Audits as it is a major source of information for the Quality System it represents. Folks tend to question less when things are as they are desired or expected, they accept information on face-value alone. Danger looms near if you aren't careful. I feel you are doing the right thing at Management Review. Don't let them sway your judgement and keep up the good work! Sorry I didn't respond sooner, but I did print it out to repond later as I feel you've touched on a significant issue.

Back to the group...
 
Thread starter Similar threads Forum Replies Date
A Green Belt Project for Improving Internal Process Auditing and Metrics Six Sigma 5
S Risk based internal auditing Internal Auditing 6
F AS9100D Internal auditing requirements Internal Auditing 3
R Does any here use an internal auditing tool that works on different platforms? Internal Auditing 3
W Does anyone have an API Q2 checklist for internal auditing? Oil and Gas Industry Standards and Regulations 1
blackholequasar Internal Auditing Inspiration - Getting volunteers to perform internal audits. Internal Auditing 22
W Internal Auditing carried out by a 3rd party - Review of previous audits AS9100, IAQG 9100, Nadcap and related Aerospace Standards and Requirements 3
S Internal Auditing for API Spec Q1 - auditor qualification requirements Oil and Gas Industry Standards and Regulations 6
E Informational Internal Audits - Wear multiple hats what can and can't I audit (so I'm not auditing my own work) Internal Auditing 144
N Online Internal Auditing Course for ISO 13485 - Suggestions ISO 13485:2016 - Medical Device Quality Management Systems 8
U Internal auditing - Company employees or contract second party Internal Auditing 10
K Internal Auditing - Umbrella QMS and Multiple Standards Oil and Gas Industry Standards and Regulations 4
D Auditing Our Outsourced 2nd-3rd Party Internal Audit Company ISO 13485:2016 - Medical Device Quality Management Systems 6
G AS9101 Rev F - Worksheets for internal auditing AS9100, IAQG 9100, Nadcap and related Aerospace Standards and Requirements 11
S ISO 13485:2016 and MDSAP internal auditing ISO 13485:2016 - Medical Device Quality Management Systems 6
S ISO 9001:2015 - Internal Auditing - Audit to the Standard? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 6
R Internal Auditor auditing Internal Audit Procedure (AS9100) Internal Auditing 18
M Is Automated Internal Auditing Possible? Internal Auditing 13
C Internal Auditing Requirements (ISO 9001:2008) Internal Auditing 3
L Auditing Design and Development in ISO 9001 (Internal Audit) Internal Auditing 1
sswaim Auditing Internal Laboratory Personnel for Competence General Auditing Discussions 4
K Internal Auditing a previous Nonconformance? Internal Auditing 19
P Recommended books on ISO 27001:2013 Implementation and Internal Auditing IEC 27001 - Information Security Management Systems (ISMS) 4
M Are auditing checklists required for Internal Audits? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 13
B ISO/TS16949 Internal Auditing - How many auditors? Internal Auditing 4
bgoers PFMEA, Internal Auditing, Corrective Action Training In Native Language (China) Training - Internal, External, Online and Distance Learning 1
Gman2 Internal Auditing Requirements before ISO 9001 Registration ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
S Auditing TS 16949 Cl. 7.6.3.1 - Internal Calibration Laboratory Requirements IATF 16949 - Automotive Quality Systems Standard 8
T ISO 9001 Internal Auditing Auditor Training in Amsterdam Training - Internal, External, Online and Distance Learning 1
S In an internal auditing desert and I'm the only one here.... AS9100, IAQG 9100, Nadcap and related Aerospace Standards and Requirements 15
G Internal Auditing in a Multi-Site Environment ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 17
B Internal Auditing of MDD and CMDR Requirements Other Medical Device Related Standards 5
O Internal Auditing in small Engineering company ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 6
T Looking for a webinar on Internal Auditing General Auditing Discussions 3
R On Auditing Internal Audit Process - How Independence can be Established Internal Auditing 4
D Auditing Abroad - Internal Audits of our European Sister Companies ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 16
R Auditing a process outside the realm of the formal Internal Audit ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
G New to Internal Auditing - Best questions to ask? Internal Auditing 17
M Auditing the Internal Audit Process - 8.2.2 General Auditing Discussions 2
J Auditing the Internal Auditing Process - Audit Nonconformance General Auditing Discussions 3
S Internal Auditing related to RISK Internal Auditing 9
W Internal Auditing - Observational Checklist for a Behavior Based System Internal Auditing 3
R Internal Auditing Checklist - Major NCR because the Checklist was not Completed Internal Auditing 17
V Depth of Internal Auditing and Training aspects in Research & Development (R & D) Internal Auditing 4
N Good Internal Auditing Training Courses Training - Internal, External, Online and Distance Learning 13
L Internal Auditing Reports / Documents - Design and Content Document Control Systems, Procedures, Forms and Templates 1
N Recommend internal auditing training 101 please (Tucson or Phoenix, AZ) Training - Internal, External, Online and Distance Learning 1
L Internal Auditing - How can I audit my QMS independently? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 12
C Internal Auditing - How to make it useful? Internal Auditing 36
K Sample Questions for Auditing Management Rep , Internal Audit and Reg. Compliance Internal Auditing 7
Similar threads


















































Top Bottom