Quality internal audit metrics - How to determine and show Audit Effectiveness

O

omartinez

#1
Hello Folks how you doing ? Today is Friday Thanks God Friday is what all people say now…..
I have a doubt about Internal audit metrics , I have Internal Audit on Time as a metric but , this metric does not show the effectiveness of my process. I mean this just show that we are meeting times, But how can be sure that this audits were effectives at the plan…

Thanks and have good weekend.
Omar
 
Elsmar Forum Sponsor
D

D.Scott

#2
Hi Omar,

I'm not sure if Friday the 13th has the same superstition in Mexico as it does here but there are a lot of us who are staying away from black cats today :biglaugh:

As for your question, tracking timeliness of audits is a good start for measuring effectiveness of the audit system. As your plan or goal is to determine suitability and effectiveness of your process plans, I would use that as part of your measurable. I would look at the contribution to continual improvement made by internal audits. Personally, I don't like using the number of NCs as a measurable but the corrective actions and preventions would be suitable. The key is to demonstrate that your internal audit process is doing the job you intended it to do. Just like other processes, you set the intended goals and objectives then through measurable data review, with management, the effectiveness of the process.

Don't over complicate your system. Determine what you want to accomplish, remembering to address the requirements of your system, and set your measurables.

Good luck and watch that you don't walk under any ladders today.

Dave
 

Helmut Jilling

Auditor / Consultant
#3
omartinez said:
Hello Folks how you doing ? Today is Friday Thanks God Friday is what all people say now…..
I have a doubt about Internal audit metrics , I have Internal Audit on Time as a metric but , this metric does not show the effectiveness of my process. I mean this just show that we are meeting times, But how can be sure that this audits were effectives at the plan…

Thanks and have good weekend.
Omar
I think you are thinking down the correct path.

Let's apply the process approach to your question. Internal auditing is a process. The purpose of the process is to make us better. How will it make us better? By finding things we are doing wrong, or things we could do better. We will write those findings as CA's, PA's and Cont. Improvements. So "findings that make us better" are the product or outputs of the audits. The more things we find that make us better, the more useful and effective the audits were.

If you agree with that logic, and if the main output is findings, shouldn't we track findings as the prmary metric? Maybe even try to track the results and savings from the audits, if you are really ambtious. Six Sigma measures the results of projects. It helps justify the cost of the audits, and if the metrics is very good, even helps build support and buy in.

I would rather have a late audit that was beneficial, than an non-value added audit that was on time...so timeliness would appear to be a less meaningful metric.
 
V

vanputten

#4
Search the Cove

If you do an advanced search of the Cove for "Audit", "effectiveness", "Measure", you will get links to similar threads. This question has been asked before.

If we track the audit process' effectives by the number of nonconformances found, I ask the following questions:

If the number of NC's found is large, does that mean the auditor did a good job or that the audit subject is poor?

If the auditor found few NC's, is that a function of the auditor or the audit subject?

How do you know?

And if you set a target for the number of NC's per audit, you can bet that the auditor will do what it takes to meet that target.

Be careful in how you determine audit process effectivenss.

Regards,

Dirk
 

Helmut Jilling

Auditor / Consultant
#5
vanputten said:
If you do an advanced search of the Cove for "Audit", "effectiveness", "Measure", you will get links to similar threads. This question has been asked before.

If we track the audit process' effectives by the number of nonconformances found, I ask the following questions:

If the number of NC's found is large, does that mean the auditor did a good job or that the audit subject is poor?

If the auditor found few NC's, is that a function of the auditor or the audit subject?

How do you know?

And if you set a target for the number of NC's per audit, you can bet that the auditor will do what it takes to meet that target.

Be careful in how you determine audit process effectivenss.

Regards,

Dirk


This whole line of reasoning stems from the OP recognizing that "ontime audits" is an interesting metric, but not meaningful. I frequently see the same metric in audits. The line of questions I asked may not be perfect, but I stand behind the line of reasoning.

You make some interesting points, but if one extrapolates too far, it veres off course.

I certainly would not set a target for how many findings were generated. It would incnetivize inapprorpiately. But I frequently see audit cycles where there were no meaningful findings after 10 audits. And, of course, I found 8 fiundings the first day. In those scenarios, obviously the audits are not getting the job done. And, as we look for root causes, the rest of this unfolds.

As an auditor, I cannot recoomend the client should measure this or that. But I can ask questions that can put them back on a meaningful track of thinking.

Is it perfect? no. But, it could be far closer to the point than Ontime Audits.

Audits are a process. Processes have outpouts. I think ISO recommends that we measure the effectiveness of the outputs of our processes. I would say that is closer to home, and a good start.

Actually, I liked the last item I added, which was to track the results of the findings. That would stand a lot of scrutiny.
 

Wes Bucey

Quite Involved in Discussions
#6
There is no question whether I am idiosyncratic about certain aspects of dealing with a QMS. With that disclaimer, I declare that an internal audit does not stand alone as a disconnected process. What happens when the internal audit team turns in a report for Management Review? How does the MR view the relationship between findings, resolutions, and PROFIT? If no findings are produced, but profit continues to tank, I presume top management will consider the effectiveness of the IA to be compromised regardless of any other metric.

Alternately, if the MR detects animosity between IA and production so that some sort of passive-aggressive dynamic keeps generating the same findings time after time without achieving a permanent resolution, MR will probably do something drastic to change the dynamic, regardless of any metric.

Bottom line:
Effectiveness of IA has nothing to do with IA metrics, it has to do with the results of the organization as a whole.
 

Helmut Jilling

Auditor / Consultant
#7
Wes Bucey said:
Bottom line:
Effectiveness of IA has nothing to do with IA metrics, it has to do with the results of the organization as a whole.

I don't agree, Wes. Many companies measure the economic benefit of CI projects. ROI is evaluated on any significant capital investment before we approave the funds. Six Sigma requires calculating the expected savings as part of the project. So does Kaizen and Lean.

Why is internal auditing the only improvement activity where no one tracks the inherent benefits? I would propose it is because it evolved out of the negatively-tainted customer or military inspection model where the auditor was never viewed as a value-added part of the business.

But, in this ISO Process model, auditing is a process to help us improve. So, auditors should be recognized as part of the team. They should be trained to view themselves as part of the team. And, they should be trained to look for preventive actions and improvement opportunities as well, not just finding people who aren't following their procedures.

When I train auditors, I begin a discussion of Current State vs. Ideal State concepts. I follow that with the statement that "an auditor's role is to identify anything that is not as it should be."

I stress they are to widen their perspective. And, in every class, the idea is embraced as a great idea, but they always say "we didn't know we were allowed to do that." I view that as a tragic lost opportnity, to not empower and train auditors to do that.

I teach them to list those observations, discuss them with their auditees to get their input. Then review them with the Lead Auditor to filter out the oddball items. Then, write CA's, PA's and OFI's to address the remaining findings that were approved as beneficial and legit.

I find companies who do this get dramatic improvements in their systems. Their metrics are consistently at benchmark levels. And they don't fuss that their ISO or TS systems don't provide value.

PS: My premise is not so much that I want companies to measure results of audit findings, though I think that will certainly come in the next few years. But I would encourage them to recognize auditing as a true improvement process and treat it as such. And, on that basis, I still believe that tracking audit findings as a metric will provide the starting point to understand whether they are getting value. Far more meaningful than whether the audit was done on time.

After all, my clients evaluate the effectiveness of whether my audits had value based on whther they got good ideas and improvements from it. Why shouldn't internal audits be viewed the same way?
 

Wes Bucey

Quite Involved in Discussions
#8
hjilling said:
I don't agree, Wes. Many companies measure the economic benefit of CI projects. ROI is evaluated on any significant capital investment before we approave the funds. Six Sigma requires calculating the expected savings as part of the project. So does Kaizen and Lean.

Why is internal auditing the only improvement activity where no one tracks the inherent benefits? I would propose it is because it evolved out of the negatively-tainted customer or military inspection model where the auditor was never viewed as a value-added part of the business.

But, in this ISO Process model, auditing is a process to help us improve. So, auditors should be recognized as part of the team. They should be trained to view themselves as part of the team. And, they should be trained to look for preventive actions and improvement opportunities as well, not just finding people who aren't following their procedures.

. . . . .

PS: My premise is not so much that I want companies to measure results of audit findings, though I think that will certainly come in the next few years. But I would encourage them to recognize auditing as a true improvement process and treat it as such. And, on that basis, I still believe that tracking audit findings as a metric will provide the starting point to understand whether they are getting value. Far more meaningful than whether the audit was done on time.

After all, my clients evaluate the effectiveness of whether my audits had value based on whther they got good ideas and improvements from it. Why shouldn't internal audits be viewed the same way?
:truce: I think we're in the same church, just different pews.

The point I thought I was making is that many "auditing teams" are constrained by their bosses to think in terms of short-term metrics on how the team performs its tasks as opposed to the benefit the team gives to the organization. Without focusing on the "big picture," those teams can end up alienating production folks being audited because they focus more on making their own "metrics" than on the benefit to the organization.

Of course, there are many organizations with smoothly functioning processes who use their metrics to point toward areas for real improvement. Sadly, there are probably more organizations who are focusing on meaningless metrics (kind of like the managers in Deming's Red Beads) than on meaningful metrics.
 

Helmut Jilling

Auditor / Consultant
#9
Wes Bucey said:
:truce: I think we're in the same church, just different pews.

The point I thought I was making is that many "auditing teams" are constrained by their bosses to think in terms of short-term metrics on how the team performs its tasks as opposed to the benefit the team gives to the organization. Without focusing on the "big picture," those teams can end up alienating production folks being audited because they focus more on making their own "metrics" than on the benefit to the organization.

Of course, there are many organizations with smoothly functioning processes who use their metrics to point toward areas for real improvement. Sadly, there are probably more organizations who are focusing on meaningless metrics (kind of like the managers in Deming's Red Beads) than on meaningful metrics.

I would agree with that. I try to steer them in the right direction, but it seems awfully easy to play to short term metrics, rather than real improvement.
 

harry

Super Moderator
#10
One interesting difference between six sigma and ISO to me is that six sigma involves heavy investment - in time, money, people etc. It is often the idea of some hot shot CEOs or some powers to be and therefore it is given prominence and resources (including your CFO, COO and what nots). It had unwittingly become a 'strategic' issue and full attention is given to it, including bloating the results where necessary.

ISO is a bread and butter issue - nothing to boast about because almost every other company has it. It is often viewed as part of production and therefore a line function. Very often it does not get the correct or necessary resources (people).

With due respect to the line people (often due to their limited exposure and experience) they are usually very good with their respective work (part of a full process) but not well verse with the operations of other sections or departments and alien as far as corporate matters are concerned. How well then can they carry out an audit? Not to mention getting the metrics right (this may not apply to bigger organisations that employ full time professional auditors).

From what was described, I note that hjilling had done a very good job for his audit training but then how much can one do within the 3-5 days ( if I am not wrong). And how well can the audit team articulate their findings and jive it with corporate/strategic objectives (the big picture) when they present it for management review?

It is my view that we should convince our clients that ISO/QMS should be made a 'strategic weapon' and be given attention that commensurates with its importance (including provision of right resources and tracking of improvements and acheiving of objectives). Over here, I deal with a lot of SMI's and often get to meet the CEOs. I had often ask for some 'one to one' time to drive this message through and with some success.

In this connection, have you guys use the 'Sydney Model' formulated by the ISO APG group? I use this model together with ISO 9004 for my presentations when I get to speak to these CEO/MDs.

The Sydney model is an attempt to alleviate the position of ISO to that of the others like six sigma and I attached the file here if you like to have a look.
 

Attachments

Last edited:
Thread starter Similar threads Forum Replies Date
P Quality objectives - must they include CAPA and internal audit topic? ISO 13485:2016 - Medical Device Quality Management Systems 28
T Internal Audit not done for Quality Department Internal Auditing 11
Q Internal Audit of Product Quality Complaint System (21 CFR Part 820) Customer Complaints 9
F Quality Audit and Internal Control Audit Quality Manager and Management Related Issues 3
S Internal Quality Audit Tracking Sheet with Auto Report Excel .xls Spreadsheet Templates and Tools 2
M Internal Quality Audit for Maintenance Department ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 4
A Risk Based Internal Quality Audit Scheduling and Planning Internal Auditing 2
M Internal Audit of our ISO 9001 Quality Systems prior to the Registration Audit Internal Auditing 9
B Where do I start when I Internal Audit the Quality Management System? Internal Auditing 8
A How to Address Internal Quality Audit Findings Internal Auditing 8
9 Internal Audit vs. Internal Quality Audit - What is the difference Internal Auditing 23
R ISO 9001 Internal Quality Audit by Consultants (External agency) Internal Auditing 5
V Internal Quality Audit Focuses Internal Auditing 8
S Motivation for IQA (Internal Quality Audit) Auditors ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 33
A How to Audit the Quality Department (Internal Audit) Internal Auditing 15
B Value adding for Internal Quality Audit training General Auditing Discussions 18
V Internal Quality Audit - Themes for auditing the company QMS Internal Auditing 1
P External Contract Auditor for Internal Quality Audit Internal Auditing 18
J Internal Audit NC ISO 9001 Clause Mapping - Quality Objective Achievement Plan ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 6
J Quality Manual Internal Audit at least once prior to certification audit ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
S Software Quality Audit - Internal Audit Checklists Software Quality Assurance 6
6 How to implement a functional system and carry out high quality internal Audit? Quality Manager and Management Related Issues 2
L Is an internal audit of quality department required? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 10
J Cost-Benefit Analysis of Internal Quality Audit Misc. Quality Assurance and Business Systems Related Topics 1
A Is an Internal Quality Audit Checklist necessary? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 10
S Internal Quality Audit Software - IQA Dash Board Internal Auditing 1
C An Internal Audit Challenge - Audit of the Quality System Effectiveness Internal Auditing 42
M Internal Quality Audit (IQA) Report Sample Document Control Systems, Procedures, Forms and Templates 11
B Internal audit and quality performance improvement Internal Auditing 11
A Internal Quality Audit - Can I follow the Auditors and Ask Questions? Internal Auditing 11
Q How to Internal Audit a Quality department - ISO 9001:2000 14001:2004 OHSAS18001:1999 Internal Auditing 20
W Internal Audit Schedule - Seeking Examples - Quality Audits Internal Auditing 14
S Internal Quality Audit Management Software General Auditing Discussions 13
M Can external quality consultant carry out internal audit? Internal Auditing 17
M Internal Quality Audit - Seeking Example Procedure Document Control Systems, Procedures, Forms and Templates 1
M Internal Quality Audit File is missing !! ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 18
M Who may audit the internal quality audit system? General Auditing Discussions 8
M ISO 9001:2000 Process Based Internal Quality Audit Plan General Auditing Discussions 11
D Internal Quality Audit - Desk Top Audit First? General Auditing Discussions 8
K Are internal audit results a true reflection of the organisations quality system Internal Auditing 2
R Monitor production quality - Internal KPIs Manufacturing and Related Processes 5
S ISO 13485 Lead Auditor - Debate between our Quality Team and Regulatory Auditor - Internal Auditor Training ISO 13485:2016 - Medical Device Quality Management Systems 17
Ron Rompen Internal Quality Alert Program Document Control Systems, Procedures, Forms and Templates 0
Gman2 Quality Record Retention (Internal Audits, CA's) ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
A Suggestions of Videos for Internal QUALITY Training Training - Internal, External, Online and Distance Learning 3
S Is it a Finding if all Internal Auditors are from the Quality Department? Internal Auditing 18
A Quality Awareness Communications through Internal TVs Misc. Quality Assurance and Business Systems Related Topics 2
L Quality Manager wants Internal Auditor to softgrade the findings - What shall I do? Internal Auditing 23
S Internal Quality Assessment/Questionnaire Misc. Quality Assurance and Business Systems Related Topics 8
K Questioning Top Management on The Quality Policy during Internal Audits Internal Auditing 7

Similar threads

Top Bottom