Hi All!
This is my first visit to the site and the forum. I feel as though I've struck gold!!

I have a question. I have been performing internal audits for three years and typically I report at management reviews which audits were performed, which audits were past due on the schedule, and which random audits were performed due to a customer complaint, major breakdown, or rework/scrap costs.
I tell them how many corrective actions were issued and which ISO element had the most non-compliances.

Now, we are looking at "measuring the effectiveness" of the internal audit process. Am I way off base in believing that the measurements we are looking for will be found in the corrective action follow ups?

Hi Toni,

Welcome to the cove. It's the best resource you can have (in my opinion).

I would agree with you in using corrective actions to measure audit effectiveness. That seems like it would at least be a good starting point.

I will keep my answer short since I am still new to this. I am sure you will get responses from plenty of knowledgable people. :p

You could present-
Number of disputed audits
Feedback of auditees
Reaction time analysis to close non conformances over different time periods
Process improvement findings

Toni Herrera said:

Hi All!
This is my first visit to the site and the forum. I feel as though I've struck gold!!

I think you have, Toni. That was my feeling as well when I first found the Cove, and I still consider it a gold mine. Welcome aboard :D .

Ok.... Over to your question:

Concerning the audits themselves, I do an annual evaluation looking at among other things:

Planned vs carried out audits.
The number of new audit findings vs closed ones during the year.
Average time from registration of a finding to completed action.

I also do a weekly follow up on:

Total no. of "active" issues.
No. of delayed actions (Yes, we have them, I'm sorry to say)
No. of actions due within a month ( Early warning to avoid the above).

/Claes

Claes Gefvenberg said:

Concerning the audits themselves, I do an annual evaluation looking at among other things:

Planned vs carried out audits.
The number of new audit findings vs closed ones during the year.
Average time from registration of a finding to completed action.



I would point out Claes, that IMHO only the first one measures auditing effectiveness. The rest are concerned with corrective/preventive action. Actions are neither necessarily developed, nor performed by internal auditors. So, it is just another system being measured.

To my mind NITIN presented a more relevant list of indicators:

NITIN said:

You could present-
Number of disputed audits
Feedback of auditees
Reaction time analysis to close non conformances over different time periods
Process improvement findings


Yours,

Anton :p

In order to consider the effectiveness of internal audits I would have thought that we need to think about the purpose of internal audits.

The 'trouble' with auditing is that it is not an exact science, as such counting numbers of non-conformances found by clause or area may not give any real insight into the effectiveness of the audit process.

True Mr. Greenaway,

I would rather seek for some intricate indicators showing somehow the ratio between the non-conformances found and non-conformances taking place. Clear that you will never be sure that you've found ALL the NC's.
So you will have to judge by indirect results like customer complaints and other satisfaction/dissatisfaction indicators.

We use:
- number of audits performed vs. planned
- number of NC's found vs. previous periods corrected through general indicators of quality in the area audited (like customer complaints, defect rate dynamics, cost dynamics etc.)
- comprehensiveness of audit (judging by audit reports)

You see, the indicators are qualitative rather than quantitative.

Regards,

Anton

Anton Ovsianko said:

I would point out Claes, that IMHO only the first one measures auditing effectiveness. The rest are concerned with corrective/preventive action. Actions are neither necessarily developed, nor performed by internal auditors. So, it is just another system being measured.

To my mind NITIN presented a more relevant list of indicators:

Yours,

Anton :p

On the other hand the very reason for doing an audit in the first place is that we want improvement. I consider the actions to correct audit findings to be part of the internal audit process.

Toni asked how to measure the effectivness of the internal audit process. If we assemble a heap of findings and no action is taken, I wouldn't call it effective.

/Claes

Cleas,

I would say that internal audit is a part of continuous improvement process, which also includes corrective and preventive action and so on. We can as well measure effectiveness of internal audit process separately from the effectiveness of corrective action and preventive action. Of cource, we should also measure their total effectiveness as a unite system.

MHO

Anton

A distant measure I use is to compare internal findings against external findings. This could be dangerous because it tempts us into changing our audits to match the registrars. We shouldn't do that. If used properly, it can show if our auditors can find things. For example, if the internal audit finds nothing and the external finds 15 majors, then we need to find out why there is such a large difference. That goes also for if the internal finds 15 and the external finds nothing.

Some of the things already mentioned are better measure. That is why I said a 'distant' measure. The important thing is to look at YOUR system and decide what measures YOU feel will show if YOUR internal audit program is working!

Warning, this place is addictive! :thedeal:

Anton Ovsianko said:

Cleas,

I would say that internal audit is a part of continuous improvement process, which also includes corrective and preventive action and so on. We can as well measure effectiveness of internal audit process separately from the effectiveness of corrective action and preventive action. Of cource, we should also measure their total effectiveness as a unite system.

MHO

Anton

Yep. There are many different ways to achieve the same result. As long as it works: Fine. :agree:

/Claes

Thank You Everyone, for your responses! It has helped me immensly. I think I may be blinded by the forest as I look for my own measurements!

Typically, when I perform a third party audit, (which is extremely easier to do than an internal audit) I ask the Management Rep if he believes his internal audits are directing the organization towards improvements. If his reply is yes, than I will play the "show me an example or two" card. The evidence provides us with a way to measure effectiveness.

I don't want to say our system is "ineffective," but I know we don't issue enough corrective/preventive actions. We are kind of hesitant due to open actions spanning back to 7/00, in fact we have begun to simply "add on findings" to open actions instead of issuing new actions. Maybe as I re-read this, the Internal Audit program can report the bottleneck of the C/A, P/A process and the effects it is having on the Internal Audit program.

Thanks ALL!!

in addition to the above:

How many NCs are for repetitive issues?
How many drove preventive action rather than CA?
How well did the audits prepare you for the registrar visit?
How well are the audits received by the auditee?
How are your auditors doing (tag along and watch)?
have an outsourced audit of your audit program periodically.

In a challenging argue with a branch friend of mine I faced a strange (for me) question. Telling him that in our company due to the nature of bussiness we do not use technical statistics, I was told that I'm wrong because I already use them processing results of internal audit.:confused: I have to admit I didn't think about ever.:bonk:
Our bussiness is a small production and single unit (please somebody do correct me if I'm wrong as I don't know an english word for production of one single-big-fat-expensive-complex product), working with small "population of events" (3 employees or 3 directors - however you want, few suppliers, some even monopolist, few clients, one or two audit/year for processes/departments a.s.o).
So: statistical techniques to internal audits?