I’d love for my question to be whether any of you score internal audits of your own 9001:2000 QMS, but our new management LOVES metrics, and not scoring isn’t an option I’m afraid. Can anyone give me an objective method of scoring requirement elements in an internal audit?

:thanx:

IMO scoring an audit will drive negative behavior such as hiding issues, not cooperating with the auditor, etc. You really want to reward the surfacing of issues so improvements may be made. In addition, I would question the effectiveness of an audit with zero findings.

A better area for metrics is the responsiveness in correcting issues surface by the audits.

How about the potential business impact of each nonconformity? (e.g., cost, potential lost business, ...)

How about the potential business impact of each nonconformity? (e.g., cost, potential lost business, ...)

Sure--why not make it as negative as possible? The very idea of "scoring" an internal audit--or any audit for that matter--runs counter to what the whole idea should be, namely confirmation that the system is operating according to plan. If something must be counted, count things that go right.

Sure--why not make it as negative as possible? The very idea of "scoring" an internal audit--or any audit for that matter--runs counter to what the whole idea should be, namely confirmation that the system is operating according to plan. If something must be counted, count things that go right.

I guess that's one way of looking at it. I prefer to think of it as showing the value that audits and nonconformities bring to the table ... identifying systems failures that can have a significant impact on the business ... a risk management tool.

And this would also likely improve the audit process, since many of the things often written up by auditors are trivial, mundane, a waste of time to write a CAR on. If they try to quantify the risk they'll find this out, and become better auditors. Management might like that, also.

I guess that's one way of looking at it. I prefer to think of it as showing the value that audits and nonconformities bring to the table ... identifying systems failures that can have a significant impact on the business ... a risk management tool.

Exactly. Instead of assigning some numerical scores, the book:
After the Quality Audit: Closing the Loop on the Audit Process, Second Edition
J. P. Russell and Terry Regel
http://qualitypress.asq.org/perl/catalog.cgi?item=H1070
provides an idea of Reason-Pain –Matrix that can explored for the internal audit findings.I think it is a good idea that can get Management’s attention quickly.
Regards,
Govind.

Can anyone give me an objective method of scoring requirement elements in an internal audit?It is very difficult to OBJECTIVELY quantify audit results, because the auditors themselves bring subjectivity into the picture. The company I work for has a tool called isrs7. The tool provides for quantified audit results and contain extensive guidance to the auditors about grading results. The protocol is designed to assess a business against requirements contained in many standards, such as ISO 9001, ISO 14001, OHSAS 18001, PAS 55, GRI:2004 and more.

I’d love for my question to be whether any of you score internal audits of your own 9001:2000 QMS, but our new management LOVES metrics, and not scoring isn’t an option I’m afraid. Can anyone give me an objective method of scoring requirement elements in an internal audit?

:thanx:

Go here: http://elsmar.com/Forums/showthread.php?t=17701
to my post and see if my system will work for you. It seems to work for me so far. :)

I recently returned to work after a period of extended absence. While I was gone the audit system got changed to include a score based on the number of questions in the checklist and the number of non conformances found and they used these to give a percentage score.

Personally I hate this system and it will be disappearing shortly from our system but if your management "want a metric" then this would give them one. I doubt anything useful will be obtained from it though.

Yes, Crusader's scoring system for Internal Quality Audits seems to be a good approach.

I guess that's one way of looking at it. I prefer to think of it as showing the value that audits and nonconformities bring to the table ... identifying systems failures that can have a significant impact on the business ... a risk management tool.
Trust me when I say I mean no disrespect here, Duke, but that sounds a lot like consultant-speak to me. This is not a dig; consultants need to be able to sell their services in terms that potential customers want to hear, and there's a lot to be said for identifying money pits. I just think that internal audits are the wrong place to be doing it.

And this would also likely improve the audit process, since many of the things often written up by auditors are trivial, mundane, a waste of time to write a CAR on. If they try to quantify the risk they'll find this out, and become better auditors. Management might like that, also.
If auditors are writing things up that aren't violations of the requirements, that's a different problem, and it means that auditors are ill-trained or just incompetent. On the other hand, if you're referring to deviation from requirements as trivial...

We shouldn't be doing internal audits to please management. As I said earlier, auditing should be considered a way to confirm a positive hypothesis. What's really a waste of time is measuring things that don't need to be measured.

My initial reaction was very much like that provided by Jim Wynne. After thinking about it some more, why not do some measurements for the audit process? The difference would be for the purpose of determining whether the audit is doing what is expected or it needs improvement.

For example, if you track the number of findings, percent of corrective action completed, which processes have the most findings etc. then you could determine if you should change your audit plan to make your audits more effective. With the information, you could determine which processes could have less audits and which processes need some more attention. I did that routinely but never called it a measurement system.

The real problem is when numbers have to be generated for the sole purpose of a monthly report or to justify the need for an audit system. If management is requiring either then they are missing the value of an viable audit system anyway.

Bill Pflanz

I was once asked by management in a previous job something similar to what the OP is asking for I managed to dissuade them eventually by arguing the results of audits are qualitative and not quantitative.

Metrics relating to the audit process such as NC closed on times, NC's by process, NC's by reason can be useful though, depending upon what you do with them.

Metrics relating to the audit process such as NC closed on times, NC's by process, NC's by reason can be useful though, depending upon what you do with them.

I agree--there's certainly nothing wrong with using audit results as a basis for changing something. What the OP is looking for, and what others have supported here, is something completly different, namely the assignment of arbitrary values so as to be able to construct some sort of pseudo-Pareto thing. In fact, the proposed audit scoring system will satisfy only the perceived need for a scoring system.

If, as some have suggested, an internal audit uncovers lots of problems, then it seems pretty clear to me that the company shouldn't be doing audits yet--they haven't gotten past gap analysis. Internal auditing of a qualilty system should happen only after there is sufficient evidence to indicate that an efficacious system actually exists. At that point, auditing becomes a confirmation process, and not point-scoring witch hunt.

I’d love for my question to be whether any of you score internal audits of your own 9001:2000 QMS, but our new management LOVES metrics, and not scoring isn’t an option I’m afraid. Can anyone give me an objective method of scoring requirement elements in an internal audit?

:thanx:

Well, I am starting to try and trend the audit nonconformances by section. I suppose you could "score" from something like this? I deleted the actual nonconformances but left everything else there. Don't want to air all the dirty laundry. :o

The first three columns are there for sorting purposes and must be filled in to do just that. The "code" column is for our different divisions - so I can see at a glance where the problems exist. I try to briefly write the NC in an effort to see the same NC easier for trends.

For example, if you track the number of findings, percent of corrective action completed, which processes have the most findings etc. then you could determine if you should change your audit plan to make your audits more effective. With the information, you could determine which processes could have less audits and which processes need some more attention. I did that routinely but never called it a measurement system.

The real problem is when numbers have to be generated for the sole purpose of a monthly report or to justify the need for an audit system. If management is requiring either then they are missing the value of an viable audit system anyway.

Bill Pflanz I agree that numbers generation is not showing value, and I want to add my 2 cents why.

I really liked the scoring tool, yet there is a weak link in its accuracy. The score is dependent on the auditor's ability, inclination, available time and access to facts which result in findings that, with any variation in any of these performance influences, can result in different numbers of findings in any given area.

Providing there can be a consistent level and type of auditor performance, one day I want to bring in more than just a score, which communicates pass and fail like a test in school does but which is not the reason why audits should be done. We should audit to see how a system is performing well, or not. So I'd like to see more detail in reporting such as in categories: percent of process/document mismatch, percent interdepartmental dysfunction, percent not fully implemented or recorded...things like that.

When management decides to take an interest in how, and even why a system is performing well or not, we can move to that. Until then, it's pass or fail by degrees and objectivity really isn't apparent in the score. :2cents:

I guess I should clarify that I attend all audits (sometimes just as an observer/trainer). So I see how each audit goes, I see the evidence first-hand, etc. When we're done auditing, I compile the results and try my best to objectively score the audit. I am hard on us. If one thing is not to what we state and/or not to the ISO req't...it's marked down in the most appropriate column(s). My system was just to place a score on each section and each division for my own purposes. For me, it helps. I don't expect it to work for everyone. But I bet it could be altered to fit anywhere. It's a starting point. :)

I find the tool a very good starting point, especially when looking at a number of sites.

If it's okay, I might adapt it to show things like results of all the year's audits, scores by category, a count of elements covered (useful for process auditing) and an aggregate table that could provide meaningful management review metrics and help the audit team plan the next year's audits.

Sure--why not make it as negative as possible? The very idea of "scoring" an internal audit--or any audit for that matter--runs counter to what the whole idea should be, namely confirmation that the system is operating according to plan. If something must be counted, count things that go right.

Just yesterday I included in an audit report an NC related to something I found along this line. The "audit procedure" stated “The internal audit is a tool used to identify non-conformities on a scheduled basis”.

This is totally incorrect and the procedure didn't meet the requirement. The standard (I was doing 14K at the time, but 9 and 18 all read basically the same) states:
ISO 14001:2004, 4.5.5
The organization shall ensure that internal audits of the environmental management system are conducted at planned intervals to
a) determine whether the environmental management system
1) conforms to planned arrangements for environmental management including the requirements of this International Standard, and
2) has been properly implemented and is maintained,

The org had been using the IA NC's as a performance metric and consequently the real purpose of the audit was not being met.

I clearly tell folks in all of the audit courses I give (at least 1 or 2 a year;) ) that greater value is provided by focusing on conformance achievement, identifying NC's when they are found (but not looked "for") and specifically bringing out those OFI's (Preventive Actions).

Good KPI's might be:
* the percentage of the audit plan and schedule that was met by the auditors
* the number of OFI's found (value added activity here)
* percentage of conformance achieved (example -10 records reviewed 9 of which were correct = 90%) this turns a negative issue (10% NC) into a positive

There can be numerous others.

The key is to make the process positive and proactive even when the findings might be in reality different. This can help create at least a perception of the audit being a positive process.

Definitely report the negative but in a positive fashion...90% right as opposed to 10% wrong.

I find the tool a very good starting point, especially when looking at a number of sites.

If it's okay, I might adapt it to show things like results of all the year's audits, scores by category, a count of elements covered (useful for process auditing) and an aggregate table that could provide meaningful management review metrics and help the audit team plan the next year's audits.

Sure it's okay! Go for it and then share it!!!:bigwave:

I like this thread. My take would be to suggest the other processes get measured and one of those measurements is the number of audit findings against them...........MMMmmmmm. I wonder what management would think of that........:notme:

Andy

I’d love for my question to be whether any of you score internal audits of your own 9001:2000 QMS, but our new management LOVES metrics, and not scoring isn’t an option I’m afraid. Can anyone give me an objective method of scoring requirement elements in an internal audit?

:thanx:
We use the following metrics:
Number of audits sheduled vs done;
Number of audits overdue vs "catch up", ie done to catch up as auditor could have been sick or away on business
Number of positive comments raised;
Number of findings raised;
Number of findings open >60days after audit;
Number of unscheduled audits done (outside the formal audit);
These work really well for us - we have an open Management Review forum with all Top Management (flexible: if off site the meeting commences) that is chaired by the MD. This meeting includes all senior managers from all disciplines and middle managers. Mid-month we review the previous month's performance and then we include first line supervisors and Union Representatives.

The company I used to work for used a 1-point system to score their audits. Each sub-clause was worth 1-point. If that sub-clause had say 3 questions, each was weighted on its importance (e.g. Q1=.3, Q2=.4, Q3=.3). Then, what I suppose was a subjective score was assessed to each question based on objective evidence (e.g. Q1 - .3, Q2 -.3, Q3 -.25). So, the grand total for that element would be .85 out of 1 possible. Anything .7 or above was consider an Opportunity For Improvement, and anything below .7 was a Nonconformance. HOW the scores of findings were derived is what the mystery was to me.

Some great thoughts in this thread and I am learning a lot. THANKS!

Not sure if this is what you are after. I check several individual areas during an internal audit. Documents verify that the process was carried out or completed as required - based on the number sampled - I calculate an error percentage rate for each area .