I am great at setting up and maintaining a QMS. Documentation, no problem. Tracking changes, easy stuff. Writing NCRs and CPARs, I'm a wizard at them. The problem comes when I have to evaluate the results of an internal audit. The person before me had this formula called an Internal Audit Index. 3 observations of the same non-conformity results in a finding. The obeservations count as 1/3 and a finding counts as 1. This was then divided by the number of opportunities for non-conformities. :confused: So the more opportunities there were the less "weight" a finding has. If there was 100 ops then 1 finding gave them an IAI of 99%. Not very objective and not very fair to other processes with fewer opportunities (# of opportunities were determined by the auditor). How do the rest of you evaluate your IAs? I need a more objective way of doing this. We have a SA coming up at the end of July and this issue needs to be resolved. Any help would be appreciated.

Nothing changes if nothing changes,
Qualitygal

Have you looked at this thread: Managing and Reporting Audits with an Excel .xls spreadsheet (http://elsmar.com/Forums/showthread.php?t=19223)? I created a system for evaluating internal audits in my organization and Jennifer Kirley has improved upon it greatly...

The person before me had this formula called an Internal Audit Index. 3 observations of the same non-conformity results in a finding. The obeservations count as 1/3 and a finding counts as 1. This was then divided by the number of opportunities for non-conformities. :confused: So the more opportunities there were the less "weight" a finding has. If there was 100 ops then 1 finding gave them an IAI of 99%. Not very objective and not very fair to other processes with fewer opportunities (# of opportunities were determined by the auditor).

Sheesh! Lord save me from ever having to work with a system like that :tg: And if the '# of opportunities' was determined by the auditor, I can't really see that was totally objective, either.

I'm not convinced that any kind of 'formula' would work well, or in fact be particularly useful unless perhaps in a fairly narrow band, say, where particular types of faults/NCFs occurred and were readily categorisable and quantifiable. I think there's a need for intelligent, experienced and skilled personnel to review here.

Would it be possible to identify a number of 'groups'/categories or the like to assign an audit, and define criteria for each of the categories?
Simplistic example: categories like say red/amber/green; an internal audit is deemed red if one or more of these criteria exist: 1 or more major nonconformances, failure to take timely action to correct previous amber, serious risk exists, etc etc...
Thus, assigning the status is based on the criteria, rather than personal judgement.

I'm not convinced that any kind of 'formula' would work well, or in fact be particularly useful unless perhaps in a fairly narrow band, say, where particular types of faults/NCFs occurred and were readily categorisable and quantifiable. I think there's a need for intelligent, experienced and skilled personnel to review here. :agree1: Some people mistakenly think that management systems can be treated as an exact science that can be explained, implemented and assessed by formulae, algorithms and equations. Management systems are behaviorally driven and impacted by the individual and collective dysfunctions present in the organization. The pursuit of precise, statistically sound audit protocols are a wasted effort, IMO. Subjectivity can not be filtered out of the process, as long as we are dealing with the human factor.

I'll add my agreement with the other replies. I don't think there is a formula that will adequately evaluate an audit. It requires applied management thinking to determine whether they are good or not.

However, if you define the criteria and measureables called for in clause 4.1, it can at least help you measure the performance.

The system you describe as your Internal Auditing Index really only skims the surface, and that is why it is difficult to set up any mathematical rating system for internal audits....for example:

Take two processes that get a IAI rating of 95% based on your system. Does this mean that both are performing to equal levels, that there are equal levels of risk associated with each? Not really, perhaps process #1 is an in-house finding which results in poor record keeping, where process #2 is a finding which is producing non-conforming work and subsequent rework with the risk of flowing down non-conforming product to the customer...how would you compare these two? Your numerical rating system would have to incorporate weighted values for different types of findings to really give you valid actionable results, not to mention taking into account past findings, repeat findings, etc. Does it consider that findings from one process may be flowing into the next process and thus producing findings there as well?

From my side of the fence the system you currently use has little value because it is not complex enough to effectively evaluate your results...what really can you do with these values you're getting aside from giving the process owner a grade which does not really take everything into account.

This is why it is simpler and more effective to utilize management review rather than attempt to develop a useful mathematical model to evaluate audit results. Can a valid model be produced...I'm certain it can, however it would take an investment of time and work and in the end you would still need to do management review of these results so why waste your efforts on this.

I'm curious, what are you using these IAI values for? Does anyone find the current system useful?

I am great at setting up and maintaining a QMS. Documentation, no problem. Tracking changes, easy stuff. Writing NCRs and CPARs, I'm a wizard at them. The problem comes when I have to evaluate the results of an internal audit. The person before me had this formula called an Internal Audit Index. 3 observations of the same non-conformity results in a finding. The obeservations count as 1/3 and a finding counts as 1. This was then divided by the number of opportunities for non-conformities. :confused: So the more opportunities there were the less "weight" a finding has. If there was 100 ops then 1 finding gave them an IAI of 99%. Not very objective and not very fair to other processes with fewer opportunities (# of opportunities were determined by the auditor). How do the rest of you evaluate your IAs? I need a more objective way of doing this. We have a SA coming up at the end of July and this issue needs to be resolved. Any help would be appreciated.

Nothing changes if nothing changes,
Qualitygal

I may be missing something here, but why exactly do you have to have a formula for evaluating an internal audit? Why can´t you just resume the audit with the # of findings (minor or major, observations or opportunities for improvement) and present that during your management meetings?

Crusader and Jennifer really did a good job on the spreadsheet :applause: if you want to grade your internal audits, you´ve got a great start with that spread sheet

I'm curious, what are you using these IAI values for? Does anyone find the current system useful?

I got this company certified in 90 days (5 minors) and a lot of the system was taken from other systems, companies, the internet etc. This was one that my former partner used (he was stuck in the 94 mindset) and no one seemed to have a problem with it before. All of us agree, including the certifying assessor :cool: that it adds no value, is too subjective and unfair. I agree that it is useless. I used to be a financial auditor and that is a lot different than ISO. One of our findings concerned our documentation and again in 6 months time I almost have to rebuild this system and one of the documents is 8.2.2. I'm called on ya'll for some guidance and I am glad I did. If I don't have to "grade" these things that is fantastic. We are having a meeting in about 15 minutes and I will present ya'lls thoughts and see where we want to go with this. I'll let you know how it works out.

Nothing changes if nothing changes,
Qualitygal

Formulas for reviewing audits? seems to be a lot of additional work to realize the same result.

What ever happened to reporting
- what was audited
- the results of the audits
- improvements made

Formulas for reviewing audits? seems to be a lot of additional work to realize the same result.

What ever happened to reporting
- what was audited
- the results of the audits
- improvements made

I agree with all of you and I will look at that spreadsheet as well. I knew it was the wrong way to evaluate the audits but I just needed validation of my thoughts. I have about 5 years experience in this field and sometimes I need to know from competent, more experienced people, that I'm on the right track.:thanx:

I agree with all of you and I will look at that spreadsheet as well. I knew it was the wrong way to evaluate the audits but I just needed validation of my thoughts. I have about 5 years experience in this field and sometimes I need to know from competent, more experienced people, that I'm on the right track.:thanx:


One of the 1st step in competence self development is understanding that you do not understand....you've progressed well beyond that.

Sidney said it all and there are some here in the Cove that have heard me say the exact same thing...management systems and auditing are not an exact science and they challenge quantification.

With the internal audit the bottom line is "are the desired and planned for results being effectively achieved?" Then you take it from there.

It's hard to determine an effective way of evaluating an audit process due to the number of processes a company may have & the objectivity of the auditors. I have found it best to evaluate the audit non-conformances vs. re-occurrences in management review. Also, internal audit results vs. external audit results is another way of evaluating the audit process.

qualitygal,

I've always stuck to and old method for scoring internal audit results: Pass or Fail. Hard to screw up a finding like that.

Al...

Hi everyone. I'm fairly new to ISO9001:2000 auditing. I have been given the task of carrying out internal auditing on the core processes for a logistics provider. We have a system implemented, in early stages which uses a weighted questionnaire for compliance which is supposed to show trends in continual improvement of the QMS but the person that has set this up has an ISO9000:1994 history. The questionnaire is carried out monthly and has a compliance rating of 80% and above to meet the standards.
Each department head uses this tool to assess their department for compliance (none have received adequate training in auditing to ISO standards).
The problem I have with using this questionnaire is that it firstly contain questions about processes which occur perhaps once per year due to nature of the business and there is no documented procedure to provide guidance on points awarded per question. Most of the questions appear to include customer,business and legal requirements but its possible for the process to meet the required standard of 80% and yet fail on a Health and Safety question. So there is the potential for areas to continue failing within the process and yet the overall result is always a pass

I have also recently completed a BSI Internal auditor course and was advised that this approach is not considered best practice as it does not get to the root cause of the problem.

How can I persuade my GM to change their approach. The focus is too much on individual performance rather than opportunities for improvement.

Appreciate your advice as I am feeling abit downhearted by the culture in my company

Welcome to The Cove, Elmo! :bigwave:

You said a mouthful with the part about culture. Devotion to quality, both internal and in suppliers, starts and ends with management's drive (or its lack) for controls--even inconvenient ones.

Let me see if I understand this right--this logistics provider is getting a separate questionnaire for each of its departments?

It's interesting that you include health and safety for this provider. I like the idea, but it's my opinion. Why is this item important to your supplier control?

I did a Google search of quality control for logistics suppliers and came up with these sites.

http://www.logisticsmgmt.com/article/CA6424067.html?ref=nbcs
http://www.logisticsmgmt.com/article/CA6365022.html
http://www.jistem.fea.usp.br/ojs/index.php/jistem/article/viewFile/66/66

Hi elmo28,

I'd go back to first principles: (that old Plan Do Check Act stuff)

* What's the aim - ie, what are you trying to achieve with these audits? Presumably the answers would include: ensuring we get what we want, mutually beneficial supplier relationships, continuous improvement, customer satisfaction, etc etc... but I'd really try to focus on bringing those into serious 'concreteness' - ie, instead of just saying 'customer satisfaction', if possible, make it specific, something like 'Customer Brown & Co receives their Goods (widgets, whatever) on time in Arkansas... '
* Then, review results. Are you (meaning you the organisation of course) achieving the results that are wanted? What's the evidence for that? Is there any? What are the facts, the data that support/disprove that?

Each department head uses this tool to assess their department for compliance (none have received adequate training in auditing to ISO standards).
Sounds like an improvement opportunity there!

The problem I have with using this questionnaire is that it firstly contain questions about processes which occur perhaps once per year due to nature of the business and there is no documented procedure to provide guidance on points awarded per question. Most of the questions appear to include customer,business and legal requirements but its possible for the process to meet the required standard of 80% and yet fail on a Health and Safety question. So there is the potential for areas to continue failing within the process and yet the overall result is always a pass

I'd have problems with that also. But then I also don't like these 'one size tools fit all' - people keep trying to find a 'totally objective and errorfree' tool for auditing, which I don't believe exists or is possible. Auditing is a skill, requiring experience, judgement and a whole lot of other attributes.

I have also recently completed a BSI Internal auditor course and was advised that this approach is not considered best practice as it does not get to the root cause of the problem.
No, it's not best practice. I'd hunt around to find as much data as you can to quote on how things have changed since the 1994 version, and where the focus is now - ie, find evidence to present to your GM.

How can I persuade my GM to change their approach. The focus is too much on individual performance rather than opportunities for improvement.

Someone in this forum (can't remember who) recommended an excellent book on organisational change called The Change Agents Handbook, a Survival Guide for Quality Improvement Champions by David Hutton. You can get it from Amazon, and it's seriously worth reading! Ignore what he says in it about ISO 9001 because that bit is now horribly out of date, but the rest of what he says is excellent. Lots of very handy and eminently practical advice.

I'm sure you'll get other useful input from here also.

Thanks for the links. We rely on the customer to dictate control measurs for suppliers of product.
We use a list of questions covering specific customer and business requirements which are different for each department based on controls, customer requirements, legal requirements. These questions have been drawn up based on detail in department procedures.
There are no guidelines for applying scores to each question so a health and safety requirement can be scored lower than another area of the process.

The attitude is moving toward covering up mistakes to gain points rather than acknowledging non conformance and analysing the root cause of the problem. Which is where my concern lies

There are no guidelines for applying scores to each question so a health and safety requirement can be scored lower than another area of the process.

The attitude is moving toward covering up mistakes to gain points rather than acknowledging non conformance and analysing the root cause of the problem. Which is where my concern lies

And you are absolutely, but absolutely, on the right track. (Let's start by being generous, and assume they are doing this because they don't really know better.)

You are right, so right. And I would be willing to bet that they hate filling it in almost as much (perhaps more!). Whereas if the focus was on the problem or failures themselves, and resolving them, then that's much more effective and productive.

Maybe you could break it down - instead of attempting a wholesale change at this point, can you take a 'bitesized chunk', and try a new approach with just one department or one area? Use that to get a win or two, which would add weight and precious data to use?

Thanks JaneB.

My GM gave reason for using set questions with scores to prove unbiased auditing approach yet he allows the head of department to audit their own area.

He suggested that in the absence of set questions it becomes the auditors opinion about the process and can be disputed by the customer (who incidentally have been sold this idea of using questionnaires as a basis for verifying the QMS is working)

Surely if the auditor is competent and plans objective, scope and criteria based on approved requirements i.e. ISO, customer and the law and reports onyl the facts there should be no dispute.

He also insists I photocopy all evidence of compliance and non compliance to prove I have audited something - is this really necessary?

Thanks JaneB.

My GM gave reason for using set questions with scores to prove unbiased auditing approach yet he allows the head of department to audit their own area.

He suggested that in the absence of set questions it becomes the auditors opinion about the process and can be disputed by the customer (who incidentally have been sold this idea of using questionnaires as a basis for verifying the QMS is working)

Surely if the auditor is competent and plans objective, scope and criteria based on approved requirements i.e. ISO, customer and the law and reports onyl the facts there should be no dispute.

He also insists I photocopy all evidence of compliance and non compliance to prove I have audited something - is this really necessary?Your GM is either trying to make matters easy or truly has no understanding regarding matters of oversight. An auditor should be basing a finding on known requirements and evidence of lack of compliance. Auditors are allowed to have opinions, but these show up as comments or efforts to work together to improve the supplier/buyer relations.

Sure, someone can dispute a finding. How messy that would be... We can guarantee there will never be a need for it if people always audit themselves!

There is a good reason why standards do not permit an organization's internal audits to be done by the process owners. Aside from the honesty question, an owner is among the least likely to recognize the shortcomings.

Gee, this sounds like a tricky situation. I think Jennifer has already given good advice, & I agree with her assessment. He certainly doesn't seem to know much at all about audit... but of course saying to one's GM 'you don't know anything' can be a damned hard thing and a CLM (career limiting move)!

That's why I urge you to find stuff you can use as objective evidence of 'best practice auditing', & perhaps present it as 'here's what the consensus of best practice says'... and then very delicately suggest room for improvement.



My GM gave reason for using set questions with scores to prove unbiased auditing approach

Yes, I can kind of see where he's coming from. If it were a case of asking the same questions/investigating the same things in each area (or whatever) then that enables consistency to be achieved. Where I disagree is with this notion of standardised 'scores'. You've already said that they don't achieve good results, so I'm having trouble understanding why one (he) would want to keep doing this!

[QUOTE=elmo28;198586]
yet he allows the head of department to audit their own area.

and therein lies the rub. It's a major no no. One simply cannot impartially, objectively audit one's own area of responsibility. I can't even spot my own typos (unless enough time has passed that I can look with fresh eyes).


He suggested that in the absence of set questions it becomes the auditors opinion about the process and can be disputed by the customer (who incidentally have been sold this idea of using questionnaires as a basis for verifying the QMS is working)
Is there room for compromise? Agree the set of questions or topics to be assessed - but gee golly gee, I'd certainly want an intelligent & trained auditor out there looking for risk, importance, etc etc.


Surely if the auditor is competent and plans objective, scope and criteria based on approved requirements i.e. ISO, customer and the law and reports only the facts there should be no dispute.
Wherever humans are involved there's room for dispute :-) but you're right, in that this should at least minimise it. It's one reason why reporting facts, focussing on objective evidence is so critically important, rather than relying on things like 'I think' or 'I feel' without any evidence.


He also insists I photocopy all evidence of compliance and non compliance to prove I have audited something - is this really necessary?
Again, he has a point to some extent.
It's reasonable to request to know what evidence was viewed (eg, in case of said dispute!). It's why auditors write - or should write - lots of notes.
If I audit & then report there's a weakness, the manager would have a reasonable expectation that you'd be able to report (if asked) how many records, say, it affected, and which ones.

So you should be able to look up your findings and say: here are the numbers of the Purchase Orders I viewed that were incorrect. I'd expect your report to indicate a % incorrect (eg, 5/50 or 10%, say). And I'd expect your report to indicate that you viewed, say, sample purchase orders over a 3-month period between Jan and March of 2007. THese kinds of things give the receiver of the audit report specific data - and make them less likely to dispute in future.


As for photocopying.... sounds like a lot of paper to me. I occasionally photocopy if it saves me wasting a lot of time writing. Or I take a 'screendump' and paste it into a file, and keep that. Either way, if someone comes back & queries how an auditor reached a conclusion, I consider - and teach this - that they have every right to know on what basis the auditor came to that conclusion.

By insisting on a set of audit questions (to avoid an auditor from being opinionated) and photocopies of all the evidence (to prove compliance/noncompliance and that the audit was done) I sense a lack of confidence in the auditor(s) and/or the audit process. It's reasonable to photocopy a representative sample of evidence, but I agree with Jane that doing it your GM's way sounds like a lot of paper.

But hey--maybe he needs a ream of an audit report to show such a method is a burden to him as well as the auditor. It's a lot of information to slog through where assembling a matrix would do, and all of that busy work takes time that could otherwise be used in another audit.

Seems this fellow has a controlling nature and/or does not believe his auditors have the company's best interests at heart. I have been in such an atmosphere and I found it very difficult.

Thanks JaneB and Jennifer. I took your advice and set up a meeting with my GM to explain the issues. For the time being we have reached a compromise with a scoring system but I am constructing process models with each of the process owners to ensure all areas of process are covered. I have somehow managed to work this into the questions to ensure all aspects are covered. The questions will be reviewed as and when process changes occur.

It's not ideal but a small step in the right direction. At least I have a healthy dialogue with my GM now. So much so in fact that he is considering funding the quality diploma to aid my development in the company.

He has agreed to review the situation with the scoring system in a few months so its all good.

Thanks for the sound advice. This Cove has been a great discovery!

Elmo, that was a really reassuring update.

To help validate the audit results, do specifically record lists of which documents were reviewed so they can be retrieved without trouble in case there are questions later. By making a clear audit trail, you can help build confidence in the process.

However, I do not list names of people I interview on the manufacturing floor unless it is for the reason of verifying their training records are up to date. In a small group there won't be any confusion about who was spoken with, but a certain anonymity can help build confidence in auditees too. To be effective, the audit process needs to be focused on the system. So the evidence you collect is designed to show the system is functioning as expected.

I'd like to add a note of caution about scoring: if you communicate audit results as those scores, you should be quite sure the people know just what those scores mean. The audit results are meant as a kind of communication. We need to make sure the language is understood by all or else the audit results will lose their meaning.

Another really cool thread! I love all this stuff about auditing and am amazed at the creativity of folks who come up with a scoring system and such like.

I haven't however seen anything which truley addresses the OP, at least, in title: How to evaluate internal audit findings......

My take is that you have to go back to the audit assignment before you can evaluate the audit results. Or, in other words you have to lknow what the goal was, if you want to know if you've arrived.

A bit like Crusader's post recently (I think it was) when the auditor didn't get the expected result, my experience is that most audit management tend to spend little to no time on planning the audit schedule, defining the scope and criteria, ensuring the auditor understands the assignment, has a good strategy, understands the task, the process, has amassed data relating to the performance of the process etc. etc. etc.

Without knowing this stuff, how would anyone make anything but a 'numerical value' of findings?? - which is, incidentally, a typical external audit reporting style.

I'm going to go right out on a limb and say that without a clear objective for the audit, knowing how the auditor is going to achieve that objective, you'll never accurately report audit findings.............

(This, also, is not what gets tuaght in current auditor training and isn't part of any training course criteria...........)