Internal Auditing & Assessing Effectiveness

[Laura M]

One of the ways we looked at effectiveness was the number of N/C's our internal audits found on a given element, or more at, with a given procedural requirement. We looked if there was a "more effective" way of meeting the requirement. For example, if a department seemed to have trouble with maintaining the product identification procedure, was there a more "effective" way. Simpler ID methodology, or whatever.
The effectiveness often led to procedural changes that were suggested at Management Review meetings.

I don't know if your are QS, or ISO, but we had a PPAP procedure for internal parts. We were forever negligent on engineering change level requirements (parts would get changed, but documentation was delinquent.) We had a procedure, but it wasn't very "effective" in accomplishing the task. We brainstormed, streamlined, and developed a easier to manage system, that became "more effective."

I do not know if that fully meets the intentions of the standard, but those are 2 examples I remember that worked for our quality system "effectiveness" requirement.

Not sure what your auditor meant by the "scope of your audit providing the data", but may be leading to "systemic correcive action", and verification activities, not just taking care of a minor without looking whether it could happen elsewhere. To me that is more of a mgmt. review function rather than audit scope issue, but I may be misunderstanding when he/she meant.

[Kevin Mader]

Dom,

Question 1:

Did the audit team flow chart the process? If you had, you may review areas within the process/system to determine where bottlenecks occur, deviations from the plan exist, or where redundacy exists within the program. How effective was the process/system during the audit? Was this noted in the Audit Report summary? Can you improve?

I would warn against counting deviations as a source to determining effectiveness. This practice is enumerative in nature and can be (is) very misleading. Better to show more objective sources of information.

Question 2:

What was the scope of the audit(s)? Was it defined in the Audit Report and that the audit team would be looking to determine Suitability, Compliance, and Effectiveness (a program can be suitable by meeting the guidelines of a standard but horribly inefficient, compliance demonstrated against this program, and fairly ineffective in achieving the desired result)? Does the program work but poorly or does the program work effectively? What measures are you using to judge this? Are results shared as part of Management Review?

Additionally, what other measures does your organization use to determine 'effectiveness'? Auditing is just one method, maybe not even the best way.

Regards,

Kevin

My team has been reviewing to nonconformances and are struggling with our response. We conduct internal audits, document nonconformances, implement C/A, and review audits at Mgt. Review meetings. Our 3rd party auditor cited two minor nonconformances: 1)that our audits does not lead to a "system effectiveness" statement as an assesment of our quality system and 2) that it was not clear that the scope of our audit provides the data for the "...continuing suitability and effectiveness..." assurance required by 4.1.3. What is the real intent of these two requiremnts? I could legislate these items into our auditing procedures as a means of providing evidence (such as a summary statemnt at Mgt. Review...yes our system is effective) but it seems adds no value to our process and only adds to the procedure. How have others met these requirements. Thanks for the help.

I think the matter of effectiveness of the QMS relates to establishment of positive performance indicators and statistical control of processes.

[Marc]

Originally posted by dominick:

1)that our audits does not lead to a "system effectiveness" statement as an assesment of our quality system

This is a hot potato issue as it is an interpretations issue. To address the issue all you have to do is be ready with an answer. Period. As long as there is an evaluation of effectiveness somewhere (management review meeting, for example) it's not up to the auditor to tell you how to come about this. I tell clients to ensure whoever presents the internal audit findings to make a statement on their opinion of the effectiveness and suitability of the systems. And it is an opinion. The presentation to management should have an idea of how the company operates as well as the systems. This is typically the quality manager. One example is internal audits. If I go in and find there are an increasing number of findings and the response to findings is slowing or very poor I will say the internal audit system is not evvective (although it may be suitable). If I see an increase in nonconformances in the nonconformance system I might consider looking at the corrective acion system to make a determination as to whether it is effective or not.

In summary, someone has to be ready to discuss what they did to determine effectiveness.

and 2) that it was not clear that the scope of our audit provides the data for the "...continuing suitability and effectiveness..." assurance required by 4.1.3.

As above, there are things to consider to determine effectiveness. Suitability is another matter - suitability is mostly opinion. Is the NC system suitable for the process and product? My opinion is..." and why.

What is the real intent of these two requiremnts?

To confuse the hell out of you...

I could legislate these items into our auditing procedures as a means of providing evidence (such as a summary statemnt at Mgt. Review...yes our NC system is effective) but it seems adds no value to our process and only adds to the procedure. How have others met these requirements. Thanks for the help.

Keep it out of the audit procedure. Let the internal auditors audit. Let the quality manager make these determinations.

For over 8 years I have been working with companies in implementing ISO and QS. The first meeting is where I explain that someone has to be ready to address every sentence in the standard. No one person has to be able and ready to explain everything, but someone better be ready to discuss each sentence. In this case it's typically the quality manager. It sounds to me like no one in your company took responsibility for these sentences. I drill my clients on every sentence - I want to have a name for every sentence. ISO9001:1994 4.4.7 Design Verification. Who will discuss this? OK - I'm the auditor - tell me how you meet this requirement.

I agree with Marc's comments about auditing to the standard, however I have often conducted internal audits in two stages. The first stage is to verify that policies comply with the standard and that procedures reflect the policies. The second is to test every clause in the procedure. I don't see how you can evaluate 'effectiveness' in this process except by evaluating process performance indicators.

Where there is a wholistic approach (integrated) towards Management Systems it may be possible to use a productivity indicator such as the ratio of profit to wages bill. An increase in this ratio would show a level of 'effectiveness' in implementing the management system.

[Marc]

Allan: The difference between us is that I see divisions in responsibilities and you do not. You say YOU do this and such when you do internal audits. I believe I am also qualified to audit to the standard and to local procedures. I do not typically see internal auditors with enough training and/or experience to interpret the specification nor do I see most internal auditors as trained enough and/or with enough experience to be making decisions such as suitability or effectiveness of the procedures. Nor do I see it as their job. I see these as issues to be addressed by the quality manager - internal audits would be inputs to the manager's decision.

In your responses you say YOU do this or you do that - which is fine for you - but most internal auditors are line personnel, departmental managers and/or supervisors and like folk.

ISO (and QS for that matter) state: "4.17 Internal quality audits

...internal quality audits to verify whether quality activities and related results comply with planned arrangements and to determine the effectiveness of the quality system."

This does not require the actual internal auditor to make the interpretation of suitability or effectiveness. It does require them to be used as an input - which is where the quality manager comes in - s/he makes the evaluation, not the internal auditor. Unless, of course, the internal auditor is the quality manager.