Internal Auditing & Assessing Effectiveness

L

Laura M

#1
One of the ways we looked at effectiveness was the number of N/C's our internal audits found on a given element, or more at, with a given procedural requirement. We looked if there was a "more effective" way of meeting the requirement. For example, if a department seemed to have trouble with maintaining the product identification procedure, was there a more "effective" way. Simpler ID methodology, or whatever.
The effectiveness often led to procedural changes that were suggested at Management Review meetings.

I don't know if your are QS, or ISO, but we had a PPAP procedure for internal parts. We were forever negligent on engineering change level requirements (parts would get changed, but documentation was delinquent.) We had a procedure, but it wasn't very "effective" in accomplishing the task. We brainstormed, streamlined, and developed a easier to manage system, that became "more effective."

I do not know if that fully meets the intentions of the standard, but those are 2 examples I remember that worked for our quality system "effectiveness" requirement.

Not sure what your auditor meant by the "scope of your audit providing the data", but may be leading to "systemic correcive action", and verification activities, not just taking care of a minor without looking whether it could happen elsewhere. To me that is more of a mgmt. review function rather than audit scope issue, but I may be misunderstanding when he/she meant.
 
Elsmar Forum Sponsor

Kevin Mader

One of THE Original Covers!
Staff member
Admin
#2
Dom,

Question 1:

Did the audit team flow chart the process? If you had, you may review areas within the process/system to determine where bottlenecks occur, deviations from the plan exist, or where redundacy exists within the program. How effective was the process/system during the audit? Was this noted in the Audit Report summary? Can you improve?

I would warn against counting deviations as a source to determining effectiveness. This practice is enumerative in nature and can be (is) very misleading. Better to show more objective sources of information.

Question 2:

What was the scope of the audit(s)? Was it defined in the Audit Report and that the audit team would be looking to determine Suitability, Compliance, and Effectiveness (a program can be suitable by meeting the guidelines of a standard but horribly inefficient, compliance demonstrated against this program, and fairly ineffective in achieving the desired result)? Does the program work but poorly or does the program work effectively? What measures are you using to judge this? Are results shared as part of Management Review?

Additionally, what other measures does your organization use to determine 'effectiveness'? Auditing is just one method, maybe not even the best way.

Regards,

Kevin
 
D

dominick

#3
My team has been reviewing to nonconformances and are struggling with our response. We conduct internal audits, document nonconformances, implement C/A, and review audits at Mgt. Review meetings. Our 3rd party auditor cited two minor nonconformances: 1)that our audits does not lead to a "system effectiveness" statement as an assesment of our quality system and 2) that it was not clear that the scope of our audit provides the data for the "...continuing suitability and effectiveness..." assurance required by 4.1.3. What is the real intent of these two requiremnts? I could legislate these items into our auditing procedures as a means of providing evidence (such as a summary statemnt at Mgt. Review...yes our system is effective) but it seems adds no value to our process and only adds to the procedure. How have others met these requirements. Thanks for the help.
 
A

Alan Cotterell

#4
I think the matter of effectiveness of the QMS relates to establishment of positive performance indicators and statistical control of processes.
 

Marc

Hunkered Down for the Duration
Staff member
Admin
#5
Originally posted by dominick:

1)that our audits does not lead to a "system effectiveness" statement as an assesment of our quality system
This is a hot potato issue as it is an interpretations issue. To address the issue all you have to do is be ready with an answer. Period. As long as there is an evaluation of effectiveness somewhere (management review meeting, for example) it's not up to the auditor to tell you how to come about this. I tell clients to ensure whoever presents the internal audit findings to make a statement on their opinion of the effectiveness and suitability of the systems. And it is an opinion. The presentation to management should have an idea of how the company operates as well as the systems. This is typically the quality manager. One example is internal audits. If I go in and find there are an increasing number of findings and the response to findings is slowing or very poor I will say the internal audit system is not evvective (although it may be suitable). If I see an increase in nonconformances in the nonconformance system I might consider looking at the corrective acion system to make a determination as to whether it is effective or not.

In summary, someone has to be ready to discuss what they did to determine effectiveness.
and 2) that it was not clear that the scope of our audit provides the data for the "...continuing suitability and effectiveness..." assurance required by 4.1.3.
As above, there are things to consider to determine effectiveness. Suitability is another matter - suitability is mostly opinion. Is the NC system suitable for the process and product? My opinion is..." and why.
What is the real intent of these two requiremnts?
To confuse the hell out of you...
I could legislate these items into our auditing procedures as a means of providing evidence (such as a summary statemnt at Mgt. Review...yes our NC system is effective) but it seems adds no value to our process and only adds to the procedure. How have others met these requirements. Thanks for the help.
Keep it out of the audit procedure. Let the internal auditors audit. Let the quality manager make these determinations.

For over 8 years I have been working with companies in implementing ISO and QS. The first meeting is where I explain that someone has to be ready to address every sentence in the standard. No one person has to be able and ready to explain everything, but someone better be ready to discuss each sentence. In this case it's typically the quality manager. It sounds to me like no one in your company took responsibility for these sentences. I drill my clients on every sentence - I want to have a name for every sentence. ISO9001:1994 4.4.7 Design Verification. Who will discuss this? OK - I'm the auditor - tell me how you meet this requirement.
 
A

Alan Cotterell

#6
I agree with Marc's comments about auditing to the standard, however I have often conducted internal audits in two stages. The first stage is to verify that policies comply with the standard and that procedures reflect the policies. The second is to test every clause in the procedure. I don't see how you can evaluate 'effectiveness' in this process except by evaluating process performance indicators.
 
A

Alan Cotterell

#7
Where there is a wholistic approach (integrated) towards Management Systems it may be possible to use a productivity indicator such as the ratio of profit to wages bill. An increase in this ratio would show a level of 'effectiveness' in implementing the management system.
 

Marc

Hunkered Down for the Duration
Staff member
Admin
#8
Allan: The difference between us is that I see divisions in responsibilities and you do not. You say YOU do this and such when you do internal audits. I believe I am also qualified to audit to the standard and to local procedures. I do not typically see internal auditors with enough training and/or experience to interpret the specification nor do I see most internal auditors as trained enough and/or with enough experience to be making decisions such as suitability or effectiveness of the procedures. Nor do I see it as their job. I see these as issues to be addressed by the quality manager - internal audits would be inputs to the manager's decision.

In your responses you say YOU do this or you do that - which is fine for you - but most internal auditors are line personnel, departmental managers and/or supervisors and like folk.

ISO (and QS for that matter) state: "4.17 Internal quality audits

...internal quality audits to verify whether quality activities and related results comply with planned arrangements and to determine the effectiveness of the quality system."

This does not require the actual internal auditor to make the interpretation of suitability or effectiveness. It does require them to be used as an input - which is where the quality manager comes in - s/he makes the evaluation, not the internal auditor. Unless, of course, the internal auditor is the quality manager.
 

Marc

Hunkered Down for the Duration
Staff member
Admin
#9
Originally posted by Alan Cotterell:

I don't see how you can evaluate 'effectiveness' in this process except by evaluating process performance indicators.
I don't see an issue here. As I stated above, internal audits are only 1 piece of the puzzle.
 
Thread starter Similar threads Forum Replies Date
L Internal Auditing & Assessing Effectiveness Internal Auditing 8
A What are the pros and cons of using an audit software for internal auditing? General Auditing Discussions 4
S Risk based internal auditing Internal Auditing 6
F AS9100D Internal auditing requirements Internal Auditing 3
R Does any here use an internal auditing tool that works on different platforms? Internal Auditing 3
W Does anyone have an API Q2 checklist for internal auditing? Oil and Gas Industry Standards and Regulations 1
blackholequasar Internal Auditing Inspiration - Getting volunteers to perform internal audits. Internal Auditing 22
W Internal Auditing carried out by a 3rd party - Review of previous audits AS9100, IAQG 9100, Nadcap and related Aerospace Standards and Requirements 3
S Internal Auditing for API Spec Q1 - auditor qualification requirements Oil and Gas Industry Standards and Regulations 6
E Informational Internal Audits - Wear multiple hats what can and can't I audit (so I'm not auditing my own work) Internal Auditing 144
N Online Internal Auditing Course for ISO 13485 - Suggestions ISO 13485:2016 - Medical Device Quality Management Systems 8
U Internal auditing - Company employees or contract second party Internal Auditing 10
K Internal Auditing - Umbrella QMS and Multiple Standards Oil and Gas Industry Standards and Regulations 4
D Auditing Our Outsourced 2nd-3rd Party Internal Audit Company ISO 13485:2016 - Medical Device Quality Management Systems 6
G AS9101 Rev F - Worksheets for internal auditing AS9100, IAQG 9100, Nadcap and related Aerospace Standards and Requirements 11
S ISO 13485:2016 and MDSAP internal auditing ISO 13485:2016 - Medical Device Quality Management Systems 6
S ISO 9001:2015 - Internal Auditing - Audit to the Standard? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 6
R Internal Auditor auditing Internal Audit Procedure (AS9100) Internal Auditing 18
M Is Automated Internal Auditing Possible? Internal Auditing 13
C Internal Auditing Requirements (ISO 9001:2008) Internal Auditing 3
L Auditing Design and Development in ISO 9001 (Internal Audit) Internal Auditing 1
sswaim Auditing Internal Laboratory Personnel for Competence General Auditing Discussions 4
K Internal Auditing a previous Nonconformance? Internal Auditing 19
P Recommended books on ISO 27001:2013 Implementation and Internal Auditing IEC 27001 - Information Security Management Systems (ISMS) 4
M Are auditing checklists required for Internal Audits? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 13
B ISO/TS16949 Internal Auditing - How many auditors? Internal Auditing 4
bgoers PFMEA, Internal Auditing, Corrective Action Training In Native Language (China) Training - Internal, External, Online and Distance Learning 1
Gman2 Internal Auditing Requirements before ISO 9001 Registration ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
S Auditing TS 16949 Cl. 7.6.3.1 - Internal Calibration Laboratory Requirements IATF 16949 - Automotive Quality Systems Standard 8
T ISO 9001 Internal Auditing Auditor Training in Amsterdam Training - Internal, External, Online and Distance Learning 1
S In an internal auditing desert and I'm the only one here.... AS9100, IAQG 9100, Nadcap and related Aerospace Standards and Requirements 15
G Internal Auditing in a Multi-Site Environment ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 17
B Internal Auditing of MDD and CMDR Requirements Other Medical Device Related Standards 5
O Internal Auditing in small Engineering company ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 6
T Looking for a webinar on Internal Auditing General Auditing Discussions 3
R On Auditing Internal Audit Process - How Independence can be Established Internal Auditing 4
D Auditing Abroad - Internal Audits of our European Sister Companies ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 16
R Auditing a process outside the realm of the formal Internal Audit ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
G New to Internal Auditing - Best questions to ask? Internal Auditing 17
M Auditing the Internal Audit Process - 8.2.2 General Auditing Discussions 2
J Auditing the Internal Auditing Process - Audit Nonconformance General Auditing Discussions 3
S Internal Auditing related to RISK Internal Auditing 9
W Internal Auditing - Observational Checklist for a Behavior Based System Internal Auditing 3
R Internal Auditing Checklist - Major NCR because the Checklist was not Completed Internal Auditing 17
V Depth of Internal Auditing and Training aspects in Research & Development (R & D) Internal Auditing 4
N Good Internal Auditing Training Courses Training - Internal, External, Online and Distance Learning 13
L Internal Auditing Reports / Documents - Design and Content Document Control Systems, Procedures, Forms and Templates 1
N Recommend internal auditing training 101 please (Tucson or Phoenix, AZ) Training - Internal, External, Online and Distance Learning 1
L Internal Auditing - How can I audit my QMS independently? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 12
C Internal Auditing - How to make it useful? Internal Auditing 36
Similar threads


















































Top Bottom