How to manage Variation among Auditors?

Chennaiite

Never-say-die
Trusted Information Resource
#1
Dear Coves,

I belong to a team that is responsible for evaluating the Quality Management System of the Suppliers. Frankly speaking, variation does exist in our evaluation process. Some evaluators exaggerate the actual risk, some under-estimate, Some ignore, etc. We say, variation in the manufacturing process is inevitable, is it the same for evaluation process as well? More interestingly, i would like to understand how Certification bodies manage the Variation among their Lead Auditors?
Thanks.
 
Elsmar Forum Sponsor

Colin

Quite Involved in Discussions
#2
Re: How to manage the variation among auditors?

You raise a valuable point regarding variability of auditors - it does exist. As long as people are doing audits there will be differences in the way in which situations are interpreted and it does annoy companies that are audited when they get 2 different auditors with differing views on the same situation.

In a previous job I worked for a major 3rd party CB as the manager of the training division where part of my duties were to provide training to our full time and associate auditors to try and improve their performance and reduce the variability you mention.

My preferred way was to run workshops where we would ask the auditors to work in teams and interpret various scenarios with a view to making decisions regarding a potential N/C, Obs or nothing wrong. If it was an N/C we would ask them to write it up and then each group would present their findings to the other groups and explain why they raised (or didn't raise) what they found.

It was quite revealing when experienced auditors couldn't agree whether something was a major N/C down or nothing to report given the same information!

Another method is to mentor auditors, review their reports and witness their audits periodically but this takes time and effort which some organisations are not willing to do. I currently operate as an independent auditor and so I work with a few different CB's where I see a variation in the methods they use. Ironically, the smallest of the CB's has, in my opinion, the best methods of reviewing the performance of their auditors.
 
#3
Re: How to manage the variation among auditors?

Dear Coves,

I belong to a team that is responsible for evaluating the Quality Management System of the Suppliers. Frankly speaking, variation does exist in our evaluation process. Some evaluators exaggerate the actual risk, some under-estimate, Some ignore, etc. We say, variation in the manufacturing process is inevitable, is it the same for evaluation process as well? More interestingly, i would like to understand how Certification bodies manage the Variation among their Lead Auditors?
Thanks.
If you consider the analogy of an audit being a 'test' or 'inspection' of the qms (product) and the auditor the 'test equipment', then 'calibration' is what's needed to ensure repeatable results. This, in my experience (as a CB auditor) is done by ensuring that similar situations encountered during an audit are handled/observed and reported in a similar manner. Doing this at some frequency, say annually, just like calibration is often done. Obviously, some review of the normal test results is required, so any anomalies can be identified (which might be the result of an 'out of cal' situation!)

Periodic verification is also appropriate - a 'health check' of auditors (not doing push-ups etc:lol:) but maybe some case studies and review of answers can identify trends. In my experience of working with CBs both internally and externally, these can work well.

As mentioned by Colpart, mentoring works well.
 

bobdoering

Stop X-bar/R Madness!!
Trusted Information Resource
#4
Re: How to manage the variation among auditors?

There is a valid point to attempt to evaluate suppliers in a consistent manner - mostly to be fair and provide a consistent expectation from the supplier (common face as a customer). It is equally difficult to do as the variety of supplier systems, criticality to the customer process, etc. start to fold into the evaluation process. But, the judging may need to be codified in a manner similar to FMEA scoring. Work on your definitions of audit result interpretations until it works for you.

However, this point was brought up as an issue with internal audit issue at my workplace, and I mentioned that it may not be a good thing to try to force the process to be consistent. Each auditor has their own experiences - mostly the indelible ones from being burnt by some system deficiency - and the variety will collect a more robust vision of the system.
 

Helmut Jilling

Auditor / Consultant
#5
Re: How to manage the variation among auditors?

You raise a valuable point regarding variability of auditors - it does exist. As long as people are doing audits there will be differences in the way in which situations are interpreted and it does annoy companies that are audited when they get 2 different auditors with differing views on the same situation.

In a previous job I worked for a major 3rd party CB as the manager of the training division where part of my duties were to provide training to our full time and associate auditors to try and improve their performance and reduce the variability you mention.

My preferred way was to run workshops where we would ask the auditors to work in teams and interpret various scenarios with a view to making decisions regarding a potential N/C, Obs or nothing wrong. If it was an N/C we would ask them to write it up and then each group would present their findings to the other groups and explain why they raised (or didn't raise) what they found.

It was quite revealing when experienced auditors couldn't agree whether something was a major N/C down or nothing to report given the same information!

Another method is to mentor auditors, review their reports and witness their audits periodically but this takes time and effort which some organisations are not willing to do. I currently operate as an independent auditor and so I work with a few different CB's where I see a variation in the methods they use. Ironically, the smallest of the CB's has, in my opinion, the best methods of reviewing the performance of their auditors.
Very good observations. The root of variation with internal auditors often begins with inadequate or inconsistent training. I am so tired of seeing 2 hr internal training sessions, and somehow that is supposed to make them good auditors.

Then, the ongoing exercises such as Colpart mentions are very good to continue to calibrate and tune auditors' abilities.

Also, the internal review activities should judge how appropriate the audits are, and the value of the findings.

Audits should be a two-way street of discovery and learning. I have been auditing for 12 years and I am still continually learning.
 

Chennaiite

Never-say-die
Trusted Information Resource
#6
Re: How to manage the variation among auditors?

And hence, Good practice will also be like Auditors meeting frequently for discussing specific areas of QMS and making sure that interpretation prevails on common grounds.

But does use of Audit/Evaluation checklist make anything good in reducing the Variation? In our case, it seems to enhance the variation.
 
J

John Martinez

#7
Re: How to manage the variation among auditors?

My preferred way was to run workshops where we would ask the auditors to work in teams and interpret various scenarios with a view to making decisions regarding a potential N/C, Obs or nothing wrong. If it was an N/C we would ask them to write it up and then each group would present their findings to the other groups and explain why they raised (or didn't raise) what they found.

It was quite revealing when experienced auditors couldn't agree whether something was a major N/C down or nothing to report given the same information!
I agree. For internal audits or for supplier audits, I recomend that the audit team provide for adequate time to "calibrate" their responses. Such as, every team member reports on their findings and if applicable the classification of the findings. So if a finding is an NC, the auditor would read it, and explain their reasons. The team (under direction of an experienced lead auditor) would disuss and agree and disagree and their reasons why.

You would be amazed at how quickly this calibrates a group of auditors on subsequent audits.

Hope this helps.
 

Helmut Jilling

Auditor / Consultant
#8
Re: How to manage the variation among auditors?

I agree. For internal audits or for supplier audits, I recomend that the audit team provide for adequate time to "calibrate" their responses. Such as, every team member reports on their findings and if applicable the classification of the findings. So if a finding is an NC, the auditor would read it, and explain their reasons. The team (under direction of an experienced lead auditor) would disuss and agree and disagree and their reasons why.

You would be amazed at how quickly this calibrates a group of auditors on subsequent audits.

Hope this helps.
That is a good approach. But, bear in mind, generally, the audit team should report their findings to the Lead Auditor, and the Lead should ensure the findings that are written, have a single voice and tone. It is his task to calibrate the activity.
 
J

John Martinez

#9
Re: How to manage the variation among auditors?

That is a good approach. But, bear in mind, generally, the audit team should report their findings to the Lead Auditor, and the Lead should ensure the findings that are written, have a single voice and tone. It is his task to calibrate the activity.
I agree. Speaking to internal audits; however, in my suggestion above, the lead auditor is more of a facilitator and tries to lead the team to the correct response. Again, works great for companies that can get someone (even a consultant) that can meet will all team members to discuss the findings in the open and each and every team member holds each other accountable.
 
Thread starter Similar threads Forum Replies Date
Marco Bernardi Finding a flat or not modeling distribution: how to manage it? Capability, Accuracy and Stability - Processes, Machines, etc. 24
A How to manage the QMS system and SOP during the transition from MDD to MDR EU Medical Device Regulations 4
silentmonkey Seeking efficient method to manage install base data Manufacturing and Related Processes 0
J Microsoft Teams to manage Internal Audit System? Internal Auditing 3
L How to manage a Metrology Department or Calibration Laboratory General Measurement Device and Calibration Topics 3
N Who shall manage PPAP process? APQP and PPAP 4
B How do you manage your project? Data Conversion and Digitization Records and Data - Quality, Legal and Other Evidence 9
Q Is ISO 9001 fully enough to manage a business? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 19
D Any Elsmar members using Traqpath to manage CAPA? Nonconformance and Corrective Action 2
S Plant within a Plant - Does anyone currently manage their facility this way? AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 2
J IMDS: How to manage same Raw Material from Two Suppliers? RoHS, REACH, ELV, IMDS and Restricted Substances 2
Q Efficient way to manage different country Medical Device Regulations Other US Medical Device Regulations 10
Q How to manage with customer % failed product ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 4
L How do you manage your Inspection Drawings? IATF 16949 - Automotive Quality Systems Standard 9
Q Conflict Management - Systemic way to manage conflicts within Teams Book, Video, Blog and Web Site Reviews and Recommendations 6
S How to track and manage all uncertainty budgets - Database? Measurement Uncertainty (MU) 3
J Anyone here use Paradigm Sofware to manage compliance with ISO standards Quality Assurance and Compliance Software Tools and Solutions 3
J How can I manage Occurence Rates in FMEA efficiently when I have many processes? FMEA and Control Plans 9
T Best place in QMS to manage Quality Objectives ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 11
L How to manage prolongation of Calibration Period on Equipment General Measurement Device and Calibration Topics 9
drgnrider Definition of "Manage" as used in ISO 14001 related Documents ISO 14001:2015 Specific Discussions 3
G How to manage/control critical items and key characteristics? AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 1
B The best way to manage changes to the documentation of PCB files & BOM Inspection, Prints (Drawings), Testing, Sampling and Related Topics 8
M Looking for software to manage ISO 9001:2008 system Quality Assurance and Compliance Software Tools and Solutions 4
D Ticketing System to Manage Five Service Lines for proposed QMS ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
K Records Control - How do you Manage tons of Paper Records? Records and Data - Quality, Legal and Other Evidence 10
F How to manage Intermediary Sales Agents ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
Q What are the Product Realisation processes that manage Finances? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
R Suggestions for a practical way to manage Contract Review Contract Review Process 5
G How to manage a problem Overseas Tooling Supplier Supplier Quality Assurance and other Supplier Issues 7
T How to manage Suppliers in accordance with ISO 13485 Requirements Supplier Quality Assurance and other Supplier Issues 5
T Action plan: How to manage actions, preventive actions and opportunities of improv? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
M How to Manage Employees (Operators) - Training for specific Jobs Manufacturing and Related Processes 4
M How do you manage Customer Specific Requirements (CSRs)? IATF 16949 - Automotive Quality Systems Standard 6
V How do you manage the Review Comments post to Approval of a Document Document Control Systems, Procedures, Forms and Templates 3
C I asked myself a question 'What does a Management System Manage?' ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 22
G How do OEM's manage Special Customer Requirements from its End Customers? Customer and Company Specific Requirements 1
Sidney Vianna OASIS Modification To Manage Auditor Type (AEA/AA) For Each Authentication AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 1
W How to manage drugs during a manufacturing process? ISO 13485:2016 - Medical Device Quality Management Systems 5
E Software that will manage my APQP documents Quality Assurance and Compliance Software Tools and Solutions 9
S How to better manage files on a personal computer? After Work and Weekend Discussion Topics 8
P How to manage loose Pin Gages without Identification on the machining floor Calibration Frequency (Interval) 19
Marc What software (if any) do you use to manage your quality system? Quality Assurance and Compliance Software Tools and Solutions 57
A Competency Records Management - How do you manage your training/competency records? Training - Internal, External, Online and Distance Learning 1
K How to Manage / Drive your Internal Supplier Supplier Quality Assurance and other Supplier Issues 7
J Software or template to manage first piece, in process and final inspection reports Software Quality Assurance 2
S How to manage (control) a copy of ISO 9001:2008 Document Control Systems, Procedures, Forms and Templates 20
E Quality Software - Who uses software to manage your Quality System? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
M Experience with IPC 1752 compliance? Manage and report ROHS information RoHS, REACH, ELV, IMDS and Restricted Substances 4
D Procedure to Manage, Repair and Modify Tooling needed Document Control Systems, Procedures, Forms and Templates 2

Similar threads

Top Bottom