Evaluating Internal Audit Results

Randy

Super Moderator
#11
I agree with all of you and I will look at that spreadsheet as well. I knew it was the wrong way to evaluate the audits but I just needed validation of my thoughts. I have about 5 years experience in this field and sometimes I need to know from competent, more experienced people, that I'm on the right track.:thanx:

One of the 1st step in competence self development is understanding that you do not understand....you've progressed well beyond that.

Sidney said it all and there are some here in the Cove that have heard me say the exact same thing...management systems and auditing are not an exact science and they challenge quantification.

With the internal audit the bottom line is "are the desired and planned for results being effectively achieved?" Then you take it from there.
 
Elsmar Forum Sponsor
L

LoriPLTZ

#12
It's hard to determine an effective way of evaluating an audit process due to the number of processes a company may have & the objectivity of the auditors. I have found it best to evaluate the audit non-conformances vs. re-occurrences in management review. Also, internal audit results vs. external audit results is another way of evaluating the audit process.
 
A

Al Dyer

#13
qualitygal,

I've always stuck to and old method for scoring internal audit results: Pass or Fail. Hard to screw up a finding like that.

Al...
 
E

elmo28

#14
Hi everyone. I'm fairly new to ISO9001:2000 auditing. I have been given the task of carrying out internal auditing on the core processes for a logistics provider. We have a system implemented, in early stages which uses a weighted questionnaire for compliance which is supposed to show trends in continual improvement of the QMS but the person that has set this up has an ISO9000:1994 history. The questionnaire is carried out monthly and has a compliance rating of 80% and above to meet the standards.
Each department head uses this tool to assess their department for compliance (none have received adequate training in auditing to ISO standards).
The problem I have with using this questionnaire is that it firstly contain questions about processes which occur perhaps once per year due to nature of the business and there is no documented procedure to provide guidance on points awarded per question. Most of the questions appear to include customer,business and legal requirements but its possible for the process to meet the required standard of 80% and yet fail on a Health and Safety question. So there is the potential for areas to continue failing within the process and yet the overall result is always a pass

I have also recently completed a BSI Internal auditor course and was advised that this approach is not considered best practice as it does not get to the root cause of the problem.

How can I persuade my GM to change their approach. The focus is too much on individual performance rather than opportunities for improvement.

Appreciate your advice as I am feeling abit downhearted by the culture in my company
 

Jen Kirley

Quality and Auditing Expert
Staff member
Admin
#15
Welcome to The Cove, Elmo! :bigwave:

You said a mouthful with the part about culture. Devotion to quality, both internal and in suppliers, starts and ends with management's drive (or its lack) for controls--even inconvenient ones.

Let me see if I understand this right--this logistics provider is getting a separate questionnaire for each of its departments?

It's interesting that you include health and safety for this provider. I like the idea, but it's my opinion. Why is this item important to your supplier control?

I did a Google search of quality control for logistics suppliers and came up with these sites.

http://www.logisticsmgmt.com/article/CA6424067.html?ref=nbcs
http://www.logisticsmgmt.com/article/CA6365022.html
http://www.jistem.fea.usp.br/ojs/index.php/jistem/article/viewFile/66/66
 
J

JaneB

#16
Hi elmo28,

I'd go back to first principles: (that old Plan Do Check Act stuff)

* What's the aim - ie, what are you trying to achieve with these audits? Presumably the answers would include: ensuring we get what we want, mutually beneficial supplier relationships, continuous improvement, customer satisfaction, etc etc... but I'd really try to focus on bringing those into serious 'concreteness' - ie, instead of just saying 'customer satisfaction', if possible, make it specific, something like 'Customer Brown & Co receives their Goods (widgets, whatever) on time in Arkansas... '
* Then, review results. Are you (meaning you the organisation of course) achieving the results that are wanted? What's the evidence for that? Is there any? What are the facts, the data that support/disprove that?

Each department head uses this tool to assess their department for compliance (none have received adequate training in auditing to ISO standards).
Sounds like an improvement opportunity there!

The problem I have with using this questionnaire is that it firstly contain questions about processes which occur perhaps once per year due to nature of the business and there is no documented procedure to provide guidance on points awarded per question. Most of the questions appear to include customer,business and legal requirements but its possible for the process to meet the required standard of 80% and yet fail on a Health and Safety question. So there is the potential for areas to continue failing within the process and yet the overall result is always a pass
I'd have problems with that also. But then I also don't like these 'one size tools fit all' - people keep trying to find a 'totally objective and errorfree' tool for auditing, which I don't believe exists or is possible. Auditing is a skill, requiring experience, judgement and a whole lot of other attributes.

I have also recently completed a BSI Internal auditor course and was advised that this approach is not considered best practice as it does not get to the root cause of the problem.
No, it's not best practice. I'd hunt around to find as much data as you can to quote on how things have changed since the 1994 version, and where the focus is now - ie, find evidence to present to your GM.

How can I persuade my GM to change their approach. The focus is too much on individual performance rather than opportunities for improvement.
Someone in this forum (can't remember who) recommended an excellent book on organisational change called The Change Agents Handbook, a Survival Guide for Quality Improvement Champions by David Hutton. You can get it from Amazon, and it's seriously worth reading! Ignore what he says in it about ISO 9001 because that bit is now horribly out of date, but the rest of what he says is excellent. Lots of very handy and eminently practical advice.

I'm sure you'll get other useful input from here also.
 
E

elmo28

#17
Thanks for the links. We rely on the customer to dictate control measurs for suppliers of product.
We use a list of questions covering specific customer and business requirements which are different for each department based on controls, customer requirements, legal requirements. These questions have been drawn up based on detail in department procedures.
There are no guidelines for applying scores to each question so a health and safety requirement can be scored lower than another area of the process.

The attitude is moving toward covering up mistakes to gain points rather than acknowledging non conformance and analysing the root cause of the problem. Which is where my concern lies
 
J

JaneB

#18
There are no guidelines for applying scores to each question so a health and safety requirement can be scored lower than another area of the process.

The attitude is moving toward covering up mistakes to gain points rather than acknowledging non conformance and analysing the root cause of the problem. Which is where my concern lies
And you are absolutely, but absolutely, on the right track. (Let's start by being generous, and assume they are doing this because they don't really know better.)

You are right, so right. And I would be willing to bet that they hate filling it in almost as much (perhaps more!). Whereas if the focus was on the problem or failures themselves, and resolving them, then that's much more effective and productive.

Maybe you could break it down - instead of attempting a wholesale change at this point, can you take a 'bitesized chunk', and try a new approach with just one department or one area? Use that to get a win or two, which would add weight and precious data to use?
 
E

elmo28

#19
Thanks JaneB.

My GM gave reason for using set questions with scores to prove unbiased auditing approach yet he allows the head of department to audit their own area.

He suggested that in the absence of set questions it becomes the auditors opinion about the process and can be disputed by the customer (who incidentally have been sold this idea of using questionnaires as a basis for verifying the QMS is working)

Surely if the auditor is competent and plans objective, scope and criteria based on approved requirements i.e. ISO, customer and the law and reports onyl the facts there should be no dispute.

He also insists I photocopy all evidence of compliance and non compliance to prove I have audited something - is this really necessary?
 

Jen Kirley

Quality and Auditing Expert
Staff member
Admin
#20
Thanks JaneB.

My GM gave reason for using set questions with scores to prove unbiased auditing approach yet he allows the head of department to audit their own area.

He suggested that in the absence of set questions it becomes the auditors opinion about the process and can be disputed by the customer (who incidentally have been sold this idea of using questionnaires as a basis for verifying the QMS is working)

Surely if the auditor is competent and plans objective, scope and criteria based on approved requirements i.e. ISO, customer and the law and reports onyl the facts there should be no dispute.

He also insists I photocopy all evidence of compliance and non compliance to prove I have audited something - is this really necessary?
Your GM is either trying to make matters easy or truly has no understanding regarding matters of oversight. An auditor should be basing a finding on known requirements and evidence of lack of compliance. Auditors are allowed to have opinions, but these show up as comments or efforts to work together to improve the supplier/buyer relations.

Sure, someone can dispute a finding. How messy that would be... We can guarantee there will never be a need for it if people always audit themselves!

There is a good reason why standards do not permit an organization's internal audits to be done by the process owners. Aside from the honesty question, an owner is among the least likely to recognize the shortcomings.
 
Thread starter Similar threads Forum Replies Date
C Evaluating the Effectiveness of Internal Audit Process Internal Auditing 16
Tagin Evaluating nonconformances for escalation using Bayesian methods? Statistical Analysis Tools, Techniques and SPC 2
R Evaluating the need for preventive action Preventive Action and Continuous Improvement 3
M Informational US FDA – FDA provides updates on the agency’s continued commitment to evaluating postmarket safety of Essure device Medical Device and FDA Regulations and Standards News 0
S Evaluating the possibilities of using qmswrapper or greenlightguru Document Control Systems, Procedures, Forms and Templates 4
C Evaluating GRR Scores %Tol vs %TV Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 9
Q Evaluating Effectiveness of a Preventive Action after Closure Preventive Action and Continuous Improvement 7
J Potential Customers Asking for EIRs / 483s when evaluating a CRO for future work US Food and Drug Administration (FDA) 2
P FDA - Evaluating Substantial Equivalence in Premarket Notifications 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 1
M Quality Control Procedure and Procedure for Evaluating Sub Contractor Document Control Systems, Procedures, Forms and Templates 1
M Evaluating Vendor (Supplier) Products using SQC Statistical Analysis Tools, Techniques and SPC 6
K Evaluating different Startup Options in EU and MDD ISO 13485:2016 - Medical Device Quality Management Systems 4
S Trading Company - Evaluating Supplier Performance and Approved Supplier List ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
J The 510(k) Program: Evaluating Substantial Equivalence in Premarket Notifications 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 2
E Evaluating Lean Knowledge: Lean Philosophy Course for Production Engineering Students Lean in Manufacturing and Service Industries 1
A Evaluating an Integrated Management System - Dissertation Topic ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
optomist1 Evaluating Capability Indices with Minitab Using Minitab Software 7
C Trend Analysis - Evaluating Quality of Pricing - ANVOA or Std Dev? Statistical Analysis Tools, Techniques and SPC 25
W Re-Evaluating Corrective Actions - Recurrence of a Customer Complaint Nonconformance and Corrective Action 7
D Good source for Checking and Evaluating Control Plans needed FMEA and Control Plans 4
R EMP (Evaluating the Measurement Process) Studies for Bias Effect Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 6
J Evaluating a Regression Model Using the Constant Variance Assumption Six Sigma 8
Wes Bucey "Downsizing" ramifications - Evaluating and Interpreting the News World News 0
A Methods of evaluating compliance to ISO 14001 ISO 14001:2015 Specific Discussions 6
M Evaluating my QMS (Quality Management System) ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 11
G 100% Sort Verification - Statistical Method for Evaluating Suspect Material Statistical Analysis Tools, Techniques and SPC 19
J Evaluating and Reporting the Results of Layered Process Audits Document Control Systems, Procedures, Forms and Templates 9
G Evaluating Software for GMP Criticality? Software Quality Assurance 3
M Evaluating adverse effect on device tested with out of tolerance instrument General Measurement Device and Calibration Topics 3
P Rockwell, Brinell, Vickers Conversion - Evaluating Aluminum 6061-T6 and 6063-T6 General Measurement Device and Calibration Topics 11
U Evaluating the aspects... All judgement based or is there a more objective way? Miscellaneous Environmental Standards and EMS Related Discussions 1
D Evaluating competence on training for new ISO 13485 standard for employees ISO 13485:2016 - Medical Device Quality Management Systems 7
Q "Service" Vendor (Supplier) Evaluation - How are you evaluating suppliers? Supplier Quality Assurance and other Supplier Issues 5
N Evaluating Process Stability - Unstable Processes - Cpk/Ppk? Capability, Accuracy and Stability - Processes, Machines, etc. 6
Z Seeking GM standard GM9684P - Procedure for evaluating parting lines Customer and Company Specific Requirements 2
S Metrics for Evaluating Design process Design and Development of Products and Processes 2
C Evaluating software that measures Ppk, Cmk, Cpk, PPI, etc. Capability, Accuracy and Stability - Processes, Machines, etc. 4
H Good starting point for a new QC Manager? Evaluating Existing Procedures Document Control Systems, Procedures, Forms and Templates 21
Gman2 Evaluating training, needed for office personnel? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 18
G Evaluating Processes - Cycle Time Concepts - Three Levels for any process Manufacturing and Related Processes 3
Marc Evaluating 5.3 c - Quality Policy - Framework General Auditing Discussions 1
T Evaluating Customer Satisfaction ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 14
B Risk Anlysis for Evaluating then Voiding or Not Issuing CPAR's Nonconformance and Corrective Action 1
M ISO 9001:2000 - Evaluating Training Effectiveness and Why ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 12
C Can anyone suggest a means of evaluating a suppliers performance? Supplier Quality Assurance and other Supplier Issues 4
Raffy Evaluation Form - Re-evaluating suppliers for their performance Supplier Quality Assurance and other Supplier Issues 8
G Evaluating Training Effectiveness - 6.2.2 Competence, Awareness and Training ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 55
S Supplier Evaluation - Evaluating transport suppliers and equipment suppliers Supplier Quality Assurance and other Supplier Issues 2
D What exactly is needed on the subcontractor list for evaluating subcontractors QS-9000 - American Automotive Manufacturers Standard 4
D Evaluating Subcontractor On-time Delivery Misc. Quality Assurance and Business Systems Related Topics 3

Similar threads

Top Bottom