Evaluating Internal Audit Results

Q

qualitygal

#1
I am great at setting up and maintaining a QMS. Documentation, no problem. Tracking changes, easy stuff. Writing NCRs and CPARs, I'm a wizard at them. The problem comes when I have to evaluate the results of an internal audit. The person before me had this formula called an Internal Audit Index. 3 observations of the same non-conformity results in a finding. The obeservations count as 1/3 and a finding counts as 1. This was then divided by the number of opportunities for non-conformities. :confused: So the more opportunities there were the less "weight" a finding has. If there was 100 ops then 1 finding gave them an IAI of 99%. Not very objective and not very fair to other processes with fewer opportunities (# of opportunities were determined by the auditor). How do the rest of you evaluate your IAs? I need a more objective way of doing this. We have a SA coming up at the end of July and this issue needs to be resolved. Any help would be appreciated.

Nothing changes if nothing changes,
Qualitygal
 
Elsmar Forum Sponsor
J

JaneB

#3
The person before me had this formula called an Internal Audit Index. 3 observations of the same non-conformity results in a finding. The obeservations count as 1/3 and a finding counts as 1. This was then divided by the number of opportunities for non-conformities. :confused: So the more opportunities there were the less "weight" a finding has. If there was 100 ops then 1 finding gave them an IAI of 99%. Not very objective and not very fair to other processes with fewer opportunities (# of opportunities were determined by the auditor).
Sheesh! Lord save me from ever having to work with a system like that :tg: And if the '# of opportunities' was determined by the auditor, I can't really see that was totally objective, either.

I'm not convinced that any kind of 'formula' would work well, or in fact be particularly useful unless perhaps in a fairly narrow band, say, where particular types of faults/NCFs occurred and were readily categorisable and quantifiable. I think there's a need for intelligent, experienced and skilled personnel to review here.

Would it be possible to identify a number of 'groups'/categories or the like to assign an audit, and define criteria for each of the categories?
Simplistic example: categories like say red/amber/green; an internal audit is deemed red if one or more of these criteria exist: 1 or more major nonconformances, failure to take timely action to correct previous amber, serious risk exists, etc etc...
Thus, assigning the status is based on the criteria, rather than personal judgement.
 

Sidney Vianna

Post Responsibly
Staff member
Admin
#4
I'm not convinced that any kind of 'formula' would work well, or in fact be particularly useful unless perhaps in a fairly narrow band, say, where particular types of faults/NCFs occurred and were readily categorisable and quantifiable. I think there's a need for intelligent, experienced and skilled personnel to review here.
:agree1: Some people mistakenly think that management systems can be treated as an exact science that can be explained, implemented and assessed by formulae, algorithms and equations. Management systems are behaviorally driven and impacted by the individual and collective dysfunctions present in the organization. The pursuit of precise, statistically sound audit protocols are a wasted effort, IMO. Subjectivity can not be filtered out of the process, as long as we are dealing with the human factor.
 
Last edited:

Helmut Jilling

Auditor / Consultant
#5
I'll add my agreement with the other replies. I don't think there is a formula that will adequately evaluate an audit. It requires applied management thinking to determine whether they are good or not.

However, if you define the criteria and measureables called for in clause 4.1, it can at least help you measure the performance.
 
B

Benjamin28

#6
The system you describe as your Internal Auditing Index really only skims the surface, and that is why it is difficult to set up any mathematical rating system for internal audits....for example:

Take two processes that get a IAI rating of 95% based on your system. Does this mean that both are performing to equal levels, that there are equal levels of risk associated with each? Not really, perhaps process #1 is an in-house finding which results in poor record keeping, where process #2 is a finding which is producing non-conforming work and subsequent rework with the risk of flowing down non-conforming product to the customer...how would you compare these two? Your numerical rating system would have to incorporate weighted values for different types of findings to really give you valid actionable results, not to mention taking into account past findings, repeat findings, etc. Does it consider that findings from one process may be flowing into the next process and thus producing findings there as well?

From my side of the fence the system you currently use has little value because it is not complex enough to effectively evaluate your results...what really can you do with these values you're getting aside from giving the process owner a grade which does not really take everything into account.

This is why it is simpler and more effective to utilize management review rather than attempt to develop a useful mathematical model to evaluate audit results. Can a valid model be produced...I'm certain it can, however it would take an investment of time and work and in the end you would still need to do management review of these results so why waste your efforts on this.

I'm curious, what are you using these IAI values for? Does anyone find the current system useful?
 
T

Ted Schmitt

#7
I am great at setting up and maintaining a QMS. Documentation, no problem. Tracking changes, easy stuff. Writing NCRs and CPARs, I'm a wizard at them. The problem comes when I have to evaluate the results of an internal audit. The person before me had this formula called an Internal Audit Index. 3 observations of the same non-conformity results in a finding. The obeservations count as 1/3 and a finding counts as 1. This was then divided by the number of opportunities for non-conformities. :confused: So the more opportunities there were the less "weight" a finding has. If there was 100 ops then 1 finding gave them an IAI of 99%. Not very objective and not very fair to other processes with fewer opportunities (# of opportunities were determined by the auditor). How do the rest of you evaluate your IAs? I need a more objective way of doing this. We have a SA coming up at the end of July and this issue needs to be resolved. Any help would be appreciated.

Nothing changes if nothing changes,
Qualitygal
I may be missing something here, but why exactly do you have to have a formula for evaluating an internal audit? Why can´t you just resume the audit with the # of findings (minor or major, observations or opportunities for improvement) and present that during your management meetings?

Crusader and Jennifer really did a good job on the spreadsheet :applause: if you want to grade your internal audits, you´ve got a great start with that spread sheet
 
Q

qualitygal

#8
I'm curious, what are you using these IAI values for? Does anyone find the current system useful?
I got this company certified in 90 days (5 minors) and a lot of the system was taken from other systems, companies, the internet etc. This was one that my former partner used (he was stuck in the 94 mindset) and no one seemed to have a problem with it before. All of us agree, including the certifying assessor :cool: that it adds no value, is too subjective and unfair. I agree that it is useless. I used to be a financial auditor and that is a lot different than ISO. One of our findings concerned our documentation and again in 6 months time I almost have to rebuild this system and one of the documents is 8.2.2. I'm called on ya'll for some guidance and I am glad I did. If I don't have to "grade" these things that is fantastic. We are having a meeting in about 15 minutes and I will present ya'lls thoughts and see where we want to go with this. I'll let you know how it works out.

Nothing changes if nothing changes,
Qualitygal
 
J

jem63

#9
Formulas for reviewing audits? seems to be a lot of additional work to realize the same result.

What ever happened to reporting
- what was audited
- the results of the audits
- improvements made
 
Q

qualitygal

#10
Formulas for reviewing audits? seems to be a lot of additional work to realize the same result.

What ever happened to reporting
- what was audited
- the results of the audits
- improvements made
I agree with all of you and I will look at that spreadsheet as well. I knew it was the wrong way to evaluate the audits but I just needed validation of my thoughts. I have about 5 years experience in this field and sometimes I need to know from competent, more experienced people, that I'm on the right track.:thanx:
 
Thread starter Similar threads Forum Replies Date
C Evaluating the Effectiveness of Internal Audit Process Internal Auditing 16
Tagin Evaluating nonconformances for escalation using Bayesian methods? Statistical Analysis Tools, Techniques and SPC 2
R Evaluating the need for preventive action Preventive Action and Continuous Improvement 3
M Informational US FDA – FDA provides updates on the agency’s continued commitment to evaluating postmarket safety of Essure device Medical Device and FDA Regulations and Standards News 0
S Evaluating the possibilities of using qmswrapper or greenlightguru Document Control Systems, Procedures, Forms and Templates 4
C Evaluating GRR Scores %Tol vs %TV Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 9
Q Evaluating Effectiveness of a Preventive Action after Closure Preventive Action and Continuous Improvement 7
J Potential Customers Asking for EIRs / 483s when evaluating a CRO for future work US Food and Drug Administration (FDA) 2
P FDA - Evaluating Substantial Equivalence in Premarket Notifications 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 1
M Quality Control Procedure and Procedure for Evaluating Sub Contractor Document Control Systems, Procedures, Forms and Templates 1
M Evaluating Vendor (Supplier) Products using SQC Statistical Analysis Tools, Techniques and SPC 6
K Evaluating different Startup Options in EU and MDD ISO 13485:2016 - Medical Device Quality Management Systems 4
S Trading Company - Evaluating Supplier Performance and Approved Supplier List ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
J The 510(k) Program: Evaluating Substantial Equivalence in Premarket Notifications 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 2
E Evaluating Lean Knowledge: Lean Philosophy Course for Production Engineering Students Lean in Manufacturing and Service Industries 1
A Evaluating an Integrated Management System - Dissertation Topic ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
optomist1 Evaluating Capability Indices with Minitab Using Minitab Software 7
C Trend Analysis - Evaluating Quality of Pricing - ANVOA or Std Dev? Statistical Analysis Tools, Techniques and SPC 25
W Re-Evaluating Corrective Actions - Recurrence of a Customer Complaint Nonconformance and Corrective Action 7
D Good source for Checking and Evaluating Control Plans needed FMEA and Control Plans 4
R EMP (Evaluating the Measurement Process) Studies for Bias Effect Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 6
J Evaluating a Regression Model Using the Constant Variance Assumption Six Sigma 8
Wes Bucey "Downsizing" ramifications - Evaluating and Interpreting the News World News 0
A Methods of evaluating compliance to ISO 14001 ISO 14001:2015 Specific Discussions 6
M Evaluating my QMS (Quality Management System) ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 11
G 100% Sort Verification - Statistical Method for Evaluating Suspect Material Statistical Analysis Tools, Techniques and SPC 19
J Evaluating and Reporting the Results of Layered Process Audits Document Control Systems, Procedures, Forms and Templates 9
G Evaluating Software for GMP Criticality? Software Quality Assurance 3
M Evaluating adverse effect on device tested with out of tolerance instrument General Measurement Device and Calibration Topics 3
P Rockwell, Brinell, Vickers Conversion - Evaluating Aluminum 6061-T6 and 6063-T6 General Measurement Device and Calibration Topics 11
U Evaluating the aspects... All judgement based or is there a more objective way? Miscellaneous Environmental Standards and EMS Related Discussions 1
D Evaluating competence on training for new ISO 13485 standard for employees ISO 13485:2016 - Medical Device Quality Management Systems 7
Q "Service" Vendor (Supplier) Evaluation - How are you evaluating suppliers? Supplier Quality Assurance and other Supplier Issues 5
N Evaluating Process Stability - Unstable Processes - Cpk/Ppk? Capability, Accuracy and Stability - Processes, Machines, etc. 6
Z Seeking GM standard GM9684P - Procedure for evaluating parting lines Customer and Company Specific Requirements 2
S Metrics for Evaluating Design process Design and Development of Products and Processes 2
C Evaluating software that measures Ppk, Cmk, Cpk, PPI, etc. Capability, Accuracy and Stability - Processes, Machines, etc. 4
H Good starting point for a new QC Manager? Evaluating Existing Procedures Document Control Systems, Procedures, Forms and Templates 21
Gman2 Evaluating training, needed for office personnel? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 18
G Evaluating Processes - Cycle Time Concepts - Three Levels for any process Manufacturing and Related Processes 3
Marc Evaluating 5.3 c - Quality Policy - Framework General Auditing Discussions 1
T Evaluating Customer Satisfaction ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 14
B Risk Anlysis for Evaluating then Voiding or Not Issuing CPAR's Nonconformance and Corrective Action 1
M ISO 9001:2000 - Evaluating Training Effectiveness and Why ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 12
C Can anyone suggest a means of evaluating a suppliers performance? Supplier Quality Assurance and other Supplier Issues 4
Raffy Evaluation Form - Re-evaluating suppliers for their performance Supplier Quality Assurance and other Supplier Issues 8
G Evaluating Training Effectiveness - 6.2.2 Competence, Awareness and Training ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 55
S Supplier Evaluation - Evaluating transport suppliers and equipment suppliers Supplier Quality Assurance and other Supplier Issues 2
D What exactly is needed on the subcontractor list for evaluating subcontractors QS-9000 - American Automotive Manufacturers Standard 4
D Evaluating Subcontractor On-time Delivery Misc. Quality Assurance and Business Systems Related Topics 3

Similar threads

Top Bottom