Process Effectiveness Metric for the process ?Internal Audit?

Steve Prevette

Deming Disciple
Staff member
Super Moderator
#11
This is a tough one, especially in these days when "audits" and "auditors" are not seen as being very value-added by management. With a question like "what is the effectiveness of auditing", you have the opportunity to document the value of doing audits.

We have a somewhat similar situation in safety here at Fluor Hanford. We have a mandatory safety inspection program, and all the inspections come back to me for analysis. We do plot the number of inspections and the inspections scores on control charts, and also lay them out with the injury charts. It does appear there is some correlation - in those time intervals where inspections drop off, injuries go up, and vice - versa. Now, I will not suppose that inspections and injuries are a direct cause and effect relationship, but there is enough correlation and theory (get people out there looking for and fixing hazards, and you'll prevent injuries) that it is used as an effectiveness measure for safety overall.

In auditing, I would want to ask the bigger picture - what process is the audit supposed to be helping, and what are the effectiveness measures for that process? Can I link audit actions (and corrective actions and improvements following an audit) to improved outcome measures?

The question is related to internal audits. Do you have external audits of the same areas, and can show those areas internally audited do not have many external findings? How about customer satisfaction measures related to areas with audits?

I think a key is to sit back and determine what the aim of doing internal audits is, and what you expect them to accomplish, and how they interact with the overall systems. Then convert that to measurables.
 
Elsmar Forum Sponsor
V

vanputten

#12
I forgot to give credit where credit is due. The content of both of my postings in this thread should give credit to Ira Epstein.


Here is a response to a similar question from a now defunct quality forum. I believe it mirrors what Steve has posted.

_______________________

You ask an excellent question. You would be better able to answer your own question if you thought of auditing as a process, which it is. All
processes have inputs, outputs and activities. Also, all processes
have goals or objectives.

Since all organizations are different, you need to determine the
objectives of your internal audit process. Examples of auditing
objectives are: to determine whether the QMS conforms to planned
arrangements, to determine whether the QMS is effectively implemented
and maintained, to uncover QMS noncompliance, to add value to the
auditee, to provide information for the continual improvement process,
to help management make decisions and to achieve organizational
objectives.

Once you have decided on the objective(s) of your process, you need to
determine how you will measure that the objective(s) are being
achieved. For example, if you determine that adding value to the
auditee is one objective, you will need to sit down with the
auditee(s) and agree on some measures. Measures to meet this objective
may include: the number of major findings, the number of minor
findings, the percentage of findings agreed to by the auditee, the
length of time for the audit, the degree of knowledge of the auditor
as measured by the auditee, the degree of disruption of the operations
because of the audit, some measure of auditor efficiency and
professionalism, and the degree of improvement resulting from
corrective action.

There are potentially a large number of measures for the auditing
process. You must select the ones which are the most significant for
your organization and the ones which directly relate to achievement of
the quality policy. I hope this helps somewhat.
 
Last edited by a moderator:
X
#13
ralphsulser said:
Here is what I have done. Developed a bar graph in excel for all my internal audits for each year, from 2002 thru current. Plotted are number of audits, number of NCs and Percent NC for each month on same chart.Trend shows definite improvement and a downward trend in NCs and Percent NC, with trend lines.
This shows effectivness of Internal Audits since the number of NCs per audit have reduced. I also show majors, minors and OFIs on another chart along with any open issues. Additionally I have an audit summary sheet developed for alll the audits which lists all dates, audits, NCs, areas, shifts, issue dates, answer due dates, target dates, actual answer dates, reaudit dates and open or closedstatus, and who has responsibiity for NC.

I would say this all on total is an indication of effectiveness. It passed our TS audit.
This is essentially the same way I have my auditing program (and reporting/metrics) set up here. We have been TS registered for almost 2 years now with no problem at all in this area.

When it comes to how effective the auditing itself actually is what I do is inspect ad review the final audit report from every audit that my auditors perform and make sure they have covered off everything they need to for the process they are auditing and that they have an *appropriate* amount of relevant backup material and objective evidence to support their findings (good or bad.)

So far so good.
 
C

CBJ Hockey Fan

#14
Internal Audit Effectiveness

O.K. many posters have posted threads concerning efficiency, but nothing concerning Internal Audit EFFECTIVENESS.

A definition of effectiveness is: Having an intended or expected effect. The power or capacity to produce a desired result.

Based of this definition, does anybody out there have measurable metric to monitor Internal Audit EFFECTIVENESS?

Please advise. THANKS :thanx:
 

Steve Prevette

Deming Disciple
Staff member
Super Moderator
#15
CBJ Hockey Fan said:
A definition of effectiveness is: Having an intended or expected effect. The power or capacity to produce a desired result.

Based of this definition, does anybody out there have measurable metric to monitor Internal Audit EFFECTIVENESS?
Such a metric can only come from YOU. What is the "desired result" of doing all these audits? Gain certification? Avoid bad headlines in the paper? Improve the cycle time in production? Reduce defect rates? What is it your company is in the business of doing?

On the basis of the answers to these questions, devise your audit schedule. As you complete audits, are actions taken as a result which then show a significant improvement in these measures? Do audit "scores" parallel the measures (that is - if the audit determined there are problems, are those problems manifested in the company performance?).

Now, there may be some audit findings that reflect risk. They have not yet demonstrably affected outcomes, but they could. In these cases, you may have to come up with measures reflected "avoided" costs or negative outcomes.
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#16
hmmm. effectiveness of internal audits?
well at the simplest level, the purpose of internal audits is to assess the level of compliance. we could think of it in the same way we might think of a visual inspection: we typically - in the simplest way know the effectiveness of an inspection by how many escapes we get. We may also look at our sample size and frequency compared with our defect rate to determine our theoretical/statistical effectivenes of catching defects. We may also perform a measurement system analysis (using a Kappa test) to see if an inspector will in fact properly detect defects and to assess operator to operator differences. Each of these approaches can be taken with the audit process itself.

A second level purpose of the audit system is to use the findings to initiate meaningful corrective action. Of course the results are somewaht confounded with the effectiveness of our corrective action system itself. However, if we assume that the corrective action process is effective. A simple asessment of this to ask if the findings are meaningful? If they are meaningful - are we improving our level of compliance on those items that we are finding in the internal audit process)

A metric? How about trending the overall finding rate and then sub trends on the pareto items of finding types (repeat findings)?
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#17
In thinkin gabout this some more you may not need a 'metric'. You could do a quarterly, every six months or yearly 'assessment'. an audit of the audit process if you will. decide which aspects of the internal audit process are important as far as achieving the purpose (primarily, do the internal audits accurately assess the organization's compliance to the QMS; secondarily does the audit process drive effective and meaningful corrective actions to improve the org's compliance?). You could record the results on a 'scorecard' and report to the management review with recommendations.
 
V

vanputten

#18
Some of my suggested process measures were measures of effectiveness. There may be better measures. These may not be easy. However, some of my suggestions for process measureables were effectiveness measures.

# of audit notices sent out on time, (effectiveness)
Resources needed to have audit notices sent out on time, (efficiency)
# of checklists created on time, (effectiveness)
Resources needed to have checklists created on time, (efficiency)
# of audit reports completed and distributed in a timely manner, (effectiveness)
Resources needed to have audit reports completed and distributed in a timely manner (efficiency)
# of disagreements on audit findings after the closing meeting, or after the audit report is published, (effectiveness)
Resources needed to have an acceptable level of disagreements on audit findings after the closing meeting (efficiency)
etc., etc.

Regards, Dirk
 
R

Ralph Kolts

#19
Audit Performance Measures

This happens to be one of my favorite subjects - - I had written a paper on this topic a few years back that I have attached for information. You will see this paper describes three types of measures for 1) audit activities, 2) outcomes, and 3) results with various examples identified in the table on page 4. I hope this might give you an idea of some measures that will work for you in your own unique environment.
 

Attachments

S

Shaun Daly

#20
IMHO I would say,

Efficiency - Completing audits on time.
Effectiveness - ?

If the purpose of audits is to ensure compliance, and ensure they are corrected, after all,

A - Audits
U - Uncover
D - Defects
I - In
T - The
S - System

Then the most applicable long term effectiveness measure will be...

Reduced numbers of audit findings?
 
Thread starter Similar threads Forum Replies Date
M How to increase survey response rate? - Process Effectiveness metric Six Sigma 5
T Hot dogs and process operation and control effectiveness? 4.1.3 a) ISO 13485:2016 - Medical Device Quality Management Systems 2
S ISO 9001:2015 4.4.1 (c) Process Effectiveness ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 17
A IATF 16949 - Non-conformity on 5.1.1.2 Process effectiveness and efficiency IATF 16949 - Automotive Quality Systems Standard 5
B IATF 16949 Cl. 7.3.2 - Documented Process (Effectiveness & Efficiency) IATF 16949 - Automotive Quality Systems Standard 7
J Process Effectiveness and Efficiency (IATF 16949) IATF 16949 - Automotive Quality Systems Standard 29
D Do people read PEARS (Process Effectiveness Assessment Report)? AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 15
B IATF 16949 Cl. 9.3.2.1 - Management Review Inputs - Process Effectiveness and Efficie IATF 16949 - Automotive Quality Systems Standard 14
S AS9100C 8.2.2 - Selected Tools & Effectiveness of IA Process and Performance AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 3
Buckyb Are KPIs a requirement to determine Process Effectiveness? AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 14
R The Value and Effectiveness of In Process Inspection Quality Manager and Management Related Issues 17
Q PEAR (Process Effectiveness Assessment Report) Requirements for AS9100 Registration AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 4
J What Practices Undermine the Effectiveness of the A3 Process? Lean in Manufacturing and Service Industries 2
G Reviewing Process Effectiveness and Efficiency IATF 16949 - Automotive Quality Systems Standard 5
J How to measure effectiveness of the Metrology Process General Measurement Device and Calibration Topics 5
Chennaiite How to Measure Effectiveness of the Design Validation Process? Design and Development of Products and Processes 7
P Measuring Process Effectiveness - Measures of Effectiveness (MOE's) AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 29
W Process Effectiveness NCR's (Nonconformance Reports) ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
Sidney Vianna AS9101 Process Effectiveness Assessment Report/Nonconformance Record Review Modules AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 10
D AS9100C PEAR (Process Effectiveness Assessment Report) for Design and Development AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 6
Q How do you determine Process Effectiveness for AS9100C Process Maps, Process Mapping and Turtle Diagrams 11
B AS9120A Process Effectiveness Assessment Report Forms AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 23
A How to calculate the Effectiveness of the Process and Product Audit System Internal Auditing 10
E Measuring Receiving Inspection Process Effectiveness AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 8
P AS9100:C PEARs (Process Effectiveness Assessment Report) AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 171
A What is the difference between Process Objectives & Criteria for Effectiveness? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 4
P Audit of the Manufacturing Process Effectiveness - 8.2.2.2 IATF 16949 - Automotive Quality Systems Standard 10
H LPA's (Layered Process Audits) - Evaluation of Effectiveness Process Audits and Layered Process Audits 22
M Effectiveness and Efficiency - Process Objective Determination IATF 16949 - Automotive Quality Systems Standard 7
S How to Evaluate the Effectiveness of a Work Instruction during the Approval Process? Document Control Systems, Procedures, Forms and Templates 13
M Showing the Effectiveness of the Internal Audit Process System - AS9100B Clause 8.2.2 AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 4
G Process Efficiency vs. Process Effectiveness for TS 16949 IATF 16949 - Automotive Quality Systems Standard 3
B Process Performance Parameters for efficiency and effectiveness Manufacturing and Related Processes 1
L Measureables: Efficiency - Effectiveness of COP (Customer Oriented Process) and KSP ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 8
C Evaluating the Effectiveness of Internal Audit Process Internal Auditing 16
P Effectiveness of a manufacturing process? General Auditing Discussions 8
J Determining effectiveness of APQP Process throughout our entire organization Preventive Action and Continuous Improvement 4
E Difficulties with Process Approach - Defining 'Effectiveness' ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 59
D Process and Product Audit Checklist - Product Realization, Effectiveness of Process General Auditing Discussions 8
J Need Help with FPY Data in Assembly Process Manufacturing and Related Processes 7
A When someone refuses to follow a process.... Misc. Quality Assurance and Business Systems Related Topics 21
E Software maintenance Process Software maintenance Process to IEC 6204? IEC 62304 - Medical Device Software Life Cycle Processes 3
R AS5553 Clause 3.1.7 f - "Implement a returns process....." AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 5
normhowe "The Problem with Quality Management: Process orientation, controllability and zero-defect processes as modern myths" Book, Video, Blog and Web Site Reviews and Recommendations 2
Judy Abbott General temperature used in the blasting process and laser process Manufacturing and Related Processes 2
B SOP for CNC turret punching machine for sheet metal process Manufacturing and Related Processes 0
A API Monogram audit review process Oil and Gas Industry Standards and Regulations 4
R AS9102 FAI Change in Material / Process Supplier AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 4
A Process mapping Process Maps, Process Mapping and Turtle Diagrams 1
R MDEL Process Canada Medical Device Regulations 4

Similar threads

Top Bottom