How to calculate the Effectiveness of the Process and Product Audit System

A

ajay.khosla

#1
Hello to all the covers

I have started doing process audits in my new firm.
All the audits are being done on time till now.
In my internal audit manual,it is specified that effectiveness of process and product audit should be calculated because in management review,this point will be discussed.
please tell me how to calculate the effectiveness of the process and product audit system?
 
Elsmar Forum Sponsor

Colin

Quite Involved in Discussions
#2
Re: Effectiveness of audits?

I think it depends on what you mean by 'calculate' the effectiveness of the audit process. I am not sure I could (or would want to) put numbers on this sort of thing.

I often see companies who do their audits strictly in terms of whether an activity complies with a procedural statement and come up with a yes/no answer with (hopefully) some recorded evidence that reflects which samples they took to arrive at that conclusion. Well, that's OK for starters but it seems a waste of an opportunity if the auditor doesn't dig a bit deeper and seek improvements as well as compliance.

Rather than just ask if a certain form is being completed, why not ask "why are we using the form" or "could the form be improved" or "could 2 forms be merged into 1" etc. Now, we have to be careful not to be telling the auditor how to do their job (remember, we are objective and impartial) and I would caution this approach in the early stages of a QMS.

The aim here for me is to make internal audits useful to the organisation, not just a 'tick the box' element of standards.
 
J

JaneB

#3
Re: Effectiveness of audits?

Hello to all the covers

I have started doing process audits in my new firm.
All the audits are being done on time till now.
In my internal audit manual,it is specified that effectiveness of process and product audit should be calculated because in management review,this point will be discussed.
please tell me how to calculate the effectiveness of the process and product audit system?
If your manual only says that it should be calculated in order 'to be discussed', it is definitely missing some information. That's not a good enough reason!

I'm not clear about whether your question relates to a/calculating effectivess of the audit process itself, or whether b/you're asking about calculating effectivness of processes/products in general.

If a/ - no, I disagree. There have been a number of discussions on this in the Cove. Essentially, there's not a lot of point in doing so, nor in trying to come up with calculations. And whether audits occur 'on time' or not is not a useful metric.

If b/ No, you don't 'calculate', you do measure and/or monitor.
There should be some specified measurements/metrics/performance indicators for products (assuming you manufacture products) and for processes used to do this. The process owner is (or should be!) responsible for a/setting the metrics and b/deciding what kind of data/indicators are needed in order to assess whether these are performing as required and of course c/doing something about it if not.

It is very definitely not the job of the auditor to 'calculate effectiveness' of product or process. But useful questions to ask during process audits are:
- what are the objectives/aims of this process?
- where are they specified? how are they measured/established? (if not, why not?)
- what can you show me to demonstrate that this is being done?
 

6thsense

Involved In Discussions
#4
Re: Effectiveness of audits?

I present the audit findings to top management. I also explicity highlight an findings that have reoccurred. Prior to each audit i go over all previous audit results.
 
J
#5
I agree with others that "calculate" is a poor term to use.
How do you "calculate" what is basically imprecise data?

I would think to change the term calculate to evalaluate.....

In any case About the best you can do is to score in one of two ways - either pass/fail (complies with procedures or not) or better, unchanged, worse. Then you can explain what contributed to the decision....

To me though the only metrix that really means anything is output. Does the output meet specifications for form fit and function. This can be applied to internal as well as external customers. So as part of a process audit I would want to check with the internal customer to see if they are satisfied with the output.

Peace
James
 
C

cclee

#6
Hi,
Effectiveness is defined as (ANSI/ISO/ASQ Q9000-2005): extent to which planned activities are realized and planned results achieved.

Whether you are doing a system, process or product audit, the evaluation of your audit program effectiveness should include: the audit performance, auditor performance and value added contributions.

Some performance indicators that I use to report the effectiveness of the audit program during managment review include but not limited to:
-chart showing internal and supplier audit findings (CA, PA, OFI) for the reporting period, pareto chart showing contribution to the findings, planned audits vs completed audits (internal/supplier audit schedule/logs)
-chart showing number of days taken to issue audit report
-cost related to the audit program and related activities
-audit long term planning and future focus areas
-auditor training program (new and experience auditors)
 
A

ajay.khosla

#7
Thanks to all the persons who gave valuable information for my question
I also think that to calculate effectiveness is not a good idea.
But i think i can evaluate the effectiveness by making pareto charts of the repeated NCs in the same process,planned audits vs. completed audits(Thanks Cclee for this) and to see whether output meets the specifications or not after completing corrective actions.
 
R

roundtoit

#8
Here are some examples of what we use to measure the effectiveness of our IA process. Our Managemnet reviews our metrics monthly, or as applicable. Decisions are made based upon our data.

1. Past audit (internal/external) findings – Follow-up audit is to provide independent verification if the actions taken were effective.
1.1. Areas of weaknesses or opportunity for improvements (OI) are being addressed and monitored for successful implementation to obtain planned results.
2. Criticality to business function – Processes in the QMS that if an error occurred would impact business and/or stakeholder confidence.
2.1. Metric – Zero nonconformities
3. Trends/Potential problem areas – Level of concern or area of interest identified by process owners, managers, directors, or quality management office.
3.1. Observation of the process to collect data should occur prior to initiating a formal audit.
3.2. Evaluation of CARs indicate a concern requiring action to correct situation before becoming a non-conformity
3.3. Metric – Audit results provided < 5 business days
3.4. Metric – Action Plan proposed < 10 business days of audit report
4. Review of CARs, audit findings, and document modifications – CARs are reviewed for isolated and systemic occurrence and actions taken.
4.1. Document modifications are being correctly implemented and control maintained.
4.2. Did the corrective and preventive actions achieve the results as planned?
4.3. Metric – successful implementation of planned results => 90%

Martin
 
T

TheRiddle

#9
Thanks to all the persons who gave valuable information for my question
I also think that to calculate effectiveness is not a good idea.
But i think i can evaluate the effectiveness by making pareto charts of the repeated NCs in the same process,planned audits vs. completed audits(Thanks Cclee for this) and to see whether output meets the specifications or not after completing corrective actions.
thank you but, can you explain how can I do that ?
 
M

MARKSWELT

#10
here are some examples of what we use to measure the effectiveness of our ia process. Our managemnet reviews our metrics monthly, or as applicable. Decisions are made based upon our data.

1. past audit (internal/external) findings – follow-up audit is to provide independent verification if the actions taken were effective.
1.1. areas of weaknesses or opportunity for improvements (oi) are being addressed and monitored for successful implementation to obtain planned results.
2. criticality to business function – processes in the qms that if an error occurred would impact business and/or stakeholder confidence.
2.1. metric – zero nonconformities
3. trends/potential problem areas – level of concern or area of interest identified by process owners, managers, directors, or quality management office.
3.1. observation of the process to collect data should occur prior to initiating a formal audit.
3.2. evaluation of cars indicate a concern requiring action to correct situation before becoming a non-conformity
3.3. metric – audit results provided < 5 business days
3.4. metric – action plan proposed < 10 business days of audit report
4. review of cars, audit findings, and document modifications – cars are reviewed for isolated and systemic occurrence and actions taken.
4.1. document modifications are being correctly implemented and control maintained.
4.2. did the corrective and preventive actions achieve the results as planned?
4.3. metric – successful implementation of planned results => 90%

martin
how many times a problem can be a repetative ? Eg: In a span of three months? Like tool change is not effective? 3 in 3 months ok or it should be less tahn 3 or up to 4 is allowed?

Any guide lines?
 
Thread starter Similar threads Forum Replies Date
P MSA Attribute Studies - Calculate Cross Tabs, Effectiveness, Miss Rate, False Alarm Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 10
N Effectiveness, miss rate and false alarm rate - Need formulas to calculate Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 4
S Bend allowance formula to calculate per degree per radii Manufacturing and Related Processes 2
P Commercial excel templates that calculate overhead, hourly rates, fee proposal Excel .xls Spreadsheet Templates and Tools 0
M How to calculate benefits? Moving some developed non controlled software/automation systems Service Industry Specific Topics 2
qualprod Calculate number of people in processes, considering cycle time/takt time = quantity of people Lean in Manufacturing and Service Industries 9
S How to calculate the maximum allowable hysteresis of a pressure gauge General Measurement Device and Calibration Topics 1
S How to Calculate the producer Risk AQL - Acceptable Quality Level 25
qualprod Methods to Calculate Available Production Rate ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 0
K Why calculate LCL in P-chart? Statistical Analysis Tools, Techniques and SPC 4
B How to calculate CPK with n = 1 Manufacturing and Related Processes 2
A Add-On software for VMC to calculate setting time, Idle Time, OEE etc.. Manufacturing and Related Processes 2
L How to calculate the Probability of Deficiency of the AQL of an Inspection AQL - Acceptable Quality Level 16
V How to Calculate Unbiasing Constant C4 Statistical Analysis Tools, Techniques and SPC 1
J How to Calculate OEE Indicator for a Workshop (15 Machines) Lean in Manufacturing and Service Industries 6
R How to calculate UCL (Upper COntrol Limit) and LCL (Lower Control Limit) Statistical Analysis Tools, Techniques and SPC 5
T I want to calculate reliability at 750 hours Reliability Analysis - Predictions, Testing and Standards 8
R How To Calculate & Print CDF and Lower Bound CDF value for some t(time) with Minitab? Using Minitab Software 1
N How to calculate the Premium Freight charges Service Industry Specific Topics 4
A How to calculate Mean and standard deviation without sample size in Minitab Using Minitab Software 5
R When do I need to calculate Process OEE? Manufacturing and Related Processes 12
kedarg6500 How to calculate Thread Pitch Diameter for non standard TPI like 5/8" 17.75 NF3 Manufacturing and Related Processes 3
C Definition and Way to Calculate some Metrics Lean in Manufacturing and Service Industries 6
M How calculate P value for Linearity and Bias using Excel Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 8
N Formula that will calculate the NoGo Major Diameter of an M20 x 1.5 6H Thread gage Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
M How to calculate Cg and Cgk for a 2-Dimensional Area Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 4
T Calculation Question - Calculate the runner weight for a part Manufacturing and Related Processes 2
M Spreadsheet on How to Calculate SMT Capacity wanted Manufacturing and Related Processes 2
F How to Calculate the unbiased Standard Deviation in Excel using Minitab Formula Capability, Accuracy and Stability - Processes, Machines, etc. 6
S Spreadsheet that will Calculate Audit Duration times based on IAF MD5 and MD11 Excel .xls Spreadsheet Templates and Tools 7
M Calculate PPMs by going back to Date of Production or Date Noticed Quality Tools, Improvement and Analysis 7
M How to calculate MTTF (Mean Time to Failure) Reliability Analysis - Predictions, Testing and Standards 1
H How to calculate the Lot Reject Rate Inspection, Prints (Drawings), Testing, Sampling and Related Topics 4
L How calculate the number of samples Statistical Analysis Tools, Techniques and SPC 2
A Simple 125 piece part Capabilty Excel .xls sheet to calculate Cpk Capability, Accuracy and Stability - Processes, Machines, etc. 3
R How to calculate MMC when is placed after datum Inspection, Prints (Drawings), Testing, Sampling and Related Topics 4
S Which one is better to calculate, Cpk or Ppk? Statistical Analysis Tools, Techniques and SPC 11
C How to calculate P value on Probability Plot Statistical Analysis Tools, Techniques and SPC 1
S Supplier Disruption Score and how to calculate it Quality Tools, Improvement and Analysis 4
D IAQG OASIS - New tool to calculate Audit Time AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 1
M How to calculate dimensions of trapezoid ? Quality Tools, Improvement and Analysis 3
J How to calculate Surveillance Audit Days in Corporate Audit Scheme IATF 16949 - Automotive Quality Systems Standard 4
S How to calculate Supplier PPM at Receiving Inspection Document Control Systems, Procedures, Forms and Templates 2
K How to Calculate Sample Size at 95% Confidence Interval on Surface Roughness Inspection, Prints (Drawings), Testing, Sampling and Related Topics 1
R How to calculate the value add time? Lean in Manufacturing and Service Industries 2
G Should I calculate both Cp and Cpk for a Unilateral Tolerance? Statistical Analysis Tools, Techniques and SPC 8
H Continuous Monitoring Instrument Data Validation and how to calculate Outliers Quality Tools, Improvement and Analysis 1
R How to calculate the Air Flow Rate in clause 201.12.4.4.105 of IEC 60601-2-16:2012 IEC 60601 - Medical Electrical Equipment Safety Standards Series 1
A How to calculate Cpk & Ppk if the tolerance changes during the period? Capability, Accuracy and Stability - Processes, Machines, etc. 5
S PPAP - How to calculate tolerance on basic dimension APQP and PPAP 5

Similar threads

Top Bottom