Audit Finding - Measurement of Process - Continuous Improvement - Trend Analysis

Eddie74

Starting to get Involved
#1
Quick Question - Audit Concern - My question, I measure my processes (KPIs) for continuous improvement with trend analysis. The goal as established with procedures is to show a continuous improvement in the processes through trend analysis. If not to conduct an investigation to determine why, and issues a CAR if needed?
Had the following "concern" by an API Auditor. The auditor was looking for a numeric assigned goal for each processes to show that it each process is measurable,
Do I need to rewrite my procedures to establish a numerical goal for each process?

Auditor concern:
Requirement - MS Standard API Spec Q1 9th Ed - Clause4.1.3 (Measurable)
Description: The quality objectives shall be measurable and consistent with the quality policy.
Objective Evidence: The target for all quality objectives is “Continuous improvement”. No measurable value for any of the quality objectives. The actual charts monitor the performance of each quality objective, but none of them have any measurable target or goal.
The facility insists that continuous improvement is measurable.
In looking at Chart KPIb, it shows the On-Time-Delivery slipping in 2021 from 93% in December 2020 to 77% in December 2021 which indicated that they did not maintain continuous improvement as stated in the Quality Objectives.
This is shown as a concern because the facility is doing a good job in tracking the numbers for each chart and looking for continuous improvement but nothing measurable as a target.
Nonconformance/Concern Description: Quality Objectives are not measurable.
The facility insisted that continuous improvement without measurable target is measurable.

My procedures State:
*4 Three negative trend points in a row results in Top Management Review. It is at the discretion of Top Management to issues a CAR for recommended improvements.

Your Thoughts and inputs shall be greatly appreciated.

Eddie
 
Elsmar Forum Sponsor

Tagin

Trusted Information Resource
#2
Saying “Continuous improvement” is like saying "happier customers". How much happier? 1%? 20%?

If your QO said something like "continuous improvement of at least x% per year, as measured by KPI trend analysis" that would probably suffice.

Without the qualifier ("at least x% per year"), there is no way to ascertain if the trend met any particular expectations.
 

Bev D

Heretical Statistician
Leader
Super Moderator
#3
First your auditor is an idiot.
I would advise that you use control charts appropriate for the data types. It isn’t clear what you mean by “trend analysis”. If you are ‘making up the rules‘ (e.g. 3 points in a row) or if you are using the EXCEL trend analysis / linear regression you are well intended and headed in teh right direction but still missing the mark. If you are using SPC then great for you - you are on the exact right track for monitoring continual improvement.

The ‘target’ in SPC would be that the metric meets any of the rules that indicate a real change has occurred. (1 out of 3 points outside the control limits, 2 out of 3 points out of control, 7/8 points in a row trending down, etc.) In teh case of SPC the ‘target‘ is the control limits that border the natural variation of he process. SOC will help you distinguish between true change and ‘lucky’ results. It is also completely justified for quality objectives. Tried and true for longer than your auditor has been alive….
 

Eddie74

Starting to get Involved
#5
Saying “Continuous improvement” is like saying "happier customers". How much happier? 1%? 20%?

If your QO said something like "continuous improvement of at least x% per year, as measured by KPI trend analysis" that would probably suffice.

Without the qualifier ("at least x% per year"), there is no way to ascertain if the trend met any particular expectations.
Hi, Tagin, The Quality Objective Goal for organization is to continually improve our processes. The KPIs are measured and charted on a monthly basis's. Trend analysis is applied to each month results usually previous months data points. The natural of our business, metal distributor, can be impacted by numerous outside sources (as with most ever bodies). Trend analysis is established over ten months. The point is if we show an improvement developing or a negative trend developing to investigate and find out why. Correct if necessary or keep performing what has established the positive trend. Why must a fix number be applied to the analysis process?
 

Eddie74

Starting to get Involved
#6
First your auditor is an idiot.
I would advise that you use control charts appropriate for the data types. It isn’t clear what you mean by “trend analysis”. If you are ‘making up the rules‘ (e.g. 3 points in a row) or if you are using the EXCEL trend analysis / linear regression you are well intended and headed in teh right direction but still missing the mark. If you are using SPC then great for you - you are on the exact right track for monitoring continual improvement.

The ‘target’ in SPC would be that the metric meets any of the rules that indicate a real change has occurred. (1 out of 3 points outside the control limits, 2 out of 3 points out of control, 7/8 points in a row trending down, etc.) In teh case of SPC the ‘target‘ is the control limits that border the natural variation of he process. SOC will help you distinguish between true change and ‘lucky’ results. It is also completely justified for quality objectives. Tried and true for longer than your auditor has been alive….
Hi, Bev D, We plot our KPIs on a monthly basis using a 10 month trend analysis, The metrics we utilize to show continual improvement is a increase in the trend. The analysis and charting is posted and emailed to the organization. Trends that show negative or positive developments are investigated if necessary a CAR is issued, positive trends are investigated, changes for the good actually do occur. Control limits are not established (can be). the point being is the establishment of a positive trend.
Eddie
 

Randy

Super Moderator
#7
The target for all quality objectives is “Continuous improvement”. No measurable value for any of the quality objectives. The actual charts monitor the performance of each quality objective, but none of them have any measurable target or goal.
The facility insists that continuous improvement is measurable.
That's like running a footrace with no defined finish line or any markers to indicate where you started and where you're going..... In the end you're just going to wind up tired with your #ss dragging in the mud. You'd be better off with asking a simple question......Are we making any more money? Yes - No

If No then figure out why and fix it

If Yes then keep on keeping on.

And there is always a measurable value for every objective, qualitative or quantitative
 

Eddie74

Starting to get Involved
#8
That's like running a footrace with no defined finish line or any markers to indicate where you started and where you're going..... In the end you're just going to wind up tired with your #ss dragging in the mud. You'd be better off with asking a simple question......Are we making any more money? Yes - No

If No then figure out why and fix it

If Yes then keep on keeping on.

And there is always a measurable value for every objective, qualitative or quantitative
Thanks Randy, can I get definition of "measurable value". Our KPIs are measured every month presented in percentage values. Basically with the trend analysis we ask the question. Is this month better or worse, in trend if you are better than the past you win, if not why? That is the goal, always improve.
Eddie
 

Bev D

Heretical Statistician
Leader
Super Moderator
#9
You aren’t listening.
Your auditor may have said it stupidly but he had a point.
So does Randy.
the thing is you are on the right track but your version of trend analysis is not valid (statistically). IF you actually use SPC with valid control limits and use the valid rules you have a statistically valid approach AND can state the goal numerically and in a statistically valid way of showing continual improvement.
 

Bev D

Heretical Statistician
Leader
Super Moderator
#10
By the way there is exactly a 50% probability that next month will be better than last month. That’s not improvement. It’s Random chance variation. There is a TON of information regarding good analysis of KPIs from Donald Wheeler to Steve Prevette. I suggest you check them out.
 
Thread starter Similar threads Forum Replies Date
J 8.3.2.1 Monitoring/Measurement Audit Finding IATF 16949 - Automotive Quality Systems Standard 6
G During internal audit - finding poor action plans ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 18
C Need help in determining applicable clause for an audit finding (based on AS9120B) ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 4
S Audit Finding - Design History File (DHF) Index: few (3 to 4) reports not identified ISO 13485:2016 - Medical Device Quality Management Systems 3
Mikey324 External calibration - Finding in our 3rd party audit General Measurement Device and Calibration Topics 58
M How to answer ISO9001:2015 audit finding of old revisions of documents being used? Document Control Systems, Procedures, Forms and Templates 8
Ed Panek Auditor MDR (Presub audit) finding EU Medical Device Regulations 2
Q ISO 9001/IATF 16949 Audit Finding Question - Document Retention IATF 16949 - Automotive Quality Systems Standard 11
Q ISO 9001-2015 Internal audit finding Internal Auditing 14
A Define timeline for Major and Miner Audit finding General Auditing Discussions 4
J Audit Finding For Not Retaining Test Results ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 7
E Received a Major finding during IATF Surveillance audit for loss of BIQS Level 3 (more than 6 SPPS in 6 months)...how should we address SYSTEMIC CA? IATF 16949 - Automotive Quality Systems Standard 11
D Customer Audit Finding ISO13485:2016 7.6 ISO 13485:2016 - Medical Device Quality Management Systems 7
B PED 2014/68/EU - Is this Traceability audit finding correct? Other ISO and International Standards and European Regulations 4
XRAY_3121 Design and Development Requirement - MDSAP Audit Finding Other Medical Device Regulations World-Wide 5
R Major nonformance finding was given during a closing meeting of a ISO9001 certification audit General Auditing Discussions 76
G Audit finding - Components being transferred inter-plant Internal Auditing 3
B IATF 16949 7.1.5.3.2 FAQ #7 Audit Finding - External Calibration Laboratory IATF 16949 - Automotive Quality Systems Standard 7
S GM/VP softgrading internal audit finding - need feedback from an audit guru! General Auditing Discussions 11
Q Customer Requirements - AS9120B Audit Finding AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 25
J Hardness inspection Audit Finding Employee error Nonconformance and Corrective Action 6
J How long should I have to correct an audit finding? VDA Standards - Germany's Automotive Standards 7
AlienraverX How to appeal a major audit finding ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 26
S Definition Would a wrong definition constitute an Audit Finding? Definitions, Acronyms, Abbreviations and Interpretations Listed Alphabetically 3
L Audit Finding Flowchart for Feedback General Auditing Discussions 6
A Audit Finding - CAPA, Improvement Initiatives not filed in CAR System ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
O CFDA (China Food and Drugs Administration) Audit Finding Classifications China Medical Device Regulations 1
B Expired Materials (threadlocker) Audit Finding Classification? Internal Auditing 1
H Cause analysis for an Audit Finding - Purchasing Procedure Content ISO 13485:2016 - Medical Device Quality Management Systems 9
A Writing an Internal Audit Report Finding for Employee Awareness of QMS Documents ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 11
S Audit finding for JPAL ISO 13485:2016 - Medical Device Quality Management Systems 1
R Can a external auditor raise a finding that is already identified in Internal Audit ? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 11
G Supplier Significant Environmental Aspects - Audit Finding ISO 14001:2015 Specific Discussions 21
C Notification to Customers regarding Major Audit Finding IATF 16949 - Automotive Quality Systems Standard 25
J Questionable Audit Finding about Revision Levels of Product Manual CD's ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 7
J Follow up on Major Audit Finding General Auditing Discussions 15
C ISO 9001:2008 Audit Major Finding - Dial Calipers not set to zero ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 16
M Internal Audit finding versus External Audit finding General Auditing Discussions 2
E Nadcap Major Audit Finding Question - Effectiveness of our RCCA AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 11
E Nadcap - Tooling Control - Major Audit Finding (Question) AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 6
A Audit Finding on 6.2.2 - Education Requirements not Verified ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 6
T Quality Objective related Surveillance Audit Finding IATF 16949 - Automotive Quality Systems Standard 6
R Auditors can NOT audit their own work audit finding Internal Auditing 17
G Thoughts on Audit Finding for not doing Gage R and R for Visual Inspection Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 8
J EU labeling requirement Notified body audit finding EU Medical Device Regulations 3
G Audit Finding against ISO/TS 16949 6.2.2 Competence, Training and Awareness IATF 16949 - Automotive Quality Systems Standard 3
L Internal Re-audit after Major Finding during ISO 9001 Re-Certification Audit ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 10
A Acceptance Criteria - ISO 9001:2008 Clause 8.2.4 - Questionable Audit Finding? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 40
A 3rd Party Audit Finding Not Clear - 4.1 Outsourced Processes AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 36
A Excel Spreadsheet Validation - Customer Audit Finding Other US Medical Device Regulations 2

Similar threads

Top Bottom