Audit Finding - Measurement of Process - Continuous Improvement - Trend Analysis

[Eddie74]

Quick Question - Audit Concern - My question, I measure my processes (KPIs) for continuous improvement with trend analysis. The goal as established with procedures is to show a continuous improvement in the processes through trend analysis. If not to conduct an investigation to determine why, and issues a CAR if needed?
Had the following "concern" by an API Auditor. The auditor was looking for a numeric assigned goal for each processes to show that it each process is measurable,
Do I need to rewrite my procedures to establish a numerical goal for each process?

Auditor concern:
Requirement - MS Standard API Spec Q1 9th Ed - Clause4.1.3 (Measurable)
Description: The quality objectives shall be measurable and consistent with the quality policy.
Objective Evidence: The target for all quality objectives is “Continuous improvement”. No measurable value for any of the quality objectives. The actual charts monitor the performance of each quality objective, but none of them have any measurable target or goal.
The facility insists that continuous improvement is measurable.
In looking at Chart KPIb, it shows the On-Time-Delivery slipping in 2021 from 93% in December 2020 to 77% in December 2021 which indicated that they did not maintain continuous improvement as stated in the Quality Objectives.
This is shown as a concern because the facility is doing a good job in tracking the numbers for each chart and looking for continuous improvement but nothing measurable as a target.
Nonconformance/Concern Description: Quality Objectives are not measurable.
The facility insisted that continuous improvement without measurable target is measurable.

My procedures State:
*4 Three negative trend points in a row results in Top Management Review. It is at the discretion of Top Management to issues a CAR for recommended improvements.

Your Thoughts and inputs shall be greatly appreciated.

Eddie

 

[Tagin]

Saying “Continuous improvement” is like saying "happier customers". How much happier? 1%? 20%?

If your QO said something like "continuous improvement of at least x% per year, as measured by KPI trend analysis" that would probably suffice.

Without the qualifier ("at least x% per year"), there is no way to ascertain if the trend met any particular expectations.

 

[Bev D]

First your auditor is an idiot.
I would advise that you use control charts appropriate for the data types. It isn’t clear what you mean by “trend analysis”. If you are ‘making up the rules‘ (e.g. 3 points in a row) or if you are using the EXCEL trend analysis / linear regression you are well intended and headed in teh right direction but still missing the mark. If you are using SPC then great for you - you are on the exact right track for monitoring continual improvement.

The ‘target’ in SPC would be that the metric meets any of the rules that indicate a real change has occurred. (1 out of 3 points outside the control limits, 2 out of 3 points out of control, 7/8 points in a row trending down, etc.) In teh case of SPC the ‘target‘ is the control limits that border the natural variation of he process. SOC will help you distinguish between true change and ‘lucky’ results. It is also completely justified for quality objectives. Tried and true for longer than your auditor has been alive….

 

[Sidney Vianna]

First your auditor is an idiot.

Mic. Drop.

 

[Eddie74]

Saying “Continuous improvement” is like saying "happier customers". How much happier? 1%? 20%?

If your QO said something like "continuous improvement of at least x% per year, as measured by KPI trend analysis" that would probably suffice.

Without the qualifier ("at least x% per year"), there is no way to ascertain if the trend met any particular expectations.

Hi, Tagin, The Quality Objective Goal for organization is to continually improve our processes. The KPIs are measured and charted on a monthly basis's. Trend analysis is applied to each month results usually previous months data points. The natural of our business, metal distributor, can be impacted by numerous outside sources (as with most ever bodies). Trend analysis is established over ten months. The point is if we show an improvement developing or a negative trend developing to investigate and find out why. Correct if necessary or keep performing what has established the positive trend. Why must a fix number be applied to the analysis process?

 

[Eddie74]

First your auditor is an idiot.
I would advise that you use control charts appropriate for the data types. It isn’t clear what you mean by “trend analysis”. If you are ‘making up the rules‘ (e.g. 3 points in a row) or if you are using the EXCEL trend analysis / linear regression you are well intended and headed in teh right direction but still missing the mark. If you are using SPC then great for you - you are on the exact right track for monitoring continual improvement.

The ‘target’ in SPC would be that the metric meets any of the rules that indicate a real change has occurred. (1 out of 3 points outside the control limits, 2 out of 3 points out of control, 7/8 points in a row trending down, etc.) In teh case of SPC the ‘target‘ is the control limits that border the natural variation of he process. SOC will help you distinguish between true change and ‘lucky’ results. It is also completely justified for quality objectives. Tried and true for longer than your auditor has been alive….

Hi, Bev D, We plot our KPIs on a monthly basis using a 10 month trend analysis, The metrics we utilize to show continual improvement is a increase in the trend. The analysis and charting is posted and emailed to the organization. Trends that show negative or positive developments are investigated if necessary a CAR is issued, positive trends are investigated, changes for the good actually do occur. Control limits are not established (can be). the point being is the establishment of a positive trend.
Eddie

 

[Randy]

The target for all quality objectives is “Continuous improvement”. No measurable value for any of the quality objectives. The actual charts monitor the performance of each quality objective, but none of them have any measurable target or goal.
The facility insists that continuous improvement is measurable.

That's like running a footrace with no defined finish line or any markers to indicate where you started and where you're going..... In the end you're just going to wind up tired with your #ss dragging in the mud. You'd be better off with asking a simple question......Are we making any more money? Yes - No

If No then figure out why and fix it

If Yes then keep on keeping on.

And there is always a measurable value for every objective, qualitative or quantitative

 

[Eddie74]

That's like running a footrace with no defined finish line or any markers to indicate where you started and where you're going..... In the end you're just going to wind up tired with your #ss dragging in the mud. You'd be better off with asking a simple question......Are we making any more money? Yes - No

If No then figure out why and fix it

If Yes then keep on keeping on.

And there is always a measurable value for every objective, qualitative or quantitative

Thanks Randy, can I get definition of "measurable value". Our KPIs are measured every month presented in percentage values. Basically with the trend analysis we ask the question. Is this month better or worse, in trend if you are better than the past you win, if not why? That is the goal, always improve.
Eddie

 

[Bev D]

You aren’t listening.
Your auditor may have said it stupidly but he had a point.
So does Randy.
the thing is you are on the right track but your version of trend analysis is not valid (statistically). IF you actually use SPC with valid control limits and use the valid rules you have a statistically valid approach AND can state the goal numerically and in a statistically valid way of showing continual improvement.

 

[Bev D]

By the way there is exactly a 50% probability that next month will be better than last month. That’s not improvement. It’s Random chance variation. There is a TON of information regarding good analysis of KPIs from Donald Wheeler to Steve Prevette. I suggest you check them out.