Audit Finding - Measurement of Process - Continuous Improvement - Trend Analysis

Mike S.

Happy to be Alive
Trusted Information Resource
Maybe I'm dense, but I still don't understand what 10 month trend analysis is, or how it relates to 3 negative trend points.

The auditor reported On-Time-Delivery slipping in 2021 from 93% in December 2020 to 77% in December 2021. That's 12 months. Did you have no trends in there triggering management review and/or a corrective action? You don't want to be reacting to noise, nor do you want to miss a signal.

Maybe an example will help us understand.
 

Eddie74

Starting to get Involved
By the way there is exactly a 50% probability that next month will be better than last month. That’s not improvement. It’s Random chance variation. There is a TON of information regarding good analysis of KPIs from Donald Wheeler to Steve Prevette. I suggest you check them out.

Ok Thanks for the information. Bev D.
Eddie
 

Sidney Vianna

Post Responsibly
Leader
Admin

Tagin

Trusted Information Resource
Hi, Tagin, The Quality Objective Goal for organization is to continually improve our processes. The KPIs are measured and charted on a monthly basis's. Trend analysis is applied to each month results usually previous months data points. The natural of our business, metal distributor, can be impacted by numerous outside sources (as with most ever bodies). Trend analysis is established over ten months. The point is if we show an improvement developing or a negative trend developing to investigate and find out why. Correct if necessary or keep performing what has established the positive trend. Why must a fix number be applied to the analysis process?

It sounds like what you are doing is monitoring what is occurring, and then reacting accordingly. You are using lessons learned ("keep performing what has established the positive trend") after-the-fact as a way to foster improvement. Useful, yes, but not a QO.

A QO, on the other hand, is intended to be proactive - a plan towards a defined goal to be achieved. That goal must be measurable, either quantitatively or qualitatively. Likewise, the activity for a QO will be proactive, and based on a plan to achieve that goal.
 

Bev D

Heretical Statistician
Leader
Super Moderator
I think the OP is on the right track but is not on the right train…

Here’s the thing about improvement: it is IMPOSSIBLE to determine ahead of time how much improvement you can achieve from any improvement action UNTIL you understand the causal mechanism(s) and it’s actual contribution to the results.

Arbitrary improvement goals are doomed to failure because there is no connection to the resources needed to achieve that goal. So anytime I hear someone say the goal is X% improvement I know they are only giving lip service to goal setting and are relying on wishes, hopes and prayers that things get better (and maybe even a little manipulation of the data, the system or the way the goal is stated).

Real goal setting is done by:
  • Need (usually related to cost or Customer retention - set into the budget)
  • Or is based on knowing the Pareto of Problems that create the metric and how many projects/resources are available in the time frame
  • Or are based on simple continual improvement such as the Kaizen approach of lean (TPS).
Each should be monitored and verified by SPC to ensure that real improvements are made and you don’t just claim victory whenever random chance happens to give you a value you like. (Which is why the OP is doing).
All are directly connected to the resources needed to achieve the improvement.

There are two basic types of real improvement:
  • Discrete projects with large step wise improvements
  • A series of small continual improvements as used in Lean.
Small continual Kaizen improvement is not appropriate for physics / quality problems,
They are appropriate for a high quality stable operation.

IF the OP is relying on Kaizen THEN it is a proactive planned approach to achieving true continual improvement.

True Kaizen involves many small improvements initiated and implemented at the lowest level of the work and is therefore not something that Management should apply project management fro each action. They should get involved if there is no improvement over some time (6 months) as the lack of improvement is likely due to ineffective or no kaizen action…

*Note true Kaizen is a series of small improvement actions taken at the lowest level of work. I am not referring to the (unfortunately named) Kaizen Blitz (which is a bizarre mismatch of Japanese and German lingo) that refers to a weak long effort to completely overhaul a process…
 

Eddie74

Starting to get Involved
Maybe I'm dense, but I still don't understand what 10 month trend analysis is, or how it relates to 3 negative trend points.

The auditor reported On-Time-Delivery slipping in 2021 from 93% in December 2020 to 77% in December 2021. That's 12 months. Did you have no trends in there triggering management review and/or a corrective action? You don't want to be reacting to noise, nor do you want to miss a signal.

Maybe an example will help us understand.

The trend (not average) is established from ten previous data points, as defined by SPC software, Yes we did, primary results from the COVID impact on our internal organization, outside process, and suppliers delivery, seemed like everything was taking a hit. This was addressed in the Monthly meetings and discussed in the MR for 2021. The auditor brushed this aside and basically stated that our processes where not being measured due to not having a stated goal / target for each process. I cannot find in any Specification, we are API Q1 and ISO 9001:2015, a requirement for a process to have a stated goal / target in order for the process to be measurable.
 

Randy

Super Moderator
The auditor brushed this aside and basically stated that our processes where not being measured due to not having a stated goal / target for each process.

There isn't a requirement for each process, that's more bogus than a $3 bill. Remember, part of your monitoring/measuring is your internal audit.....Nobody ever thinks of that. Measuring also doesn't require a ruler, scale, meter or any numbers beyond 0/1 (binary yes or no). You can have it as simple or complicated as you want and $^rew what the auditor wants.......He's not part of the equation and when he tries to make himself so then he's no longer objective or impartial and you need to toss his fanny out the door right then and there and file a complaint about his conduct.
 

Tagin

Trusted Information Resource
The auditor brushed this aside and basically stated that our processes where not being measured due to not having a stated goal / target for each process. I cannot find in any Specification, we are API Q1 and ISO 9001:2015, a requirement for a process to have a stated goal / target in order for the process to be measurable.

It's not about the processes being measurable. It's about the quality objective being measurable.

Per 9001:2015 6.2.1 "The quality objectives shall...b) be measurable". That is, what is is required to be measurable is the objective (i.e., the target or goal).

In this case, the process is being measured, but there is no specified target or goal set to be measured. Hence the auditor's statement:
No measurable value for any of the quality objectives. The actual charts monitor the performance of each quality objective, but none of them have any measurable target or goal.
 

Bev D

Heretical Statistician
Leader
Super Moderator
Sorry I’m with Randy on this one. An objective of continual improvement can and is measurable and is a target. Keep getting better. That is a target. No standard says that it has to be a specific stated numerical value. (And if they did Deming would come back and slap the authors)
I’ve had many quality objectives that are stated as continual improvement and never had an auditor complain.
Some of my objectives have been stated that the organization will always be working on the top 5 Problems. Never had a finding.
 

Tagin

Trusted Information Resource
An objective of continual improvement can and is measurable and is a target. Keep getting better. That is a target.

Perhaps that was all the OP needed to do - to include in the definition of their QO something like "the goal of the QO is that the 10-month trend will maintain a positive slope". But absent a stated goal, the KPI tracking of processes is simply monitoring.

No standard says that it has to be a specific stated numerical value.

Who said it had to be numeric? Both TS9002 and APG guidance agree that 'measurable' can refer to either quantitative or qualitative methods.
 
Top Bottom