What do you do as an auditor, (internal, 2nd or 3rd party) whenever you talk to a process owner about thier turtle and they give you examples of how they measure the effectiveness of thier processes by giving you some metrics. The metrics are super weak. They are measuring what they say they will, and are meeting the mark....but what if it is ineffective for measuring what they intend. At what point does it become opinion. For instance, example given for HR process to measure/monitor training effectiveness is "% of 90 day review forms completed on time."

I ask them what their quality goals are, and how their metrics show performance in terms of their quality goals.

If their quality goals are "__ 90 day review forms completed on time" then I ask how this supports the business goals; what the information means to them; how it helps them do their jobs well, etc.

I have been known (internal auditor) to specifically coach new managers of immature systems in what that these metrics could look like: what could be measured, in what terms expressed, how the data might be collected, how it might be tracked and shared. (If they have no one else to get this information from, why not me? I'm a QA professional, not a meter maid.)

wow, great response. I'll be sure to follow that path. Anyone else?

wow, great response. I'll be sure to follow that path. Anyone else?You are very kind, but beware that this can be a slippery slope. If metrics don't exist and are required, they will need a CA and you ought not go far into their problem solving for them.

But people don't come from the egg knowing this stuff. It's no good to just tell them when they are doing it wrong. We can also point to information sources to teach how to do it well. We might even develop some internal training. :magic:

What do you do as an auditor, (internal, 2nd or 3rd party) whenever you talk to a process owner about thier turtle and they give you examples of how they measure the effectiveness of thier processes by giving you some metrics. The metrics are super weak. They are measuring what they say they will, and are meeting the mark....but what if it is ineffective for measuring what they intend. At what point does it become opinion. For instance, example given for HR process to measure/monitor training effectiveness is "% of 90 day review forms completed on time."

How does this metric link with the goals and objectives?

This metric is a measure of how well "review forms are completed", which could be OK, if it was a problem in the past. It says nothing about the effectiveness of the training or the quality of information found in the form. What is done with the review forms? Are they analyzed? Are action started, completed and verified?

Always ask the next questions.

What do you do as an auditor, (internal, 2nd or 3rd party) whenever you talk to a process owner about thier turtle and they give you examples of how they measure the effectiveness of thier processes by giving you some metrics. The metrics are super weak. They are measuring what they say they will, and are meeting the mark....but what if it is ineffective for measuring what they intend. At what point does it become opinion. For instance, example given for HR process to measure/monitor training effectiveness is "% of 90 day review forms completed on time."

I'd hope not to ask about a turtle - unless they keep them as pets. If your organization has chosen to document processes using them, that's the mistake IMHO, not having weak metrics. That's just a symptom. I'm guessing management don't understand the application of the turtle to their real-life situation, were probabaly not given much opportunity to buy into the concept so 'went along'. Their process metrics must be based on what affects customers, regulatory compliance and business performance as a whole - not, as I suspect, something trumped up to fit the use of a turtle.

What do you do as an auditor, (internal, 2nd or 3rd party) whenever you talk to a process owner about thier turtle and they give you examples of how they measure the effectiveness of thier processes by giving you some metrics. The metrics are super weak. They are measuring what they say they will, and are meeting the mark....but what if it is ineffective for measuring what they intend. At what point does it become opinion. For instance, example given for HR process to measure/monitor training effectiveness is "% of 90 day review forms completed on time."

Is this an hypothetical question/example or an actual situation?

Because I read your other posts...is there an actual audit going on?

Stijloor.

This particular question is from an internal audit. Other questions I have curreently in other threads are there because I'm having a brainstorm of questions I keep forgetting to ask.
And yes, stage 2 TS audit going on right now.

This particular question is from an internal audit. Other questions I have curreently in other threads are there because I'm having a brainstorm of questions I keep forgetting to ask.
And yes, stage 2 TS audit going on right now.

Thanks. You're getting excellent answers from my Fellow Covers!
Will you share the responses to your inquiries with Managers? ;)
Just curious....

How's the audit going?

Stijloor.

I'd hope not to ask about a turtle - unless they keep them as pets. If your organization has chosen to document processes using them, that's the mistake IMHO, not having weak metrics. That's just a symptom. I'm guessing management don't understand the application of the turtle to their real-life situation, were probabaly not given much opportunity to buy into the concept so 'went along'. Their process metrics must be based on what affects customers, regulatory compliance and business performance as a whole - not, as I suspect, something trumped up to fit the use of a turtle.

Not often I question you, but I can't see what prevents metrics expressed in a turtle leg from being good ones.

I'm no fan of turtles or RYG dashboards for that matter, but these are just tools.

I would guess that management just doesn't understand - period, and just wanted a plak on the wall.

Would you care to expand on metrics-turtles, what would be a better way?

What do you do as an auditor, (internal, 2nd or 3rd party) whenever you talk to a process owner about thier turtle and they give you examples of how they measure the effectiveness of thier processes by giving you some metrics. The metrics are super weak. They are measuring what they say they will, and are meeting the mark....but what if it is ineffective for measuring what they intend. At what point does it become opinion. For instance, example given for HR process to measure/monitor training effectiveness is "% of 90 day review forms completed on time."

Keep from interjecting too much opinion by continually unpeeling the onion. Simply:

"This tells me that the managers are following the procedure, completing their 90 reviews on time. Can you show me how you measure the effectiveness of the training system in general?"

If they can't - if there are no other metrics - you don't have anything to work with, and the organization obviously hasn't monitored the effectiveness of the training system. There should be something stated in their procedures about how they measure effectiveness and what the criteria is. If theirs is weak, give them guidance on measuring effectiveness. Consider nonconformities in other areas that relate to people following procedures - could be evidence from their own CAPA system, audit program, or your own audit observations.

I really try to dissuade companies from using processing time as an effectiveness measure. It tells you nothing, except that people are doing "something" in the time frame required by procedures. It doesn't tell you anything about that "something's" accuracy or effectiveness.

Show them things they can use to measure effectiveness of each system, by demonstrating how the processes interrelate. Measuring the specific performance of a particular system the way that they do, demonstrates that they possibly don't understand how the processes interrelate.

In other words, training effectiveness can be measured by testing, on-the-job training, performance reviews, nonconforming product, manufacturing scrap, internal audit results, inputs into the CAPA system, etc.