MSA on set of parts with low variation

[Risca]

Dear Forum,

I am working on implementing a new measurement equipment (image measurement system from Keyence) for in-process control of parts used as consumables in an in-vitro diagnostics device. We have the obligation to have an external notified body.

The parts in question are injection moulded CD-like substrates. There are mainly three critical parameters (outer diameter, inner diameter and the outer to inner diameter circular runout) that all are measured with this new, image measurement device.

The manufacturing process is very capable, and the variation of these parameters is very low, e.g., the inner diameter only varied 0,004 mm on samples sampled from 6 different batches manufactured over 6 months. This is not a good starting point for MSA, as you most certainly know. The problem is, I cant manipulate these parameters very easy. The cricular runout, yes, a tiny bit, but the ID and OD is nearly impossible to change without having to manufacture new parts for the injection moulder or change the raw material to a plastic that shrinks more. But, regardless of the stable process, we must monitor these parameters.

I attach a excel-file with som preliminary test results (only two appraisers). The range between measurement on the same parts for the different parameters is in line with the expected masurement error specified by the manufacturer.

So now the question. Any ideas how to validate this measurement method correctly? I am out of ideas, besides trying to make some kind of bad rationale on why we approve the test method only based on the low P/ Tol values and low range between measurements. If I adjusted the specification limits using the probable error method, maybe thats enough "to cover for the bad TMV"?

Thank you,

 

[Matt's Quality Handle]

The MSA will output two numbers, which give answers to very different questions.

Precision to Total Ratio answers the question "Is this measurement system good enough to control my process within the expected process variation?" Or in other words, "Is this measurement good enough to use for SPC and the associated rules?" The answer to this question is no.

Precision to Tolerance Ratio answers the question "Is this measurement system good enough to control my process within the specification limits?" In other words, "Is this measurement system good enough to tell good parts from bad parts?" The answer to this question is yes.

So that goes back to your initial question:

Dear Forum,

I am working on implementing a new measurement equipment (image measurement system from Keyence) for in-process control of parts used as consumables in an in-vitro diagnostics device. We have the obligation to have an external notified body.

If by "in-process control," you're talking about applying to SPC and control charts, then it is not sufficient. But if the use is to flag out of spec parts in a sampling plan, then it is sufficient.

You talk about an external body. Unfortunately, my real world experience with this is in automotive industry. I've run into this with SQE's who don't think this critically, and couldn't (wouldn't?) understand this. They would just apply the standard that everything has to be less than 30%, regardless of how we were using it. So it was frequently a fight.

 

[Miner]

If you do find yourself in a situation where the measurement system is not acceptable for its intended use (i.e., inspection or SPC), you may have a number of options beyond improving the measurement system or purchasing a new system. Not all of these are feasible in a high volume situation.

• Limit use of the measurement system to a single person, eliminating the Reproducibility variation. While not feasible for manufacturing, it is often very feasible for laboratory equipment.
• Take multiple measurements and average them. Use the average as the measurement. This will reduce the measurement variation by the square root of the sample size (e.g., the average of a sample of 4 will have half the variation). Again, this may not be feasible for high volumes, but is usually feasible for low volumes and in laboratory situations.

 

[Ron Rompen]

Just to think outside the box (and I'm not sure if this would be acceptable or not), do you manufacture other parts with similar features which DO show a greater range of variation? Try sampling those, using the SAME tolerance limits as your current parts, and see what kind of results you get. In my mind, this would satisfy the requirements of demonstrating the discrimination (accuracy/precision) of your measuring instrument, while giving you enough variation between parts to have a valid study.

 

[Risca]

Dear Matt's Quality Handle,

Thank you for your quick reply that really answered my question. I have had trubbel to find a good definition to the P / Tot value and therefor to really understand what the ratio really tells me. The specification limits were set before the process were completely developed, so the limits does not correspond to the expected process variation, but are set to avoid faulty products due to dimension chain error.

With in-process control, I refer to verifying that parts are within the limits, or not. In other words, the test method is good enough for that purpose. I do, however, use the historical data to see if the process is stable and how it behaves over time, just to learn more about wear of components and to see if the seasonal humidity fluctuation influence the process. That is the process developer in me who cares for my baby (my dear process..). I do not use it for SPC today, but would hope to be able to further down the road.

We are a fast growing company (from 150 to 200 employees during 2021), and our QA/RA department have had difficulties to grow with the same rate, so they are a bit understaffed. They lacks a good statistician for example. I am working for the development department, so the QA/RA is not really on my table (but I enjoy working with it from time to time!). Unfortunately, my situation is a bit like your experience from automotive industry. Managers that comes from other industries that just applies their standard standards. Standards that barely is sufficient for manufacturing brooms. Then comes our notified body TÜV and scrutinizes our processes and I am the one standing there, ashamed. In other words, our SOPs for different validation and qualification activities are not exhaustive and I am trying to implement new things all the time. One thing I have had difficulties to motivate is the adjustment of specification limits due to measurement error. I can't understand why my colleagues don't understand the importance of it. I have earlier used a method similar to the Watershed limits and probable error described by Wheeler ( I can not post link, but google specifications and Measurement Error spc excel if interested). My manager only dismisses it with the argument that we will have to discard more products! Stupid..

What would you do? When is it justified to tighten the specification limits in my case? I mean, If you have a look at my data set, there is an obvious measurement error (and of course, no measurement gauge is perfect, right!). Yes, my process i capable. But what If the next punch (that creates the center hole of my component in the injection moulder) is slightly larger and the ID is comes closer to the upper limit? The risk of misclassification may become significant. I don't want to be the cause of a faulty products that miss-diagnose patients at the hospital!


Thank you again, at least now I have some better arguments for my TMV report!

 

[Risca]

If you do find yourself in a situation where the measurement system is not acceptable for its intended use (i.e., inspection or SPC), you may have a number of options beyond improving the measurement system or purchasing a new system. Not all of these are feasible in a high volume situation.

• Limit use of the measurement system to a single person, eliminating the Reproducibility variation. While not feasible for manufacturing, it is often very feasible for laboratory equipment.
• Take multiple measurements and average them. Use the average as the measurement. This will reduce the measurement variation by the square root of the sample size (e.g., the average of a sample of 4 will have half the variation). Again, this may not be feasible for high volumes, but is usually feasible for low volumes and in laboratory situations.

Thank you for your reply. That was an interesting input. We measure quite a lot of components, but the new measurement system I bought is very quick and automated. To make it automatically measure the same sample several times and present the average is definitely a possibility! I will look into that! Good idea!

 

[Risca]

Just to think outside the box (and I'm not sure if this would be acceptable or not), do you manufacture other parts with similar features which DO show a greater range of variation? Try sampling those, using the SAME tolerance limits as your current parts, and see what kind of results you get. In my mind, this would satisfy the requirements of demonstrating the discrimination (accuracy/precision) of your measuring instrument, while giving you enough variation between parts to have a valid study.

I welcome even outside the box suggestions!
We do manufacture similar parts. Unfortunately, the same issue with dimensions with low variation there as well. Since it is 1 mm thick polystyrene discs, I tried to mill them in one of our milling machines. But the precision in work piece set up and milling operation is a bit to poor. But, If I mill 100 discs at least 10 would be within limits. I guess this is the way I have to go, even though it is quite time consuming! To get the process up and running I will do with the current samples with low variation to show OK P / Tol and then continue with a new test method validation with"better" samples to be able to use the test method for SPC.

Thank you Ron for taking time. I am so impressed by all you people out there helping strangers on forums without benifying on it. True everyday heroes!

 

[Miner]

Thank you for your reply. That was an interesting input. We measure quite a lot of components, but the new measurement system I bought is very quick and automated. To make it automatically measure the same sample several times and present the average is definitely a possibility! I will look into that! Good idea!

If the test is fully automated, run a Type 1 Repeatability study instead of a full R&R study.

 

[Risca]

If the test is fully automated, run a Type 1 Repeatability study instead of a full R&R study.

I was originally planing on that. But the validation leader wanted me to make the TMV a part of the software validation. Since you program the measurement programs for the system yourself, he wanted me to validate each program separately. My problem then was how to determine the true dimensions of the part to be measured? To use calibrated rings or gage blocks is not possible in that case. And to send some samples to e.g., Mitutoyo fore reference measurement would take 3 months.. Times I don't have!

I think I will use the input from Matt's to complete this first TMV, and then investigate the possibilities to fabricate samples, somehow, that coves the entire specification range.

 

[John Predmore]

Many optical dimensional measurements are based on best fit circles, which are internally-calculated average of a sample of locations around the circumference. If you had a disk with a high spot on the outer diameter, compared to a disk with no high spot, you might barely see a difference using a best fit circle algorithm. Any time you average a number of sample readings, the standard deviation of the mean is reduced by the square root of the number of samples. You may want to look at the minimum circumscribed circle for the OD, which will be bigger than the best-fit diameter, although the center of the circumscribed circle may be offset.

What you call circular runout, do you want the largest minus the smallest radial distance? If you subtract the maximum inscribed circle from the minimum circumscribed circle, I think that would generally be bigger than the largest radial distance, unless the largest OD and smallest ID line-up. With automatic inspection, you have to think about what the software is actually doing with the measurements and compare that to the design intent you want to study, and what is the purpose of the study. Is it to qualify the procedure, to monitor variation, or to improve the system? I am just trying to offer outside-the-box thinking.