PPM Defect Rate as a Performance Metric - Are there Flaws?

[leftoverture]

Our company, as with many others, has historically tracked our quality performance largely by tracking our PPM (parts per million). I see PPM as a flawed metric, though, for two primary reasons:

1) It is very dependent on how many parts you actually produce and/or ship. So if you have a low shipping month, for instance, a relatively small rejection can result in an unusual spike in the metric.

2) It is not cost-related. As a job shop, we have many high-volume, low cost products that tend to influence the metrics more than the lower volume, high cost products.

Coming up with a better metric probably isn't that hard, but measuring PPM is pretty much expected by our customers and can be a hard sell to those for whom PPM "is they way we've always done it".

I would be interested in knowing if anyone else has found the same flaws in the PPM metric and how have you approached it?

Thanks!

 

[hogheavenfarm]

This problem plagues other metrics too. Some of our customers require ppm, others do not, but regardless what measure you utilize, there will be things that skew it. When I was in extrusion, we had a better handle on this as the machines ran 24/7 at a specific rate, the ppm here was actually pretty valid. Now we manufacture to print, maybe one item, maybe 30, maybe 1000. This is where skews become evident. When possible, use what best reflects your particular reality, when required to show ppm, I add a disclaimer, "*ppm may not reflect historical data from this product line". This always generates questions which show that a particular product line may have very good defect numbers when measured against itself. Of course, you had better be prepared and have historical data handy (and better than the ppm). In the opposite case I leave well enough alone.

 

[Sebastian]

I see PPM as a flawed metric, though, for two primary reasons:
1) It is very dependent on how many parts you actually produce and/or ship. So if you have a low shipping month, for instance, a relatively small rejection can result in an unusual spike in the metric.

"Dependent" sounds negative here. Why? Counting rejected pieces irrespectively of production volume is a wrong idea. Comparing reject value to production volume, as PPM does, allows comparison between production lots. It is fair. PPM bases on assumption that process has some nonconforming product ratio (process capability), which results in increase of rejected quantity proportional to production quantity increase.

2) It is not cost-related. As a job shop, we have many high-volume, low cost products that tend to influence the metrics more than the lower volume, high cost products.

In previous company I worked for, I had started from PPM metric and then moved to cost, as 10 rejected pieces of Product A resulting in 1234 PPM has higher cost than 1200 rejected pieces of Product B resulting in 5678 PPM, who would be normally No.1 target for improvement.
Reject cost is better, but PPM is not that bad.

 

[Bev D]

There isn't really any problem with a direct ppm calculation*. Ppm is just like percentages except the normalized base is a million instead of a hundred. The issue most people encounter comes from the non-homogenous distribution of defects in the process stream. There are 2 types of defect distributions: one is a random, homogenous distribution of defects (such as from a common cause system) and the other is a clustering of defects from an assignable cause such as a set up error or a defective batch of raw material. The latter creates a spike in defects. If your denominator for ppm comes from a single lot you will show a fairly large spike in ppm. Which is completely accurate for that short period of time but is not representative of long term defect rates IF you correct the assignable causes AND put permanent preventive solutions in place.

An additional issue is that Customers and Suppliers have difficulty separating manufactured defect rates and shipped defect rates. Both parties really need to understand the various effects of these two different things.

As for cost remember that a even an inexpensive part (for you) can be an expensive failure for your customer...


*There are major issues with a theoretical ppm calculation based on a distributional model such as with Cpk indices - these are usually just bogus made-up numbers....

 

[leftoverture]

"Dependent" sounds negative here. Why? Counting rejected pieces irrespectively of production volume is a wrong idea. Comparing reject value to production volume, as PPM does, allows comparison between production lots. It is fair.

A few reasons it is negative. 1) we are a job shop and make many different parts for many different customers. If I look at it for just one part number, it is a good measure of our defect rate, but across many different part numbers it starts to get skewed and loses it's significance.

2) Our quality incentive is based on our return rate. Ours is very, very low compared to most companies, but still, a return of 500 pieces could have varying impact on our incentive depending on how many parts we shipped that month.

3) We do monitor our company wide PPM, both internal and external, and we track trends, but if customer A orders part B which has a very high volume, it will cause a short term skew in the data. And if customer A is slow and doesn't order part B, the PPM will likewise artificially go up.

So, in the end, it is an imperfect measure. We understand the pitfalls, and try to interpret the data accordingly, but I still would like to find a better way (since continual improvement is practically our middle name here!). So I was just wondering if others have found a better way of measuring defect rates.

 

[Sebastian]

It seems to me, that nevertheless various products are coming from a same manufacturing process, for whom one common PPM value is calculated, there are dramatic differences between products individual PPM values.
Normally, your manufacturing process design process activities shall build process without significant differences, but somehow it failed in some cases. Additionally probably during designed process validation there was not noticed performance drop for some products and you put them all in one bag and calculated common PPM. Maybe someone had noticed that, but it did not change anything anyway.
I would suggest for "bad boys" identified during process validation, setting individual PPM targets and then monitoring them separately during serial production.

 

[hogheavenfarm]

But that is the issue, the products do not come from the same manufacturing process. When your product line contains everything from delicate electronic devices to armor plate, there are vastly different processes involved. We have one product with a 30% fallout rate. There is little we can do to improve it, (and I always try every time we run it), since the product is customer material, cut to customer prints (which we are not allowed to even see), and given a customer supplied check spec. It either falls within tolerance or not. The customer pays for all the material as well. When we run this, our internal PPM goes off the chart, and sometimes borderline cases like this escape, causing our external PPM to spike as well. Other products can be run by the thousands with virtually no defects at all. Yes , the PPM is an accurate average overall, but is not useful for determining capability of a particular process or product line. I cringe when customers simply want to see our "PPM defect rate" to determine if we are "good enough" to do business with.

 

[leftoverture]

But that is the issue, the products do not come from the same manufacturing process. When your product line contains everything from delicate electronic devices to armor plate, there are vastly different processes involved. We have one product with a 30% fallout rate. There is little we can do to improve it, (and I always try every time we run it), since the product is customer material, cut to customer prints (which we are not allowed to even see), and given a customer supplied check spec. It either falls within tolerance or not. The customer pays for all the material as well. When we run this, our internal PPM goes off the chart, and sometimes borderline cases like this escape, causing our external PPM to spike as well. Other products can be run by the thousands with virtually no defects at all. Yes , the PPM is an accurate average overall, but is not useful for determining capability of a particular process or product line. I cringe when customers simply want to see our "PPM defect rate" to determine if we are "good enough" to do business with.

Exactly! Glad to see I am not the only one who has this battle.

Sent from my LG-TP260 using Tapatalk

 

[Golfman25]

A few reasons it is negative. 1) we are a job shop and make many different parts for many different customers. If I look at it for just one part number, it is a good measure of our defect rate, but across many different part numbers it starts to get skewed and loses it's significance.

2) Our quality incentive is based on our return rate. Ours is very, very low compared to most companies, but still, a return of 500 pieces could have varying impact on our incentive depending on how many parts we shipped that month.

3) We do monitor our company wide PPM, both internal and external, and we track trends, but if customer A orders part B which has a very high volume, it will cause a short term skew in the data. And if customer A is slow and doesn't order part B, the PPM will likewise artificially go up.

So, in the end, it is an imperfect measure. We understand the pitfalls, and try to interpret the data accordingly, but I still would like to find a better way (since continual improvement is practically our middle name here!). So I was just wondering if others have found a better way of measuring defect rates.

Since you're a job shop and have spikes and variation, you might try using a rolling average. That will help smooth everything out.

 

[hogheavenfarm]

We do this internally (rolling avg), but unfortunately most customers want us to report PPM for the current year to date. Its hard to make them understand that there is a difference between product lines. I may start separating the lines into categories, and then just report PPM on those lines. We'll see. Currently PPM is 3,900 , but can take a big tumble if certain product lines come up.