A Case Study - Flaws in a performance measure report

[Steve Prevette]

This is a paper I wrote many years ago, hopefully so many years ago that it does not point a finger at any currently guilty parties. This was a case study I wrote up pointing out flaws in a performance measure report being generated by the Department of Energy. This is along the lines of my previous "Liars Figure and Figures Lie", but with a real-world example.

 

[Craig H.]

Steve:

I like how you are able to take what can be a confusing issue and make it straightforward, and in a relatively small amount of space. Nice article.

Would you expand a little on how you decided where the breaks were in the final chart?

 

[Wes Bucey]

Please repost the piece with your name attached (other than in "file properties.")

 

[Steve Prevette]

Please repost the piece with your name attached (other than in "file properties.")

Good point. File has been updated, thanks.

 

[Steve Prevette]

Steve:

I like how you are able to take what can be a confusing issue and make it straightforward, and in a relatively small amount of space. Nice article.

Would you expand a little on how you decided where the breaks were in the final chart?

Probably this subject would make a good article. Here is the short version, generic to any control chart:

1. Make a set of average and control limits using the first 25 points (or all points if there are less than 25).

2. If there are any statistically significant trends within the baseline:

a. If there are only a few points in the pattern, and the data appears to return to the baseline, throw those points out of the average and baseline.

b. If it looks like there is a permanent shift, end the previous baseline prior to the shift, and start a new baseline after the shift.

3. If there are statistically significant trends after the baseline time interval, and it appears to be a permanent shift, start a new baseline using the first 25 points following the shift.

4. Keep working through the data from oldest to newest until you get to the current datum.

As a criteria for "permanent shift", I use the "MW" rule - if the data make 3 changes of direction, then there are enough data for a new baseline.

This process works well for me. There are other variations though.

 

[Bill Pflanz]

Steve,

Nice use of a control chart for a non-manufacturing process. I have done similar control charts for safety data but always used a 3 month moving average control chart (per Grant & Leavenworth Statistical Quality Control).

If all you have is quarterly data spread over 5 years than the moving average chart or the u chart may be difficult to interpret since it is possible that the process has changed. If the process has changed, then you would continue to use the control limits until enough new data is collected for the new stable process.

Since the u chart is used in cases where the samples are of different size it is my understanding that each sample value must be plotted within its own u chart limits if the sample size varies significantly. Are you assuming that the number of individuals in the study has remained relatively constant? If the sample size is constant than I think a moving average chart is appropriate to use and the control limits changed using the rules you described.

Bill Pflanz

 

[Steve Prevette]

Steve,

Nice use of a control chart for a non-manufacturing process. I have done similar control charts for safety data but always used a 3 month moving average control chart (per Grant & Leavenworth Statistical Quality Control).

If all you have is quarterly data spread over 5 years than the moving average chart or the u chart may be difficult to interpret since it is possible that the process has changed. If the process has changed, then you would continue to use the control limits until enough new data is collected for the new stable process.

Since the u chart is used in cases where the samples are of different size it is my understanding that each sample value must be plotted within its own u chart limits if the sample size varies significantly. Are you assuming that the number of individuals in the study has remained relatively constant? If the sample size is constant than I think a moving average chart is appropriate to use and the control limits changed using the rules you described.

Bill Pflanz

I definitely strongly disagree with the use of moving averages. We want a fixed baseline average to compare the data to in order to see if there is a trend. The moving average would hide that.

Yes, I do take into account the hours that go into the u chart denominator, and they are factored into the control limits = average + 3 times the square root of (the average divided by hours times 200,000). The control chart itself (with non-moving average) gives you the signal of when the data has changed. You can then see what process changes that signal correlates to.

Oddly enough, people are so used to control charts here in the safety world, that when I approach Operations with offers to make control charts, their response is "That works in safety, but won't work in Operations". Go figure.