Relationship between ANOVA and measurement accuracy - Mechanical strenth of a closure

[davis007]

We recently had an issue with a batch of parts. One critical factor, a mechanical strenth of a closure was out of specification. We worked on the issue and got the the overall average within spec We ship and certify the average so technically this is "OK". I pulled 240 parts from the next run, 30 per cavity (8 cavity tool), and ran the test again. ANOVA below to me indicates that cavity (part) A is higher than the rest, I did run some parwise comparisons and get the same result.

When I shared this with the production manager he told me that that there was to much variability in the test method to draw this conclusion. I thought that was the whole point of t tests and ANOVA to help ensure that the differnces seen were "real" and NOT due to measurement or other issues not related to the factors you are studying.

Am I off my rocker?

One-way ANOVA: opening force versus PART

Source DF SS MS F P
PART 7 1.0607 0.1515 2.42 0.021
Error 232 14.5474 0.0627
Total 239 15.6080

S = 0.2504 R-Sq = 6.80% R-Sq(adj) = 3.98%

The graph is not formating correctly so I will leave it out. But it shows that the A cavity is higher thn the rest although the CI do overlap.

 

[Miner]

Could you attach your data? While the p value seems to indicate significance, the R-squared (adj) is extremely low.

 

[davis007]

Data attached.

 

[Statistical Steven]

I thought that was the whole point of t tests and ANOVA to help ensure that the differnces seen were "real" and NOT due to measurement or other issues not related to the factors you are studying.

Am I off my rocker?

Are you off your rocker? Probably not, but you are confusing some things. ANOVA and t-tests compare the variability between the 8 cavity means to the average variability within a cavity. If the cavity to cavity variability is large, then we say that it is significant. You can then do different multiple comparison procedures to determine which cavity is different.

The production manager comments are more about the test method to obtain the closure readings. You can take the square root of the MSE (S=0.25) and divide that by the overall mean of the 240 parts. This will give you a CV. If the CV is larger than 10%, the measurement system might be too variable.

Remember, the MSE is suppose to be the variability between parts, but it also includes variability of the measurement system.

Having said all that, if the measurement system is too variable, it usually means you will NOT see a difference when there is a difference in the cavities. Since you saw a significant difference in cavities, it probably is real.

 

[Tim Folkerts]

Re: Relationship between ANOVA and measurement accuracy - Mechanical strenth of a clo

I agree that the mean of "A" is higher than the mean by statistically significant amount. I get the same numbers you do - no surprise! I'm not sure just what you could do, but it might be worth checking into why "A" might be higher.

Steven had an interesting point about variations within the measurement system itself, rather than in the parts. Woudl it be possible to repeatedly measure a single part? Does the mechanical strength diminsh after 30 tests, or would it stay about the same? If you did test the same item 30 times, then you could estimate the variability due the the measurement system itself. (As long as the slope of strength vs trial # isn't significantly different from 0, the strength isn't changing.)


Tim F

 

[Bev D]

I took a quick look at the data: there is probably NOT a true difference. yes the value was below .05, but the r SQ value was so low indicating a that IF a difference does exist it's very small and cavity to cavity differences are not the primary cause of the variation. So you could 'fix' the cavity thing and end up not improving yoru performance one bit because of the effect of the pareto of X's.

I have attached a JMP output (as a picture in powerpoint) that shows the variation (ALWAYS plot your data, look at your data and THINK about your data...summary statistics - while powerful - can be very misleading)

a few other points to consider:
1. you have 8 cavities - a low p value could simply be "alpha error". remember alpha is the risk that you 'detected' a difference when one doesn't actually exist.
2. Your production manager may be spot on: IF you have a lot of measurement error you coudl get this result as well. A real difference may exist but the error is so large you can't see the difference. You comment concerning ANOVA, and t tests 'overcoming' measurement is valid but ONLY if the results for each level of the factor are separated a great distance (very little to no overalp in the results). So my recommendation is to perform a measurement study adn look for the measurement error vs the actual variation you are getting...

 

[davis007]

Tim:

You hit the nail on the head. The reason for the analysis was to see if we had room to imporve things further. One question we all had was is there a difference between cavities. The data I shared telles me there is and I suggested as much and several possible reasons for this along with ways to check if any of these were true. But was met with the orginal resistance that you can not see that because the test is to variable. (How he nows that the test is variable I have no idea as he has never run it.) I being new to some of this was not prepared to respond other than my gut feeling that the analysis "accounted" for this posibility and that you might miss a difference when one exist if test variability is to high but certianly should be less likely to see one were it it not real. (Thanks to Steven for saying that a little better than I could.)

Regarding retesting the parts over and over to gauge the repeatabiliyt of the test - good idea and I will. I belive that we will see some degradation fohte closure strength over time but this is apart that is designed for daily use so over 30 - 50 openings I would expect little change.

I am just trying to have all my "facts" straight before I take another run at the produciton department on checking the cavity variation.

 

[davis007]

BevD thanks for the other side of the story. I have done pairwise comparisons and they do indicate a difference.

I have also looked at the plotted data and there is considerable overlap of the data as you point out. BUT there was one other thing that really caught my eye and you would not see with out further explination.


These parts are produced on a two shot machine. One material is molded in four cavites onto four cold cores. These four cores are then rotated to mate with four other cavities and the second material is molded. Thus really we have 8 cores that alternate between two sets of four cavities.

Parts A the highest average and H the lowest average arr produced on two different cores. BUT they use the same cavities for the first and second injection. I find it hard to belive that random noise from a test method would conspire to give me a false positive for effect of cavity and the LOWEST and HIGHEST averages be paired in this way. Not impossible but seems a bit coincidental.

Thanks for reminding me about alpha error. I was trying to remember what that error was called.

 

[Statistical Steven]

I took a quick look at the data: there is probably NOT a true difference. yes the value was below .05, but the r SQ value was so low indicating a that IF a difference does exist it's very small and cavity to cavity differences are not the primary cause of the variation. So you could 'fix' the cavity thing and end up not improving yoru performance one bit because of the effect of the pareto of X's.

Can you please explain to mean the meaning of R-squared in the ANOVA context? I can understand it for regression analysis,but I am unable to comprehend its meaning when the levels are categorical, and their spacing has no meaning.

I have attached a JMP output (as a picture in powerpoint) that shows the variation (ALWAYS plot your data, look at your data and THINK about your data...summary statistics - while powerful - can be very misleading)

a few other points to consider:
1. you have 8 cavities - a low p value could simply be "alpha error". remember alpha is the risk that you 'detected' a difference when one doesn't actually exist.
This is true, but the the p-value for ANOVA is an omnibus test that looks for any difference without a statement of which levels are different. The alpha error would come from doing the 28 pairwise comparisons and seeing one of them being significant at alpha=0.05

2. Your production manager may be spot on: IF you have a lot of measurement error you coudl get this result as well. A real difference may exist but the error is so large you can't see the difference. You comment concerning ANOVA, and t tests 'overcoming' measurement is valid but ONLY if the results for each level of the factor are separated a great distance (very little to no overalp in the results). So my recommendation is to perform a measurement study adn look for the measurement error vs the actual variation you are getting...

BINGO, without a measurement study, it is very difficult to know if the measurement system is an issue. Though the Square root of the MSE divided by the mean gives only a 6% CV, so it is unlikely that the measurement system is way off.

 

[Bev D]

BINGO, without a measurement study, it is very difficult to know if the measurement system is an issue. Though the Square root of the MSE divided by the mean gives only a 6% CV, so it is unlikely that the measurement system is way off.

Maybe - but I've not seen %CV be a reliable indicator of measurement error (and my current company makes measuring instruments for animals. I've seen horrible measurement error with %CVs less than 10%. ) That's why he needs to do the measuremetn study. It's the 3% r-sq that tells me he's chasing butterflies.