SBS - The best value in QMS software

Relationship between ANOVA and measurement accuracy - Mechanical strenth of a closure

D

davis007

#1
We recently had an issue with a batch of parts. One critical factor, a mechanical strenth of a closure was out of specification. We worked on the issue and got the the overall average within spec We ship and certify the average so technically this is "OK". I pulled 240 parts from the next run, 30 per cavity (8 cavity tool), and ran the test again. ANOVA below to me indicates that cavity (part) A is higher than the rest, I did run some parwise comparisons and get the same result.

When I shared this with the production manager he told me that that there was to much variability in the test method to draw this conclusion. I thought that was the whole point of t tests and ANOVA to help ensure that the differnces seen were "real" and NOT due to measurement or other issues not related to the factors you are studying.

Am I off my rocker?

One-way ANOVA: opening force versus PART

Source DF SS MS F P
PART 7 1.0607 0.1515 2.42 0.021
Error 232 14.5474 0.0627
Total 239 15.6080

S = 0.2504 R-Sq = 6.80% R-Sq(adj) = 3.98%

The graph is not formating correctly so I will leave it out. But it shows that the A cavity is higher thn the rest although the CI do overlap.
 
Elsmar Forum Sponsor

Miner

Forum Moderator
Staff member
Admin
#2
Could you attach your data? While the p value seems to indicate significance, the R-squared (adj) is extremely low.
 

Statistical Steven

Statistician
Staff member
Super Moderator
#4
davis007 said:
I thought that was the whole point of t tests and ANOVA to help ensure that the differnces seen were "real" and NOT due to measurement or other issues not related to the factors you are studying.

Am I off my rocker?
Are you off your rocker? Probably not, but you are confusing some things. ANOVA and t-tests compare the variability between the 8 cavity means to the average variability within a cavity. If the cavity to cavity variability is large, then we say that it is significant. You can then do different multiple comparison procedures to determine which cavity is different.

The production manager comments are more about the test method to obtain the closure readings. You can take the square root of the MSE (S=0.25) and divide that by the overall mean of the 240 parts. This will give you a CV. If the CV is larger than 10%, the measurement system might be too variable.

Remember, the MSE is suppose to be the variability between parts, but it also includes variability of the measurement system.

Having said all that, if the measurement system is too variable, it usually means you will NOT see a difference when there is a difference in the cavities. Since you saw a significant difference in cavities, it probably is real.
 

Tim Folkerts

Super Moderator
#5
Re: Relationship between ANOVA and measurement accuracy - Mechanical strenth of a clo

I agree that the mean of "A" is higher than the mean by statistically significant amount. I get the same numbers you do - no surprise! I'm not sure just what you could do, but it might be worth checking into why "A" might be higher.

Steven had an interesting point about variations within the measurement system itself, rather than in the parts. Woudl it be possible to repeatedly measure a single part? Does the mechanical strength diminsh after 30 tests, or would it stay about the same? If you did test the same item 30 times, then you could estimate the variability due the the measurement system itself. (As long as the slope of strength vs trial # isn't significantly different from 0, the strength isn't changing.)


Tim F
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#6
I took a quick look at the data: there is probably NOT a true difference. yes the value was below .05, but the r SQ value was so low indicating a that IF a difference does exist it's very small and cavity to cavity differences are not the primary cause of the variation. So you could 'fix' the cavity thing and end up not improving yoru performance one bit because of the effect of the pareto of X's.

I have attached a JMP output (as a picture in powerpoint) that shows the variation (ALWAYS plot your data, look at your data and THINK about your data...summary statistics - while powerful - can be very misleading)

a few other points to consider:
1. you have 8 cavities - a low p value could simply be "alpha error". remember alpha is the risk that you 'detected' a difference when one doesn't actually exist.
2. Your production manager may be spot on: IF you have a lot of measurement error you coudl get this result as well. A real difference may exist but the error is so large you can't see the difference. You comment concerning ANOVA, and t tests 'overcoming' measurement is valid but ONLY if the results for each level of the factor are separated a great distance (very little to no overalp in the results). So my recommendation is to perform a measurement study adn look for the measurement error vs the actual variation you are getting...
 

Attachments

Last edited:
D

davis007

#7
Tim:

You hit the nail on the head. The reason for the analysis was to see if we had room to imporve things further. One question we all had was is there a difference between cavities. The data I shared telles me there is and I suggested as much and several possible reasons for this along with ways to check if any of these were true. But was met with the orginal resistance that you can not see that because the test is to variable. (How he nows that the test is variable I have no idea as he has never run it.) I being new to some of this was not prepared to respond other than my gut feeling that the analysis "accounted" for this posibility and that you might miss a difference when one exist if test variability is to high but certianly should be less likely to see one were it it not real. (Thanks to Steven for saying that a little better than I could.)

Regarding retesting the parts over and over to gauge the repeatabiliyt of the test - good idea and I will. I belive that we will see some degradation fohte closure strength over time but this is apart that is designed for daily use so over 30 - 50 openings I would expect little change.

I am just trying to have all my "facts" straight before I take another run at the produciton department on checking the cavity variation.
 
D

davis007

#8
BevD thanks for the other side of the story. I have done pairwise comparisons and they do indicate a difference.

I have also looked at the plotted data and there is considerable overlap of the data as you point out. BUT there was one other thing that really caught my eye and you would not see with out further explination.


These parts are produced on a two shot machine. One material is molded in four cavites onto four cold cores. These four cores are then rotated to mate with four other cavities and the second material is molded. Thus really we have 8 cores that alternate between two sets of four cavities.

Parts A the highest average and H the lowest average arr produced on two different cores. BUT they use the same cavities for the first and second injection. I find it hard to belive that random noise from a test method would conspire to give me a false positive for effect of cavity and the LOWEST and HIGHEST averages be paired in this way. Not impossible but seems a bit coincidental.

Thanks for reminding me about alpha error. I was trying to remember what that error was called.
 

Statistical Steven

Statistician
Staff member
Super Moderator
#9
Bev D said:
I took a quick look at the data: there is probably NOT a true difference. yes the value was below .05, but the r SQ value was so low indicating a that IF a difference does exist it's very small and cavity to cavity differences are not the primary cause of the variation. So you could 'fix' the cavity thing and end up not improving yoru performance one bit because of the effect of the pareto of X's.

Can you please explain to mean the meaning of R-squared in the ANOVA context? I can understand it for regression analysis,but I am unable to comprehend its meaning when the levels are categorical, and their spacing has no meaning.

I have attached a JMP output (as a picture in powerpoint) that shows the variation (ALWAYS plot your data, look at your data and THINK about your data...summary statistics - while powerful - can be very misleading)

a few other points to consider:
1. you have 8 cavities - a low p value could simply be "alpha error". remember alpha is the risk that you 'detected' a difference when one doesn't actually exist.
This is true, but the the p-value for ANOVA is an omnibus test that looks for any difference without a statement of which levels are different. The alpha error would come from doing the 28 pairwise comparisons and seeing one of them being significant at alpha=0.05

2. Your production manager may be spot on: IF you have a lot of measurement error you coudl get this result as well. A real difference may exist but the error is so large you can't see the difference. You comment concerning ANOVA, and t tests 'overcoming' measurement is valid but ONLY if the results for each level of the factor are separated a great distance (very little to no overalp in the results). So my recommendation is to perform a measurement study adn look for the measurement error vs the actual variation you are getting...
BINGO, without a measurement study, it is very difficult to know if the measurement system is an issue. Though the Square root of the MSE divided by the mean gives only a 6% CV, so it is unlikely that the measurement system is way off.
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#10
Statistical Steven said:
BINGO, without a measurement study, it is very difficult to know if the measurement system is an issue. Though the Square root of the MSE divided by the mean gives only a 6% CV, so it is unlikely that the measurement system is way off.
Maybe - but I've not seen %CV be a reliable indicator of measurement error (and my current company makes measuring instruments for animals. I've seen horrible measurement error with %CVs less than 10%. ) That's why he needs to do the measuremetn study. It's the 3% r-sq that tells me he's chasing butterflies.
 
Thread starter Similar threads Forum Replies Date
T Relationship between ISO 9001 and ISO – IEC BS EN 870079- 34 2020 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
S Relationship between IEC 62304 problem resolution and ISO 13485 IEC 62304 - Medical Device Software Life Cycle Processes 8
C Understanding the relationship between 62304 and the MDD ER IEC 62304 - Medical Device Software Life Cycle Processes 7
D Relationship between Diameter and Flow Rate & the Length Statistical Analysis Tools, Techniques and SPC 3
T Relationship between ISO9001 and Lean Lean in Manufacturing and Service Industries 15
P The relationship between SPC and a Quality Management System Statistical Analysis Tools, Techniques and SPC 4
T Relationship between TS16949 and 5 Core Tools & relationship between 5 Core Tools IATF 16949 - Automotive Quality Systems Standard 5
N What is the relationship between a DFMEA and PFMEA? FMEA and Control Plans 1
Q What is the relationship between Standard Deviation and 6 Sigma? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 4
J Relationship between Quality, Reliability, Price and Delivery Misc. Quality Assurance and Business Systems Related Topics 7
D AIAG FMEA 4? Ed. Explanation: Relationship between PFMEA and Special Characteristics FMEA and Control Plans 8
S Relationship between Normal Distribution and AQL AQL - Acceptable Quality Level 13
D Relationship between IEC 60601-1 3rd Ed and IEC 60601-2-10 IEC 60601 - Medical Electrical Equipment Safety Standards Series 8
S The relationship between IEC 60601-2-18 and IEC 60601-2-37 - Ultrasound Project IEC 60601 - Medical Electrical Equipment Safety Standards Series 2
P Relationship between Crimping ID & Crimping Depth - Minitab usage Statistical Analysis Tools, Techniques and SPC 3
kedarg6500 Relationship between Cp and Cpk Capability, Accuracy and Stability - Processes, Machines, etc. 4
L Arima Output - Relationship between my raw data and the residuals generated Using Minitab Software 1
C The relationship between OEM and the subcontracted manufacturer 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 1
Ajit Basrur Relationship between Blend Uniformity and Content Uniformity Pharmaceuticals (21 CFR Part 210, 21 CFR Part 211 and related Regulations) 2
D Relationship between ISO 9001:2000 and EFQM ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
L What's the relationship between LVD (low voltage directive) and MDD EU Medical Device Regulations 3
D The relationship between the FMEA and Control Plan FMEA and Control Plans 4
K Relationship between "Corrections/Removals" procedure and "Recall" procedure US Food and Drug Administration (FDA) 11
M Relationship between APQP and CM (Configuration Management) APQP and PPAP 11
S Relationship and Differences between ISO 16949 and ISO 15504 (SPICE) IATF 16949 - Automotive Quality Systems Standard 3
kedarg6500 Relationship between Cp & Cpk Statistical Analysis Tools, Techniques and SPC 5
M Relationship between quality assurance costs and failure costs Quality Tools, Improvement and Analysis 4
L Relationship between Calibration and a Gage R&R study? General Measurement Device and Calibration Topics 9
V Relationship between Severity and Class FMEA and Control Plans 3
A ISO TS 16949 - Relationship between requirements and processes IATF 16949 - Automotive Quality Systems Standard 0
J FMEA - What is the relationship between Detection and Prevention? QP Newbie question. FMEA and Control Plans 12
J Relationship Between SAP and ISO9001:2000 Quality Assurance and Compliance Software Tools and Solutions 1
P The relationship between ISO 17025 vs. ANSI Z540-1 ISO 17025 related Discussions 2
L Is there any relationship between Cpk & Right First Time? First Pass Yield (FPY)? Capability, Accuracy and Stability - Processes, Machines, etc. 4
L 'Distributor' definition & MDR/MSR relationship EU Medical Device Regulations 3
S APQP Relationship Diagram APQP and PPAP 2
J CEN/TR 17223:2018 Guidance on the relationship between13485 and MDR.. worth the money?? ISO 13485:2016 - Medical Device Quality Management Systems 2
K Does OEM/OBL-Relationship need to involve an EU Authorized Representative? EU Medical Device Regulations 11
MDD_QNA What is the relationship with the CSA group? Canada Medical Device Regulations 10
T ITAR/EAR relationship and Flow Down to Suppliers AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 5
C DFMEA vs. PFMEA Relationship: The cause in DFMEA becomes Failure in the PFMEA FMEA and Control Plans 2
J Responsibilities, Authorities and Relationships - What does Relationship mean? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 4
5 ISO 13485 relationship to 21 CFR 820 ISO 13485:2016 - Medical Device Quality Management Systems 17
alonFAI Definition Failure Review Board and FRACAS - Relationship and differences Definitions, Acronyms, Abbreviations and Interpretations Listed Alphabetically 2
T Capability Level Relationship with ISO/IEC 20000 Achievement IT (Information Technology) Service Management 2
N Agile 9.3.1 Relationship Searches - Meaning of Message Manufacturing and Related Processes 1
C CRMs (Customer Relationship Management) to help achieve ISO 9001 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 6
C ISO9001 and ISO 13485 Relationship? Can we do both effectively? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 4
B Relationship Pictures Coffee Break and Water Cooler Discussions 10
shimonv Operator vs. Manufacturer Relationship and ISO 13485 / CE Requirements ISO 13485:2016 - Medical Device Quality Management Systems 5

Similar threads

Top Bottom