Gage R&R study - Ordinal data

[Tahirawan77]

Hi,

I would like to perform a Gage R&R study where the data type is 'Ordinal' in nature. The defect can be categorized as No defect, Low, Medium and High

The process in question is drilling of holes in composite material. After the drilling process there is risk of making a 'splinter' at the edge of hole. The size of the splinter can vary from 'No splinter' to Low, Medium and High splinter. I cannot / do not want to measure the splinter size as this is not measured in the production due to time constraints. Instead each hole is inspected visually by QC inspector and rated as one of the above category. Low Splinter will not be reworked, any Medium splinter will be reworked where as High will be rejected.

I would like to plan the study as Attribute Gauge R&R study with 30 holes where holes with 'No Splinter' is marked as 'Good' and all other form of splintering can be categorized as 'Fail'. But by doing so I will miss the ability of the operators to identify different defect types (medium defect identified as High and vice versa).

I would like to know if there is any other way how I can perform the study where the data type is 'Ordinal' or any other suggestions to improve the Gage R&R study

Thanks

 

[Ron Rompen]

Maybe I'm oversimplifying this, but why not just substitute numerical values for the ordinal values (0 to 3)?

 

[John Predmore]

The Kappa statistic is used to assess agreement of multiple reviewers producing categorical ratings. Cohen's Kappa is for two mutually exclusive categories, and Fleiss' Kappa for three or more categories. Repeatability of a reviewer is not the same metric as inter-reviewer agreement, but Kappa analysis may give you some useful information. If you are comparing reviewers to a known correct categorization result, you could also use Chi-square analysis.

 

[Bev D]

Maybe I'm oversimplifying this, but why not just substitute numerical values for the ordinal values (0 to 3)?

Because ordinal ‘data’ doesn’t meet the requirements for mathematical calculations used in the continuous data R&R. It is also extremely chunky so the SD calculation will be greatly over estimated.

 

[Miner]

The Kappa statistic is used to assess agreement of multiple reviewers producing categorical ratings. Cohen's Kappa is for two mutually exclusive categories, and Fleiss' Kappa for three or more categories. Repeatability of a reviewer is not the same metric as inter-reviewer agreement, but Kappa analysis may give you some useful information. If you are comparing reviewers to a known correct categorization result, you could also use Chi-square analysis.

The various Kappa statistics are used for nominal (unordered) categories. For ordinal (ordered) data, perform an attribute agreement analysis then use Kendall's coefficient of concordance. Kappa only considers whether the nominal categories were matched or unmatched. Kendall's takes into consideration how close on the ordinal scale you were.

 

[Tahirawan77]

The Kappa statistic is used to assess agreement of multiple reviewers producing categorical ratings. Cohen's Kappa is for two mutually exclusive categories, and Fleiss' Kappa for three or more categories. Repeatability of a reviewer is not the same metric as inter-reviewer agreement, but Kappa analysis may give you some useful information. If you are comparing reviewers to a known correct categorization result, you could also use Chi-square analysis.

Thanks John for your reply but just to make sure i got it correct. Since i have four categories in my study, so i need to perform Fleiss Kappa?
My plan is to first give the samples to a 'specialist' and ask him to rate the samples and then give it to three operators to perform MSA. As your link points that there is no formal way of interpreting the results but i plan to use a score of 0.8 or higher for passing the MSA?

 

[Miner]

Thanks John for your reply but just to make sure i got it correct. Since i have four categories in my study, so i need to perform Fleiss Kappa?
My plan is to first give the samples to a 'specialist' and ask him to rate the samples and then give it to three operators to perform MSA. As your link points that there is no formal way of interpreting the results but i plan to use a score of 0.8 or higher for passing the MSA?

No. Use Kendall's coefficient of concordance for ordinal data. Kappa is for nominal data. The results of the specialist will be your "known standard."

A score >/= 0.7 is good; >/= 0.9 is excellent