Attribute MSA (Kappa Study)

[hazwan2283]

Hi All,
I am Hazwan. Kindly would like to ask the experts help here on the attribute MSA (Kappa study) topic. This is my question. Is it possible to have negative value of Kappa ?

As for humble understanding, formula of Kappa is (Po-Pe)/(1-Pe). So let say if i have Po =0.2 and let say Pe = 0.45. Then mathematically speaking my Kappa value is -0.45. Is this acceptable value for Kappa to be negative? Or is it Kappa must be from 0 onwards.

Sincerely,
Hazwan

 

[Miner]

Kappa values may range from -1 to 1. Kappa values between -1 and 0 are due solely to chance and would be unacceptable. 0 to 0.6 are extremely poor, 0.6 to 0.7 are marginal, 0.7 to 0.9 are acceptable, and 0.9 to 1 are excellent.

 

[hazwan2283]

So @Miner , you are saying that the value may range to -1 to 1. I thought that answers my question but then your second statement confuses me further. So does negative value means invalid data or something?. .. Kindly please help

 

[Miner]

Not invalid but the agreements are solely due to chance, as if each person flipped a coin and made their decision based on the outcome.

 

[Bev D]

Think about the math. A zero is a perfectly random response. A perfect 1 means that all choices were correctly identified. A Pass was a Pass and a Fail was a fail. So...a-1 must be a perfectly incorrect identification. All passes were identified as fails and all fails were identified as passes. While a 1 is possible a -1 would generate some questions as to how the study was carried out. A negative number with a correctly structured and executed study design means that the assessors are particularly bad at identifying passing and failing parts.

A decent introductory article on the KAPPA method is: David Futrell, “When Quality is a Matter of Taste, Use Reliability Indexes”, Quality Progress, Vol. 28, No. 5, May 1995, pp. 81-86 The article is free to ASQ members...you might find it for free here or here

It's about understanding the math...one should never simply crunch numbers in a formula without understanding how the formula works....and this leads me to the next caveat: the study design is even more important than the math. The Kappa score is influenced by the sample size and the mix of passes and fails and more importantly the marginality of the passes and fails. While the math provides some protection from a poorly mixed small sample study (it corrects for randomly selecting the right answer) it can't overcome a study that has solid and obvious pass and fail parts, if your process has many marginal parts.

 

[hazwan2283]

Think about the math. A zero is a perfectly random response. A perfect 1 means that all choices were correctly identified. A Pass was a Pass and a Fail was a fail. So...a-1 must be a perfectly incorrect identification. All passes were identified as fails and all fails were identified as passes. While a 1 is possible a -1 would generate some questions as to how the study was carried out. A negative number with a correctly structured and executed study design means that the assessors are particularly bad at identifying passing and failing parts.

A decent introductory article on the KAPPA method is: David Futrell, “When Quality is a Matter of Taste, Use Reliability Indexes”, Quality Progress, Vol. 28, No. 5, May 1995, pp. 81-86 The article is free to ASQ members...you might find it for free here or here

It's about understanding the math...one should never simply crunch numbers in a formula without understanding how the formula works....and this leads me to the next caveat: the study design is even more important than the math. The Kappa score is influenced by the sample size and the mix of passes and fails and more importantly the marginality of the passes and fails. While the math provides some protection from a poorly mixed small sample study (it corrects for randomly selecting the right answer) it can't overcome a study that has solid and obvious pass and fail parts, if your process has many marginal parts.

it is super clear now. Thank you very much to @Miner and to @Bev D for the detailed sharing

Sincerely,
Hazwan