Measurement Uncertainty in Subjective Tests
Organization: Disorganised Inc.
Newsgroups: misc.industry.quality
Subject: Measurement uncertainty
Date: Sat, 15 Apr 2000 17:06:30 GMT
We regularly measure the effects of certain treatments on textile materials and fabrics.
These effects are evaluated by a panel and the result is a discrete value on a scale, say 0 to 5 (or in another range).
Our auditor requires uncertainty of measurement for all of our tests. I have doubts whether the concept makes sense in this context, but apparantly that is not a discussion that our auditor is prepared to enter in. He wants UM, point.
Any hints or ideas ?
TIA
Johan
----------------
Newsgroups: misc.industry.quality
Subject: Re: Measurement uncertainty
Organization: WebUseNet Corp
Date: Sun, 16 Apr 2000 08:36:41 -0400
If you are not performing a quantitative measurement then the
measurement uncertainty will be null and void. It sounds as if what
you are doing is more of a survey to say. Without a measureable unit
i.e. ohms, liters, etc you can not do this. I would go around the
auditor and call someone else at the same agency to get assistance.
Remember, the auditors are folks just like us and may have been
instructed that "EVERYTHING" must have an uncertainty when in fact
some tests are simply for mild confidence.
---------------------
Newsgroups: misc.industry.quality
Subject: Re: Measurement uncertainty
Date: Sun, 16 Apr 2000 15:41:56 GMT
Greetings Johan,
I think the concept of uncertainty does make sense in this context. Why don't you just go ahead and measure the repeatability etc. of the measurement. You will probably find that the rating scale is more variable than a physical measurement would be.
Do this experiment. Select 6-7 colored samples that cover your range of products. Cut each sample in half so that you have two identical samples of seven products. (The exact number of products and samples is not important.)
Select your panel members and others as testers to perform the experiment. Assign code numbers to the samples so that the testers cannot tell which samples are pairs. Let the testers see only one sample at a time. Don't let them see each other do the test, and don't let them collaborate to achieve agreement. Randomize the order of testing.
Perform the complete test under standardized lighting conditions on three different days at least one week apart. Let one day be early morning, next at mid-day, and last late in the day. (Eyeballs vary.)
Analyze the individual test scores (not averages) with analysis of variance. Determine the size of these sources of variability:
variability between identical samples (within day, product, tester)
variability between days (within product, tester)
variability between testers (within day, product, sample)
Report the means and standard deviations that describe each component of variability. Do not round the results to discrete values.
Perform hypothesis tests for each component of variance. This will show which components are probably real. If you need information on hypotheses testing see the message on this forum by Kelly Speiser, Subject: "Statistical Hypothesis Testing". She has a book on hypothesis testing.
When you finish, you will have better knowledge of your test method. This will lead to improvements.
And you will exceed the expectations of your auditor.
If you want to go the next step and make the accept/reject color decisions statistically, see my website at: www.samplingplans.com/colormeasurement.htm
The method shown there appplies to rating scale color judgements as well as physical measurements.
Sincerely, Stan Hilliard
CQE,CQR,CQA,PE
------------------------
Newsgroups: misc.industry.quality
Subject: Re: Measurement uncertainty
Date: 16 Apr 2000 23:31:37 GMT
Yes, I agree with everyone that this auditor is wacky in asking measurement uncertainty.
The ISO wording (section 4.11.1 para 2) Inspection, measuring, and test equipment shall be used in a manner which ensures that the measurement uncertainty is known and is consistent with the required measurement capability.
I would also be curious if this were a required inspection you are doing versus something that you just test for process control. If it is a process control item and there is other equipment or such other inspections that take place which would verify the textile as meeting the specified requirements... then calibration is not required.
I believe what the auditor is doing mixing an inspection item concept and a workmanship criteria item together. And since the auditor was unwilling to discuss, their ignorance on the subject was obvious.
An auditor under the scope of 10011 is supposed to have documented objective evidence of what they are discussing and how the standard applies as well as what makes up the nonconformance. The nonconformance also has to be of a benefit to your company (it's in the standard folks).
Obviously they bungled their finding and I would challenge it with their appeals process and make them demonstrate what it is they are trying to enforce.
Slan Leat!
Phil McManus
Organization: Disorganised Inc.
Newsgroups: misc.industry.quality
Subject: Measurement uncertainty
Date: Sat, 15 Apr 2000 17:06:30 GMT
We regularly measure the effects of certain treatments on textile materials and fabrics.
These effects are evaluated by a panel and the result is a discrete value on a scale, say 0 to 5 (or in another range).
Our auditor requires uncertainty of measurement for all of our tests. I have doubts whether the concept makes sense in this context, but apparantly that is not a discussion that our auditor is prepared to enter in. He wants UM, point.
Any hints or ideas ?
TIA
Johan
----------------
Newsgroups: misc.industry.quality
Subject: Re: Measurement uncertainty
Organization: WebUseNet Corp
Date: Sun, 16 Apr 2000 08:36:41 -0400
If you are not performing a quantitative measurement then the
measurement uncertainty will be null and void. It sounds as if what
you are doing is more of a survey to say. Without a measureable unit
i.e. ohms, liters, etc you can not do this. I would go around the
auditor and call someone else at the same agency to get assistance.
Remember, the auditors are folks just like us and may have been
instructed that "EVERYTHING" must have an uncertainty when in fact
some tests are simply for mild confidence.
---------------------
Newsgroups: misc.industry.quality
Subject: Re: Measurement uncertainty
Date: Sun, 16 Apr 2000 15:41:56 GMT
Greetings Johan,
I think the concept of uncertainty does make sense in this context. Why don't you just go ahead and measure the repeatability etc. of the measurement. You will probably find that the rating scale is more variable than a physical measurement would be.
Do this experiment. Select 6-7 colored samples that cover your range of products. Cut each sample in half so that you have two identical samples of seven products. (The exact number of products and samples is not important.)
Select your panel members and others as testers to perform the experiment. Assign code numbers to the samples so that the testers cannot tell which samples are pairs. Let the testers see only one sample at a time. Don't let them see each other do the test, and don't let them collaborate to achieve agreement. Randomize the order of testing.
Perform the complete test under standardized lighting conditions on three different days at least one week apart. Let one day be early morning, next at mid-day, and last late in the day. (Eyeballs vary.)
Analyze the individual test scores (not averages) with analysis of variance. Determine the size of these sources of variability:
variability between identical samples (within day, product, tester)
variability between days (within product, tester)
variability between testers (within day, product, sample)
Report the means and standard deviations that describe each component of variability. Do not round the results to discrete values.
Perform hypothesis tests for each component of variance. This will show which components are probably real. If you need information on hypotheses testing see the message on this forum by Kelly Speiser, Subject: "Statistical Hypothesis Testing". She has a book on hypothesis testing.
When you finish, you will have better knowledge of your test method. This will lead to improvements.
And you will exceed the expectations of your auditor.
If you want to go the next step and make the accept/reject color decisions statistically, see my website at: www.samplingplans.com/colormeasurement.htm
The method shown there appplies to rating scale color judgements as well as physical measurements.
Sincerely, Stan Hilliard
CQE,CQR,CQA,PE
------------------------
Newsgroups: misc.industry.quality
Subject: Re: Measurement uncertainty
Date: 16 Apr 2000 23:31:37 GMT
Yes, I agree with everyone that this auditor is wacky in asking measurement uncertainty.
The ISO wording (section 4.11.1 para 2) Inspection, measuring, and test equipment shall be used in a manner which ensures that the measurement uncertainty is known and is consistent with the required measurement capability.
I would also be curious if this were a required inspection you are doing versus something that you just test for process control. If it is a process control item and there is other equipment or such other inspections that take place which would verify the textile as meeting the specified requirements... then calibration is not required.
I believe what the auditor is doing mixing an inspection item concept and a workmanship criteria item together. And since the auditor was unwilling to discuss, their ignorance on the subject was obvious.
An auditor under the scope of 10011 is supposed to have documented objective evidence of what they are discussing and how the standard applies as well as what makes up the nonconformance. The nonconformance also has to be of a benefit to your company (it's in the standard folks).
Obviously they bungled their finding and I would challenge it with their appeals process and make them demonstrate what it is they are trying to enforce.
Slan Leat!
Phil McManus