Measurement Uncertainty in Subjective Tests such as Color or Taste

[Marc]

Measurement Uncertainty in Subjective Tests

Organization: Disorganised Inc.
Newsgroups: misc.industry.quality
Subject: Measurement uncertainty
Date: Sat, 15 Apr 2000 17:06:30 GMT

We regularly measure the effects of certain treatments on textile materials and fabrics.

These effects are evaluated by a panel and the result is a discrete value on a scale, say 0 to 5 (or in another range).

Our auditor requires uncertainty of measurement for all of our tests. I have doubts whether the concept makes sense in this context, but apparantly that is not a discussion that our auditor is prepared to enter in. He wants UM, point.

Any hints or ideas ?

TIA

Johan

----------------

Newsgroups: misc.industry.quality
Subject: Re: Measurement uncertainty
Organization: WebUseNet Corp
Date: Sun, 16 Apr 2000 08:36:41 -0400

If you are not performing a quantitative measurement then the
measurement uncertainty will be null and void. It sounds as if what
you are doing is more of a survey to say. Without a measureable unit
i.e. ohms, liters, etc you can not do this. I would go around the
auditor and call someone else at the same agency to get assistance.
Remember, the auditors are folks just like us and may have been
instructed that "EVERYTHING" must have an uncertainty when in fact
some tests are simply for mild confidence.

---------------------

Newsgroups: misc.industry.quality
Subject: Re: Measurement uncertainty
Date: Sun, 16 Apr 2000 15:41:56 GMT

Greetings Johan,

I think the concept of uncertainty does make sense in this context. Why don't you just go ahead and measure the repeatability etc. of the measurement. You will probably find that the rating scale is more variable than a physical measurement would be.

Do this experiment. Select 6-7 colored samples that cover your range of products. Cut each sample in half so that you have two identical samples of seven products. (The exact number of products and samples is not important.)

Select your panel members and others as testers to perform the experiment. Assign code numbers to the samples so that the testers cannot tell which samples are pairs. Let the testers see only one sample at a time. Don't let them see each other do the test, and don't let them collaborate to achieve agreement. Randomize the order of testing.

Perform the complete test under standardized lighting conditions on three different days at least one week apart. Let one day be early morning, next at mid-day, and last late in the day. (Eyeballs vary.)

Analyze the individual test scores (not averages) with analysis of variance. Determine the size of these sources of variability:

variability between identical samples (within day, product, tester)

variability between days (within product, tester)

variability between testers (within day, product, sample)

Report the means and standard deviations that describe each component of variability. Do not round the results to discrete values.

Perform hypothesis tests for each component of variance. This will show which components are probably real. If you need information on hypotheses testing see the message on this forum by Kelly Speiser, Subject: "Statistical Hypothesis Testing". She has a book on hypothesis testing.

When you finish, you will have better knowledge of your test method. This will lead to improvements.

And you will exceed the expectations of your auditor.

If you want to go the next step and make the accept/reject color decisions statistically, see my website at: www.samplingplans.com/colormeasurement.htm

The method shown there appplies to rating scale color judgements as well as physical measurements.

Sincerely, Stan Hilliard
CQE,CQR,CQA,PE

------------------------

Newsgroups: misc.industry.quality
Subject: Re: Measurement uncertainty
Date: 16 Apr 2000 23:31:37 GMT

Yes, I agree with everyone that this auditor is wacky in asking measurement uncertainty.

The ISO wording (section 4.11.1 para 2) Inspection, measuring, and test equipment shall be used in a manner which ensures that the measurement uncertainty is known and is consistent with the required measurement capability.

I would also be curious if this were a required inspection you are doing versus something that you just test for process control. If it is a process control item and there is other equipment or such other inspections that take place which would verify the textile as meeting the specified requirements... then calibration is not required.

I believe what the auditor is doing mixing an inspection item concept and a workmanship criteria item together. And since the auditor was unwilling to discuss, their ignorance on the subject was obvious.

An auditor under the scope of 10011 is supposed to have documented objective evidence of what they are discussing and how the standard applies as well as what makes up the nonconformance. The nonconformance also has to be of a benefit to your company (it's in the standard folks).

Obviously they bungled their finding and I would challenge it with their appeals process and make them demonstrate what it is they are trying to enforce.

Slan Leat!

Phil McManus

 

[John C]

Johan,
You state that you 'measure' effects. I think this gets you off to a bad start since, to measure implies an objective judgement. Yet yours appears to be totally subjective. So I suggest you drop the word measure and replace it with something like 'assign a subjective value' or whatever.
If the auditor still requires you to apply measurement uncertainty then tell him/her that if you had the equipment to evaluate all the variables associated with turning what are generally considered to be subjective assessments, into scientifically valid objective visual and mental tests, then you would use that equipment directly and not via that roundabout route but, you don't have the equipment and, as far as you know, it does't exist.
Stan,
Be vary careful when suggesting methods to meet unreasonable expectations because, whether or not they are practicable and effective, the auditor will be using them to claim that he has seen a method of achieving this and will be asking for more of it in the future, convinced that some people actually do it.
rgds, John C

 

[Brian Dowsett]

Johan,
I'd say you're already one step ahead of your auditor. Your organisation must have already identified some measurement uncertainty in order to make the decision to use a panel to do the evaluations. Surely the purpose of measuring the uncertainty is so you can either allow for it or improve the measuring system. You have taken the second approach and gone for a group decision to lessen the subjectivity.
I'd definitely agree with John C from Cork, don't placate auditors unless it's absolutely necessary.
If there's no alternative with your man, then do a simple attribute MSA (see the shiny blue QS9000 book)first with single inspectors then with the group, proving that the group has less variation in the measurement.

Cheers

Brian

 

[romelnar]

measurement uncertainty

i've been tryin to locate the word "measurement uncertainty" in the iso9k2k and i cannot find one.

if it really is not there, what was the reason for its deletion? is it still required in this new revision?

anybody? thanks.

 

[Marc]

ISO 9001:2000 does not specifically address measurement uncertainty, but as with QS-9000 there is an expectation that you understand measurement systems and account for measurement uncertainty 'where appropriate'.

 

[Ed]

In the old standard, the phrase "measurement uncertainty is known" could have and probably has driven some companies to calculate mesurement uncertainty for all devices (usually in the form of Gage R&R). The real world is dominated by over selection of instrument capabilities. For example, choosing an instrument that can resolved one more decimal place than the tolerance applied to the specification - otherwise known as the 10:1 rule. In that case, the measurement uncertainty isn't known exactly, however, the capability is known to exceed the need. In ISO 9001:2000, the governing words are "ensure valid results". The result is much less confusion and unnecessary work.

 

[Charles Wathen]

Measurement Uncertainty...

Do we really need to do this?

I'm a supervisor of a calibration lab within a company. We calibrate ~ 97% of all our equipment in house. Our standards are sent out either to the mfg or to an approved vendor. Some of the other standards are calibrated in house.

I was thinking that it might be a good idea to do measurement uncertainties on the calibrations we do for our standards, but not for other instruments like calipers, micrometers, etc. that are used by mfg. Thoughts?

Most of the measurement uncertainty stuff I've read, including on this board, gives me a headache! I have a decent Excel spreadsheet I use for reporting this, but I'm still not sure if I'm doing this right. For example:

I have an Omega Thermocouple Calibrator that does RTD simulation. It has an accuracy of ±.18°C, and a resolution of .1°C.

We calibrate this Omega Thermocouple Calibrator using the following equipment:
RTD Simulator with an accuracy of ±.005% Resolution:.001 ohm
Keithley 2000 DMM accuracy: 30ppm + 30ppm of range resolution .0001 ohm.

After performing the calculations, I get an Expanded Uncertainty of .115874. My questions is: should I be including the resolution of the Omega Thermocouple Calibrator in this? If I leave it out, i get .009662. Remember, the Omega Thermocouple Calibrator is the device I'm calibrating. It seems like it would be OK to include the resolution of the Omega Thermocouple Calibrator as the Expanded Uncertainty is less than the ±.18°C spec for this unit of measure.

Any thoughs are greatly appreciated!

 

[Jerry Eldred]

Since it is a temp calibrator, if you are generating output, the calibrator's resolution is not a factor in it's accuracy. For example if you put out 100 degrees C into a UUT. You set the calibrator for 100.0. The calibrator's resolution is not a factor. It is spec'd at +/-0.18 deg C. Therefore when you apply that 100 deg C output to the UUT, you know that the 100 C applied is within +/-0.18 degrees of nominal. The only limiter is that if you want to fine adjust the calibrator output until the UUT reaches a nominal, you are limited to 0.1 degree increments.

Resolution is not a contributor to variability. In the analogue days, when you had to read a meter, parallax error was always a factor. But the inception of digits gave us a meter reader who took out parallax error. If, for example you have a 0.1 degree resolution on a meter. If you measure a 100 ohm resistor on that meter. If you make 1000 measurements over a course of a month, and on every reading, the meter reads 100.0, that digit of resolution did not impact variability. If theoretically you were able to see those smaller digits of resolution not present, there would be variation at some lower level.

If it is a meter for it's input, specs mean something a little different than for a monitor meter for a calibrator's output. The least significant digit in an output monitor meter (the model that comes to mind, although I haven't verified whether this is the one you are discussing, is the CL505A). When you use a CL505A in an output mode, the digits are present to provide information as to what you have the the output for. The digit (in that case) will stay where you set it. The output will match that fixed setpoint to within the tolerance.

As long as you are at least 4:1, the UUT should never have a tolerance of less than +/-0.72 (or whatever 4:1 to the RSS of multiple standards is).

 

[Ryan Wilde]

Charles,

I was thinking that it might be a good idea to do measurement uncertainties on the calibrations we do for our standards, but not for other instruments like calipers, micrometers, etc. that are used by mfg. Thoughts?

It is very common (including the company that I work for) to NOT calculate uncertainty estimates for instruments such as micrometers and calipers. Most apply a TUR statement, which is (at this time) acceptable by the accreditation bodies that I am familiar with. I would assume that you calibrate calipers/micrometers with a Grade 2 gage block set, which easily exceeds the accepted 4:1 ratio.

My questions is: should I be including the resolution of the Omega Thermocouple Calibrator in this?

The resolution of the UUT must be included in the uncertainty estimate. With a resolution of 0.1°C, if you were to state that your measurement uncertainty was 0.010°C, I would certainly question you. Basically, I could vary my source by as much as the resolution without seeing a switch in reading.

Example: Input a value of 74.951°C - unit reads 75°C. Input a value of 75.049 - unit reads 75.0°C. The difference in input is 0.098°C with no change in reading.

My estimated uncertainty could not possibly be any tighter that 0.098ºC, and I didn't take into account standard uncertainties, Type A uncertainties, etc.

Here is the misconception: Expanded Measurement uncertainty and tolerance are related. They are not. The Expanded Measurement Uncertainty is the uncertainty of the measurement that you are taking - regardless of tolerance. In other words, the UUT is reading 75.0°C ± 0.116°C. If it reads 74.5°C, the measurement is 74.5°C ± 0.116°C (your uncertainty), but it is out of tolerance.

Where tolerance and uncertainty meet is in TUR calculation. Tolerance is 0.18°C, the combined uncertainty of my standards is 0.009°C. That is approximately 20:1 in my book, and I am very satisfied that I am making good measurements that will reflect UUT condition with very little influence from my standards.

Ryan

 

[Charles Wathen]

Thank you both for your inputs.

I do have one question remaining: do you guys document your TUR on the calibration certificate, or do you simply file somewhere the calculation? I just had an audit last week, and the auditor asked me if I maintain a 10:1 ratio when performing calibrations. I stated to him that 10:1 is ideal, but 4:1 is the minimum. There were no further questions on this, but I'm wondering if this should be documented somewhere. Thoughts?