Uncertainty and Proficiency Testing

[Marc]

Date: Fri, 8 Oct 1999 14:44:45 -0700
From: Jan Johansen <jjelect@teleport.com>
To: Greg Gogates <iso25@fasor.com>
Subject: Uncertainty and Proficiency Testing

Dear ISO-25er’s,

We recently received the results of a proficiency test back. The test was for a 5 * digit multimeter. This test was accomplished NAPT(National Association for Proficiency Testing). They handled everything in a professional manner We passed with flying colors. However after reviewing the data from all other participants, I have some observations that I think bears some discussion.

In order to make this as easy as possible I will only deal with one data point, the 10VDC on the 20VDC range. I am going to list the values from the labs that were within +/- .0001 VDC of agreeing with my reading and their stated uncertainties.

Lab Value Uncert
1 9.9985 9e-005
2 9.9985 .00098
3 9.9984 .0018
4 9.9984 .00043
5 9.9983 .0001
6 9.9984 .00013
7 9.9983 .0009
8 9.9985 .0006
9 9.9983 .0001
10 9.9984 .0003

Reported Nominal Value for the test was 9.99839 +/- 8.8 e-005

As you can see the results are quite varied. Here are the observations.

1. The measuring instrument is a 5 * DMM. The best resolution on the 20 VDC range is .0001. How can the nominal value be reported to .00001? It is my contention that a value should never be reported past the maximum resolution of the readout.

2. The definition of uncertainty say’s ‘ … the result of a measurement , that characterizes that characterizes the dispersion of the values that could reasonable be attributed to the measurand.’

3. The device providing the measurand is the 5 * DMM. The one year specification for the DMM is +/- .0015% Rdg + 3 counts or +/- 0.0018 VDC.

4. It doesn’t matter who’s calibrator you use (Fluke, Wavetek), the majority of the uncertainty is in the DMM since it is the unit providing the measurand. Since a specification is a valid Type B uncertainty, it appears that most of the labs did not include it and only reported the uncertainty of their standards, NOT the measurement uncertainty.

5. We could have connected the DMM to our 10 V in-house reference, but it would not have improved our uncertainty by much, because of what is providing the measurement.

I would like to know if I am all wet or am I right. I am uncertain that even this forum will have consensus but it seems to be a good place to air this kind of information.

Thank you

Jan Johansen
JJ Electronics
jjelect@teleport.com

[Marc]

Date: Tue, 12 Oct 1999 08:11:11 -0700
From: "Nielsen, Larry E" <NIELSELE@sce.com>
To: 'Greg Gogates' <iso25@fasor.com>
Subject: RE: Uncertainty and Proficiency Testing

Jan,

Prior to working for Southern California Edison I worked with the Navy METCAL program as a contractor in evaluating calibration lab performance via a proficiency testing program. In order to evaluate the competence of a laboratory using proficiency testing, the measurement "problem" or experimental design must be properly posed, and the artifacts must be up to the task. This is the function of the organizer or provider of the service, in this case NAPT.

If you set out to evaluate several laboratory's whose customer base or workload consists of calibrating 5* digit DMMs, it should be presumed that the laboratory is in possession of "standards" or a calibrator with an uncertainty at least 25% smaller than the intended work. Therefore, in order to pass judgement on the measuring process, the uncertainty of the artifact must be equal to or less than that of the calibrator. In this case a stable 6* digit DMM (preferably a pair of them) would have been a better choice for the artifact.

Based on your brief description of the testing, I don't presume to know all the facts, and we must acknowledge that maintenance of ever increasing test advantage is not always possible. However, it appears to me that you wasted your time and money on this one. Some good guidance on this subject may be found in NCSL RP-15.

Larry E. Nielsen
So. Cal. Edison - Metrology
7300 Fenwick Lane
Westminster, CA 92683
(714) 895-0489; fax (714) 895-0686
e-mail: nielsele@sce.com

[Marc]

Date: Wed, 13 Oct 1999 10:36:45 -0500
From: TSmith <TSmith@nctl.com>
To: Greg Gogates <iso25@fasor.com>
Subject: Re: Uncertainty and Proficiency Testing

Greetings Jan,

As a participant of 9 proficiency tests in 1999, I was taken back by the fact that you chose to ignore the basic rules of Proficiency Testing : To publish proficiency testing data without the written consent of NAPT and use the information for advertising (stating your colors were flying.) I can not attempt to understand your agenda nor could I expect any other metrologist that understands proficiency testing or guide 43. I thought one of the conditions of participation was not to use the results for advertisement.

After reading one of my reports issued by NAPT, I noticed that on the report is states that the information contained in this report can not be reproduced expect in full, I am sure that a lot of folks are going to be confused by the limited amount of information that you shared with them. When you only give bits and pieces of the puzzle how can you expect anyone to put it together. (I have written permission to publish this paragraph)

It also states on the report that the values contained in the report were reported to NAPT, I was led to believe that NAPT was an administrator of proficiency testing. Not a calibration provider. All values are reported to the association and the data is then reduced by the association. Isn't this what we want from an third party administer of proficiency tests?

The values reported by all the participants is what should be reviewed by us. After all this was one of the reasons for participating in the test was to help all labs better understand where they are in relationship to other labs making the same type of measurement. I agree that some of the uncertainties reported by other participants seem a little out of whack, now that they have seen how they compare to other they can re-evaluate their measurement process. We all know that labs making measurements have a long way to go to properly understand and report their uncertainties. Let's work on solving that problems together.

I spoke with the folks at NAPT and they shared with me how they arrived at the nominal value for this PT. The nominal value was given to them by the primary lab, which inputted the value into the meter to achieve the desired set point. In this case the standard providing the source was greater than the resolution of the meter, hence the input value required to achieve the reading on the meter under test.

Jan, are you going to also tell us those tests that you dont fly your colors so brightly. I can only say that I feel somewhat confused as to your reason for ignoring the basic rules for this test and to air your results. You may want to reconsider your future actions if you want to participate in any PT's that follow guide 43 and are administered by a third party in this country. Talk to NAPT. Obtain information. Grind your ax somewhere else.

Tom Smith
Uncertain of your motive
understanding your frustration
Maintaining self control.

[Marc]

Date: Wed, 13 Oct 1999 21:01:00 EDT
From: Rcveale@aol.com
To: iso25@fasor.com
Subject: Re: Uncertainty and Proficiency Testing

Jan,

You are right in at least one point. This Group doesn't agree on the answer to your question.

I don't know much about multimeters, but if it were a digital micrometer that read to the nearest 0.0001 unit, I think it would not be unreasonable to report an uncertainty less than 0.0001.

All digital micrometers that I have seen recently round the last place rather than truncate. Thus, a reading of 0.9985 means that the measurement is greater than 0.99845 and less than 0.99855.

Of course the uncertainty of a measurement might be many times this number. Perhaps the repeatability is five time the resolution. Perhaps we are measuring a rubber ball.

But if the resolution were the only, or dominate, factor in the error budget, we would compute the uncertainty by dividing the resolution by two (for an instrument that rounds the last digit), assume a rectangular distribution, divide by the square root of three, multiply by two (k=2), and arrive at an expanded uncertainty of +/-0.00006 unit.

It appears to me that rounding the +/-0.00006 to +/-0.0001 would be throwing away useful information.

Ralph Veale
ISO Guide 25 Assessor