# Should resolution be included in uncertainty budget for digital caliper or micrometer calibration?

#### caliperCalibration

##### Registered
Hi everyone,

I am working on creating a budget for uncertainty when calibrating a digital caliper and a digital micrometer. One of the first factors that I thought to include was resolution. Recently, I have begun to think that resolution should NOT be included, but it seems like every sample budget or source I can find has included resolution. I will explain my reasoning below (for a caliper but its the same for a micrometer) and hopefully people can chime in with their thoughts!

Okay so in the situation of calibrating a caliper, the measurand is the difference between what the caliper is reading and what the length of the gage block is (AKA the error of the caliper). I have this expressed with the equation error_caliper = reading - L_nominal*(1+CTE*delta_T). I am using the GUM method of taking the partial derivatives of this equation to obtain sensitivity coefficients, and then I just need the uncertainty of each variable in the equation.

This leads me to consider the uncertainty of the reading. The only factor that I think should be included is repeatability, not resolution. If we assume for a minute that repeatability is zero, then I think that the uncertainty of the reading would be 0. It would be an exact value since the reading could only ever be that value. There is no doubt in what the value truly is, as the value is just the reading and NOT the length of what is being measured.

The calibration situation that I have described is in contrast to using a caliper to measure the length of something. In that case, I would say the measurand is the length of the part, and therefore resolution would be included since the "true length" of the part could be anywhere within a range of values for the given reading on the caliper.

The fact that all the sources I have looked at seem to include resolution (like even the Z 540.3 handbook!) is causing me confusion. I would love to hear other's thoughts on this!

Thanks

#### Guest

##### On Holiday
the measurand is the difference between what the caliper is reading and what the length of the gage block is

I think you may have this wrong... Does this definition make a difference?

"The measurand, or measured quantity, is the physical quantity in metrology subject to measurement."

#### dwperron

Trusted Information Resource
I know that I always used the resolution as an uncertainty contributor in gages like micrometers, calipers, and height gages.
Often they are the largest uncertainty contributor.
I did this to stay "conservative" on my uncertainties, and to prevent being "caught" in an audit by someone who will be obviously much smarter than I am.

However, there is a gentleman at Mitutoyo, Jim Salsbury, who has been campaigning for years that resolution should not be considered as an uncertainty contributor for micrometers.

He has presented several papers on the topic:

Uncertainty Issues In Calibration. (Conference) | OSTI.GOV

https://ncsli.org/store/viewproduct.aspx?id=16892871

Last edited:

#### caliperCalibration

##### Registered
I think you may have this wrong... Does this definition make a difference?

"The measurand, or measured quantity, is the physical quantity in metrology subject to measurement."
My thinking is that the error of the caliper is still the measurand, although we are finding a value for it with a function of other measurements. We are accepting a caliper based on an acceptance limit on the error, so the error must be the measurand!

#### caliperCalibration

##### Registered
dwperron,

These are great resources -- exactly what I was looking for. Thank you!

#### greif

##### Involved In Discussions
It occurs to me that by including both resolution and repeatability, you will cover when resolution is poor (say, reading 0.001" on a digital micrometer), which will result typically in a zero value for repeatability, so resolution will then be a dominant factor. In many cases resolution is pretty small and not a dominant factor.