Should resolution be included in uncertainty budget for digital caliper or micrometer calibration?

Hi everyone,

I am working on creating a budget for uncertainty when calibrating a digital caliper and a digital micrometer. One of the first factors that I thought to include was resolution. Recently, I have begun to think that resolution should NOT be included, but it seems like every sample budget or source I can find has included resolution. I will explain my reasoning below (for a caliper but its the same for a micrometer) and hopefully people can chime in with their thoughts!

Okay so in the situation of calibrating a caliper, the measurand is the difference between what the caliper is reading and what the length of the gage block is (AKA the error of the caliper). I have this expressed with the equation error_caliper = reading - L_nominal*(1+CTE*delta_T). I am using the GUM method of taking the partial derivatives of this equation to obtain sensitivity coefficients, and then I just need the uncertainty of each variable in the equation.

This leads me to consider the uncertainty of the reading. The only factor that I think should be included is repeatability, not resolution. If we assume for a minute that repeatability is zero, then I think that the uncertainty of the reading would be 0. It would be an exact value since the reading could only ever be that value. There is no doubt in what the value truly is, as the value is just the reading and NOT the length of what is being measured.

The calibration situation that I have described is in contrast to using a caliper to measure the length of something. In that case, I would say the measurand is the length of the part, and therefore resolution would be included since the "true length" of the part could be anywhere within a range of values for the given reading on the caliper.

The fact that all the sources I have looked at seem to include resolution (like even the Z 540.3 handbook!) is causing me confusion. I would love to hear other's thoughts on this!

Elsmar Forum Sponsor


Trusted Information Resource
I know that I always used the resolution as an uncertainty contributor in gages like micrometers, calipers, and height gages.
Often they are the largest uncertainty contributor.
I did this to stay "conservative" on my uncertainties, and to prevent being "caught" in an audit by someone who will be obviously much smarter than I am.

However, there is a gentleman at Mitutoyo, Jim Salsbury, who has been campaigning for years that resolution should not be considered as an uncertainty contributor for micrometers.

He has presented several papers on the topic:

Uncertainty Issues In Calibration. (Conference) | OSTI.GOV

Tech Paper Archives - Mitutoyo America Corporation
Last edited:
I think you may have this wrong... Does this definition make a difference?

"The measurand, or measured quantity, is the physical quantity in metrology subject to measurement."
My thinking is that the error of the caliper is still the measurand, although we are finding a value for it with a function of other measurements. We are accepting a caliper based on an acceptance limit on the error, so the error must be the measurand!


Involved In Discussions
It occurs to me that by including both resolution and repeatability, you will cover when resolution is poor (say, reading 0.001" on a digital micrometer), which will result typically in a zero value for repeatability, so resolution will then be a dominant factor. In many cases resolution is pretty small and not a dominant factor.
Thread starter Similar threads Forum Replies Date
E Please help! ! I AM FRASTRATED SO MUCH! Should inactive ingredient be included in the unit formula if it is removed during the manufacturing process? US Food and Drug Administration (FDA) 0
I IEC 60812 or ISO 14971 for PFMEA? What should we use? ISO 14971 - Medical Device Risk Management 3
qualprod Do sum of results of quality objectives should met a high level goal? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 10
F Should I perform several Gage R&R for every caliper to cover all specs range of use? Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 9
E Opening meeting for Third-Party Audit--Who should say what? General Auditing Discussions 22
8 MSA--Should I Use Parts or Gage Pins? Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 2
B Should I buy IEC 62304:2006, IEC 62304:2006/AMD 1:2015 or both? EU Medical Device Regulations 1
W "Accurate to ±2%" means "64% of measurements should be within 2% of the actual value"? US Medical Device Regulations 4
H Should we stop inspecting a part if its never been rejected? Reliability Analysis - Predictions, Testing and Standards 6
D What distributor requirement should we (manufacturer) require from the Distributor? EU Medical Device Regulations 0
M Should there be another column in the Optimization section (step 6) of AIAG-VDA DFMEA form? FMEA and Control Plans 1
C By when should harmonized standards be complied with? EU Medical Device Regulations 5
W Strategy for determining which components from a system should be "ME EQUIPMENT" -- home healthcare environment IEC 60601 - Medical Electrical Equipment Safety Standards Series 6
B Documented information - Should be controlled? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 10
Q Is engineering a process and should it have its own process turtle? IATF 16949 - Automotive Quality Systems Standard 7
P Which MDCG Document should I use for PMS Plan and PSUR? EU Medical Device Regulations 2
M Who should quality representative report to? ISO 13485:2016 - Medical Device Quality Management Systems 5
H When should the first PSUR be issued? EU Medical Device Regulations 5
I If i do not want to be an initial importer should i register with FDA? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 0
O Should a Covid vaccine and testing policy be included as part of ISO9001 or AS9100 risk management? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 6
B UKRP to what level should you audit Class I Technical Documentation? UK Medical Device Regulations 0
C When should you quit programming? Job Openings, Consulting and Employment Opportunities 9
Ed Panek External Standards List - Should this document include previously revised standards? ISO 13485:2016 - Medical Device Quality Management Systems 4
T How should I approach REACH, CM, etc. as a job shop? RoHS, REACH, ELV, IMDS and Restricted Substances 18
A Should we assign the PRRC before the date of application of MDR (26 May 2021)? EU Medical Device Regulations 0
J UDI-DI how should we interpret Device version or model to determine if a new UDI-DI is needed? EU Medical Device Regulations 0
Sidney Vianna Interesting Discussion Should ISO 9004 be changed from a guidance document to a requirements standard? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 13
A Should I take an online course for a career in Occupational Health and Safety? Career and Occupation Discussions 2
J Should a Class 1 medical device with an option to measure body weight be considered Class 1m? EU Medical Device Regulations 0
K Should APQP/PPAP has its own section in a QM? Quality Management System (QMS) Manuals 1
S What should i choose for "testing procedure" characteristics? (N95) General Information Resources 0
P Should eIFU link per ISO 15223-1:2016 be added to labels out of scope of Reg 207/2012? EU Medical Device Regulations 1
S Which Sampling Plan(s) Should I Use? Inspection, Prints (Drawings), Testing, Sampling and Related Topics 13
A Document release vs its related training. Which should come first? ISO 13485:2016 - Medical Device Quality Management Systems 18
S Which department should prepare the control plan? could you show me a standard regarding to this matter. FMEA and Control Plans 17
J Help settle a disagreement: Should external providers of preventive maintenance be on your ASL? AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 5
N Master Samples - What should we be keeping? IATF 16949 - Automotive Quality Systems Standard 9
G Supplier delivered recent PPAP, should he deliver yearly layout inspection? IATF 16949 - Automotive Quality Systems Standard 7
John Broomfield Should ISO 9004 Become a Requirements Standard? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 70
A Capability Study - in the beginning of your career what should you have known about the tool Quality Tools, Improvement and Analysis 11
J Should Loading and Unloading be Included in Cycle Times? Lean in Manufacturing and Service Industries 14
E Manufacturers should develop a testing device for covid19 Service Industry Specific Topics 0
T 510(k) submission - Which name should I use in the submission? Other US Medical Device Regulations 3
N ISO 19011:2018 - 5.4.2 "...audit program should engage in appropriate continual development..." Training - Internal, External, Online and Distance Learning 4
G Should I perform Gage R&R only at the beginning of a new project? Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 6
DuncanGibbons Should the requirements FAA/EASA Part 21 be addressed within the QMS and AS9100D quality manual? AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 5
M Should 510(k) Predicates be Actively Listed Devices? Other US Medical Device Regulations 12
B Why the Greek god Hephaestus should have done a design FMEA (DFMEA) on his giant robot APQP and PPAP 1
J On PFMEA for danger labels - Label always should be assigned severity 10 ? FMEA and Control Plans 3
H Who should be listed as the manufacturer/distributor on the box? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 15

Similar threads

Top Bottom