# Measurement Resolution - How far to carry the decimal place out

J

#### Jon O

Measurement Resolution

Hello All,

I am curious to know how many of you determine how far to carry the decimal place out for measurement purposes. Are you basing this on your GR&R's? If you have have a process you have not yet GR&R'd (yes, should be done first) what do you use to start out with? 4, 5, 6 decimal places?? and what is the reasoning behind your selection? Please let me know what others are doing.

Regards,

Jon

D

#### D.Scott

Jon - I'm not sure I am the one to try to answer your question but it looks like all the stats guys/gals are on leave.

Just to get you started, I look first at the requirements of the job I am measuring. If the requirements are to 2 decimals, I look for 3 decimals in the measurement system. There is a "rule of thumb" of 10:1 which is commonly followed for most measurements in our industry.

As far as determining the suitability of the measurement system you would need a gage R&R.

To arbitrarilly set 4, 5 or 6 decimals as a start, would be overkill if the measurement you were making had spec limits with 1 decimal. The pros can elaborate here much better than I, but there is a thing called "Signifficant Numbers" which illustrates the logic in dropping numbers after a certain point (example = 100/3 = 33.33333333 and so on - sooner or later, who needs all those 3s?). The same theory is what allows "rounding up or down".

Don't lock yourself into a whole bunch of decimals unless the part/process spec requires it. We found that when we put in our new measurement system and "showed off" with 8 decimals on a layout that the part we used to measure at 3 decimals now became a requirement at 6.

Hope some of the others can shed more light for you.

Dave

J

#### Jon O

D. Scott,

Thanks for your feedback on the subject, it does help. I am also curious to hear others techniques and stratagies.

Regards,

Jon

A

#### Al Dyer

I've always used the 10 to 1 rule. If your target is .01 I would at least use a gage discriminated to .001

#### Marc

##### Hunkered Down for the Duration with a Mask on...
Staff member
Al really repeated what D Scott said - and as far as Mr. Scott's explaination, I agree.

Basically it boils down to (in the auto world early APQP) the fact that during early design criticval characteristics start emerging. At that point you should be determining to what precision you need a dimension to be. This is where the M&TE people come in and sit down and say we have equipment to measure that or we don't and thus will have to buy.

> Don't lock yourself into a whole bunch of decimals unless
> the part/process spec requires it. We found that when we
> put in our new measurement system and "showed off" with 8
> decimals on a layout that the part we used to measure at 3
> decimals now became a requirement at 6.

I agree with the first sentence, but will say that the precision of the instrument should never dictate the dimension precision. That should be a design function. If there was a part I had and the print gave me 'y' mm +/- 0.2 mm and I used a CMM precise to 0.0001, I would only use the first 2 digits after the decimal point from the CMM readings. Mr. Scott amply described these as the 'significant digits' - also Al's 10 to 1 rule. If you don't do this you're heading for trouble. If you are measuring at 0.01 yet are capable of measuring at 0.0001 (precision), the fact that you're capable of the measurement precision beyond the print callout may get you into trouble simply because your manufacturing equipment is not capable of that precision. As a last comment, if you change a dimension precision on a part based upon your measurement precision capability, it's going to be more expensive to make that part. Tightening a tolerance - which is what I think is being implied - is typically going to make the part more expensive.

The Batavia experiment was good proof that if you don't have the M&TE people in early, and you have 'company idiot' design engineers, you can end up on the other side of the stick - the precision necessary is more than your instrument (gage, CMM, whatever) can measure to.

There is another 'rule' to consider. It's something like 25% MAX (of tolerance) for combined 'uncertainty' (the 4 to 1 rule) of the measurement system as a whole. That should be taken into consideration when setting tolerances during design phase as well. Make sure the M&TE people are there when decisions like this are reviewed during the design stage.

I think.... Me not always right :thedeal: Not my specialty.

S

#### SSanap

The measuring equipment should have least count of 1/10 of the decimal of the tolerance limits. If the tolerance is + 0.1 unit the measurement should be upto 0.01unit.

Thanx,

SSanap

#### bobdoering

Trusted Information Resource
There has been an attempt to answer 3 questions here, that really need to be addressed separately:

What should the resolution of a gage be in relationship to the tolerance? 10:1 is a backyard rule of thumb that is adequate for just that - back yard work. For important measurement systems, the answer is a minimum ndc>=5, although I recommend ndc>=10, and for SPC ndc>=10 where PV=UCL-LCL.

What should the ratio of a master reference gage be in relationship to the gage being calibrated? Two schools of thought - 10:1 to the gage discrimination or 4:1 to the gage discrimination

and, the original question:
How far to carry the decimal place out in relationship to the tolerance? Rule of thumb: If the tolerance is wide enough to contain more than 10 units - for example 20-40mm, then 1mm increments is fine. But, if the tolerance is less than 10 increments, then use 10:1 to the unit measure - for example 3-5 mm, report to 0.1 mm. Bottom line, report to at least 10 increments between the specifications.

S

#### sixsigmais

I agree with D,Scott that
If the requirements are to 2 decimals, I look for 3 decimals in the measurement system
Yes, you are looking for more but it consider a waste, just give more accuracy than the requirement is good enough

#### Miner

##### Forum Moderator
Staff member
Donald Wheeler has proposed a sound statistical approach to making this decision. Read Intro to MSA of Continuous Data – Part 7: R&R using Wheeler’s Honest Gage Study The file attached to this blog entry contains links to Wheeler's articles as well as calculates the smallest and largest Effective Measurement Increment (EMI).

The resolution of the gage should be smaller than the largest EMI, and decimal places smaller than the smallest EMI do not provide additional value and should not be recorded.

B

#### BrQ

Is it acceptable to use a "for reference only" sticker, along with a calibration, if your caliper does not have adequate resolution to pass a gage R&R?
I am talking about steel tooling here, calipers are used "on the bench" but all final decisions are based on CMM / Micrometer (higher res) measurement results.
I know that the obvious answer is to buy higher res calipers, but this is currently not an option.

Does pitch/increment/resolution of a ruled scale apply to measurement uncertainty as line item? Measurement Uncertainty (MU) 10
B Total Resolution of an Measurement Instrument Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 1
P Measurement Equipment - 10:1, 4:1 - Is it resolution or accuracy General Measurement Device and Calibration Topics 1
When Does Resolution of Measurement Standard Apply in Uncertainty Calculations General Measurement Device and Calibration Topics 14
E Measuring Parts without Proper Tools - Measurement Device Resolution Inspection, Prints (Drawings), Testing, Sampling and Related Topics 10
J Definition Accuracy and Resolution in a Measurement System: What are the differences? Definitions, Acronyms, Abbreviations and Interpretations Listed Alphabetically 12
D The Origin of the 10:1 Resolution Ratio for Measurement Equipment General Measurement Device and Calibration Topics 17
Gage Resolution / Measurement Accuracy - Laser scan micrometer has a 6 digit display General Measurement Device and Calibration Topics 4
P Measurement Resolution of In-Process Automated Inspection System - Stamping House Inspection, Prints (Drawings), Testing, Sampling and Related Topics 13
E Cpk's, specifications and tiny sd's - Resolution of measurement equipment Capability, Accuracy and Stability - Processes, Machines, etc. 20
W Resolution and Sensitivity - What is measurement instrument or gage 'Sensitivity'? General Measurement Device and Calibration Topics 12
M Gage R&R for Optical System - Measurement Parameters and Resolution of the system Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 1
QMS monitoring, measurement, analysis and evaluation requirement - Template ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
Physical Part Measurement General Measurement Device and Calibration Topics 6
Measurement Uncertainty in Optical Microscopy Measurement Uncertainty (MU) 1
ISO 6508 and portable hardness measurement instruments General Measurement Device and Calibration Topics 0
How to Establish the Calibration & Measurement Capability (CMC)? ISO 17025 related Discussions 1
Measurement of residual magnetism General Measurement Device and Calibration Topics 2
Surgical angle measurement guide device with an application software Medical Device and FDA Regulations and Standards News 1
Disabling measurement data during fault conditions IEC 60601 - Medical Electrical Equipment Safety Standards Series 5
Most versatile measurement device General Measurement Device and Calibration Topics 2
Surface scratch measurement General Measurement Device and Calibration Topics 1
Drawing tolerance vs. Measurement device General Measurement Device and Calibration Topics 7
MSA on automated measurement system - Multiple Step Vision System Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 3
Thermo-Hygrometer - Measurement of 10DegC @10%RH ISO 17025 related Discussions 4
GDT Flatness measurement question Inspection, Prints (Drawings), Testing, Sampling and Related Topics 10
Software Validation for Measurement Tools used in Process Validation ISO 13485:2016 - Medical Device Quality Management Systems 2
Best Measurement Systems Demos in California? General Measurement Device and Calibration Topics 1
Measurement Error - How to determine what is acceptable? Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 3
Process Mapping - Inputs/Outputs/Detail Activities/Control points/Measurement? Process Maps, Process Mapping and Turtle Diagrams 4
IEC 60601-2-10 - Accuracy of Pulse Parameters - Required Measurement Uncertainty IEC 60601 - Medical Electrical Equipment Safety Standards Series 3
Program or application for standard time measurement in industry from video surveillance? Human Factors and Ergonomics in Engineering 1
Defining frequency of measurement tools callibration Calibration and Metrology Software and Hardware 3
How to “validate/qualify” a new measurement equipment? Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 4
Verification and Validation of Measurement Systems Misc. Quality Assurance and Business Systems Related Topics 0
How do you add accuracies for tools with different measurement devices? General Measurement Device and Calibration Topics 1
Viable count Measurement ISO Standard Other Medical Device Related Standards 1
Gauge R&R on a torque measurement Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 11
Air Quality Measurement Hardware and Software General Measurement Device and Calibration Topics 11
Dimension Measurement Tool Recommendation General Measurement Device and Calibration Topics 21
Plane measurement with a 3D touch machine like a CMM. Calibration and Metrology Software and Hardware 3
Diameters measurement with 3D touch machine like a CMM Calibration and Metrology Software and Hardware 4
Uncertainty in measurement larger than tolerance Measurement Uncertainty (MU) 2
Measurement Uncertainties Budget for Thermometer Measurement Uncertainty (MU) 6
Performance Measurement ISO 9001: 2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
Monitoring performance - How do I determine performance measurement basis within my organization? Misc. Quality Assurance and Business Systems Related Topics 4
What information to put on measurement Dimensional Results APQP and PPAP 7
O Ring capability and measurement - What is the automotive 'norm' for capability studies on O Rings? General Measurement Device and Calibration Topics 3
Electrical Metrology (Source/Generation and Measurement) General Measurement Device and Calibration Topics 1
Gauge R&R on multiple dimensions using 3D measurement system Capability, Accuracy and Stability - Processes, Machines, etc. 6