# Conversion of fraction to decimal for R&R of english ruler

P

#### potato

Experts,

We are in the process of performing a Gage R&R on a ruler that has 1/64" calibrated increments. ASTM E29 does not specifically address the conversion of a fraction to a decimal...however, I see two paths here:

1) round each measurement when the data is taken. In this case, our systems say that since the instrument is a ruler, then the error is 1/2 the increment, so in this case it is 1/128. Converted to decimal, it is .0078125. Adding and subtracting that from the decimal equivalent of the 1/64" = .015625", then the first uncertain digit would be the 2nd one. So, a reading of 1/64" = .02".

2) E29 states that for calculations, you can carry decimals until the end. So, mathematically, 1/64" = .015625" is a calculation, and then that data could be used to evaluate the R&R. This would assume all of the digits are significant, since they are a result of a calculation.

Which is the correct way to interpret E29? I know on this forum many do not agree with an R&R on a ruler, but we have to do it per our systems. FYI, we have systems to periodically check the ruler, recalibrate (or some say reverify)...Also, in the future, we will use a metric ruler, but unfortunately we are stuck with this for now. Anyway, my question is really a mathematical one...

#### Miner

##### Forum Moderator
Staff member
I doubt that the difference between the two approaches will make a substantive difference in the results of the R&R study. Try it both ways and see.

From a purist perspective, you should cut it off at fewer decimal places, because that reflects the actual resolution of the measurement device. From a practical perspective, it probably will make little difference.

#### bobdoering

Trusted Information Resource
The beauty of the gage R&R is it will tell you how many decimal points are significant when you review the ndc.

P

#### potato

Miner, you are correct, there is little difference (we ran it both ways). However, it seems completely arbitrary that if you use a ruler that just so happens to not have several digits when you convert the fraction, that you don't have to round. For example, if we used a 1/50 ruler, we would not have to round at all. Just by changing the denominator to 64, we now have to remove those digits.

On the other hand, I see the point about removing the digits based upon resolution. But, that doesn't explain why 1/50 just so happens to be lucky...

#### bobdoering

Trusted Information Resource
1/64 = .0156 1/126=.0078. Based on the fact that you can not be as accurate as 1/128 reading 1/64, any decimal past .001 is clearly insignificant - you leave those off for sure. In the end, you will find resolution in .01 increments will be reportable, but barely significant.

#### bobdoering

Trusted Information Resource
I use 1/100" rulers to explain the difference between gage resolution and significant resolution (ndc). Sure, the gage reads out in .01" increments, but it is not that accurate...because no one can see, read or count the darn marks. There is also greater errors at that resolution in parallax, etc. Can't be done....might need a magnifier - but that's cheating. Besides, you will get manipulative error and observational error if you try to use a magnifier.

P

#### potato

OK...so assuming we need to round before we record the data, maybe my organization doesn't understand which digit is uncertain...

Here is our current logic:
1/64"=.015625
half of that is the error, so
1/128'=.0078125
So, a measurement of 1/64" could be plus or minus the error.
lowest=.0078125
highest=.0234375
So, the first uncertain number is the second digit. According to our review, you report the all of the certain digits and the first uncertain. In this case (if C=certain, U=uncertain), we report .CU, out to two digits.
Therefore, 1/64=.015625 = .02

Is this the incorrect interpretation of uncertain digits?

Conversion of coordinate dimension holes to position dimension Design and Development of Products and Processes 3
MDD to MDR - Technical File conversion EU Medical Device Regulations 4
S Class II to OTC conversion of a medical device 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 1
B How do you manage your project? Data Conversion and Digitization Records and Data - Quality, Legal and Other Evidence 9
Surface Roughness - Conversion of RS on a Print to RZ General Measurement Device and Calibration Topics 3
P Effective way for DC/DC conversion for 2 MOPP @ 250VAC IEC 60601 - Medical Electrical Equipment Safety Standards Series 3
B Conversion and Rounding Standard for converting metric to english and vice versa Other ISO and International Standards and European Regulations 4
P IEC 60601-1-11 - Conversion to Double Insulation IEC 60601 - Medical Electrical Equipment Safety Standards Series 2
M Customer Conversion and Retention / Satisfaction Measurement Index Benchmarking 4
S Quality Objectives for Aerospace Chemical Conversion & other Surface Treatment Plant AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 3
B QMS Conversion including Documentation and Understanding our Processes ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
R Preparation for AS9100B conversion to AS9100C AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 5
S FSC Chain of Custody - Volume Control & Conversion Factors Various Other Specifications, Standards, and related Requirements 7
S Surface Finish - Conversion factor to transform Rz Jis in Rz Inspection, Prints (Drawings), Testing, Sampling and Related Topics 6
T Specification Conversion from Sets to Individual Measurements? Inspection, Prints (Drawings), Testing, Sampling and Related Topics 5
F Conversion from Agile - Need cheaper FDA compliant software Qualification and Validation (including 21 CFR Part 11) 7
C Conversion Help!! Print with this dimension .2917+/-.0005 :1 Inspection, Prints (Drawings), Testing, Sampling and Related Topics 9
Unit Conversion Add-in for Excel Excel .xls Spreadsheet Templates and Tools 7
I Medical Device Directive, Conversion from class IIb to class III EU Medical Device Regulations 4
A CMM Software Conversion - Converting from Zeiss/Calypso to Brown & sharpe PCDMIS Inspection, Prints (Drawings), Testing, Sampling and Related Topics 7
K Micron to Inch Conversion Misc. Quality Assurance and Business Systems Related Topics 10
P Rockwell, Brinell, Vickers Conversion - Evaluating Aluminum 6061-T6 and 6063-T6 General Measurement Device and Calibration Topics 11
E Cost Savings based on 2005 Production - Metric Conversion Problem Misc. Quality Assurance and Business Systems Related Topics 4
Conversion of surface finish measurements Inspection, Prints (Drawings), Testing, Sampling and Related Topics 4
T What is the conversion for PPM defects vs. Sigma Equivalents? Six Sigma 42
Microfinish conversion - Micrometers (?) to microinches? Manufacturing and Related Processes 3
Conversion TO ISO9001:2000 / TS16949:2002E IATF 16949 - Automotive Quality Systems Standard 7
K Conversion from ISO 9001:1994 to ISO 13485:2003 ISO 13485:2016 - Medical Device Quality Management Systems 4
I ISO 9000:2000 Conversion Presentation Request ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 1
Best 'Units' Conversion Web Site and/or Program Book, Video, Blog and Web Site Reviews and Recommendations 6
S Conversion from ISO 9001:1994 to 9001:2000 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
G Calculating the Fraction Defective Confidence Interval for a Lot Statistical Analysis Tools, Techniques and SPC 2
I Screening: Blackett Burman versus Fraction Factorial Quality Tools, Improvement and Analysis 2
Dr Wheeler gives his thoughts on "Estimating the Fraction Nonconforming" Capability, Accuracy and Stability - Processes, Machines, etc. 0
M What are the strong points of Fraction Nonconforming vs. Cpk Capability, Accuracy and Stability - Processes, Machines, etc. 6
J Sample size for fraction defective of a population Six Sigma 1
G Decimal places on Drawing (Print) Callouts Measurement Uncertainty (MU) 7
H Tolerance, decimal places, and measuring equipment capability Inspection, Prints (Drawings), Testing, Sampling and Related Topics 2
B Decimal Places - Redux Using Minitab Software 4
Excel Number Formatting - Auto Decimal Place by What You Type Excel .xls Spreadsheet Templates and Tools 16
F How many measurement data decimal positions to report and be in conformance ISO 13485:2016 - Medical Device Quality Management Systems 28
L Changing the number of decimal places in MINITAB 16 Using Minitab Software 1
Calibration of Pin Gages using a Calibrated Caliper that measures to 4 decimal places General Measurement Device and Calibration Topics 15
B How to change Minitab Decimal Places Using Minitab Software 4
M Controlling Data Labels on Graphs - Number of Decimal Places Reported - Minitab Using Minitab Software 7
S Feature perpendicularity dimension should be good up to how many decimal points? Manufacturing and Related Processes 8
C Conversions - Decimal degrees into thousands - CMM that measures an angle Inspection, Prints (Drawings), Testing, Sampling and Related Topics 22
C Interpretation of Decimal Readings Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 13
J Samples Selection in Gage R&R - Number of Decimal Places Required? Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 1
D ISO 6508-2 and Hardness Gage Testers - Apostrophes where I expect a decimal point Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 1