P

#### potato

We are in the process of performing a Gage R&R on a ruler that has 1/64" calibrated increments. ASTM E29 does not specifically address the conversion of a fraction to a decimal...however, I see two paths here:

1) round each measurement when the data is taken. In this case, our systems say that since the instrument is a ruler, then the error is 1/2 the increment, so in this case it is 1/128. Converted to decimal, it is .0078125. Adding and subtracting that from the decimal equivalent of the 1/64" = .015625", then the first uncertain digit would be the 2nd one. So, a reading of 1/64" = .02".

2) E29 states that for calculations, you can carry decimals until the end. So, mathematically, 1/64" = .015625" is a calculation, and then that data could be used to evaluate the R&R. This would assume all of the digits are significant, since they are a result of a calculation.

Which is the correct way to interpret E29? I know on this forum many do not agree with an R&R on a ruler, but we have to do it per our systems. FYI, we have systems to periodically check the ruler, recalibrate (or some say reverify)...Also, in the future, we will use a metric ruler, but unfortunately we are stuck with this for now. Anyway, my question is really a mathematical one...