Search the Elsmar Cove!
**Search ALL of** with DuckDuckGo including content not in the forum - Search results with No ads.

FAI and ANSI Y14.5 Trailing Zeros and Print Tolerances

Brad Gover

Involved In Discussions
Hi everybody! I hope I am posting this thread on the right Forum. I have a question in reporting a measurement point on AS 9102 forms. The print calls out for a diameter of 0.302" +/- 0.005". Our inspection plan requires the use of a CMM to measure the diameter. The CMM reports out the diameter to be 0.307005". The inspectors are labeling this as Fail. The print requires 3 significant figures and I am wondering if I should report this to 0.307". I have not found any documentation requiring trailing zeros or reporting out data to the resolution of the instrument used to verify part conformance to print. Can I have the CMM programmer report out data to 4 digits and round to three digits?

John Predmore

Involved In Discussions
I personally would report results with same number of decimal places as the dimension and/or tolerance, plus 1.

The added precision of one place communicates to the customer whether your company rounds up or truncates to conclude a dimension falls inside or outside a tolerance band. The extra place of precision is consistent with one version of the Rule of Ten to One (see below). Because of measurement uncertainty, a precision of millionths of an inch is little guarantee that a measurement taken in millionths is accurate or repeatable. A CMM is notorious for measuring very precisely, but CMM results for a diameter will vary based on number of sample points, whether the diameter is best-fit or inscribed/circumscribed circle, where on the cylinder the points are taken.

Can someone please clarify the Rule of 10 to 1, what it is exactly, what I use it for, and so on? An example would help.
A more thoughtful answer to your question would to be guided by an understanding of the criticality of the dimension/tolerance. My assumption is the customer's part drawing communicates criticality in the number of places of the dimension/tolerance. If the customer truly wanted .307005" to be rejected, the customer should have communicated this with a print dimension of 0.302000 +/- .005000".

Of course, if your first article parts are in the region of 0.307005", you would be wise to take rapid action, or soon your parts will measure 0.3075" or above, and your process yield will deteriorate to unacceptable rates.

Jim Wynne

Super Moderator
This subject has been discussed before, but this is a bit of a special case, I think. First, we should not assume that a customer is trying to communicate in code with their drawing specifications. .302 ± .005 means that .307 is the high limit, no matter how many decimal places you add. .307005 is > .307, so the high limit has been exceeded.

As John Predmore has pointed out, measuring diameters is a tricky business, because there is a theoretically infinite number of diameters that can be measured in any given cylinder. You should never measure hole size with a CMM unless roundness is a concern. Furthermore, the chances are vanishingly small that the sixth decimal place is important. It's common to measure such holes with gage pins of the "minus" variety (actual size being .0002" smaller than nominal) and then report the size of the largest pin that will fit in the hole. Another option is just to report the hole size at the high limit. IMO, the problem here is (a) not making sure that the size is comfortably away from the tolerance limits in production and (b) using the wrong measuring device.


Quite Involved in Discussions
After considering the requirements of the print .302 +/- .005 - first is not a critical dimension, second how many points did you take, third, what is the roundness and fourth what is the temperature of the material / environment.

Train the oprator to think outside the box, incorporate all te factors and make a decision. it is.307 with i n tolerance - in the future is a plug gage
Top Bottom