P

#### Paul A. Pelletier

My question is:

What is the proper way to specify and evaluate a dimensional requirement in terms of its designed precision versus the uncertainty and accuracy of its measurement?

Background

ASME Y 14.5M section 2.4 (Interpretation of Limits) states that: "All limits are absolute. Dimensional limits, regardless of the number of decimal places, are used as if they were continued with zeros (ex. 12.2 means 12.20....0). To determine conformance with limits, the measured value is compared directly with the specified value and any deviation outside the specified limiting value signifies nonconformance with the limits".

The as specified precision of a dimensional requirement (or any value for that matter) is generally represented by the number of decimal places to the right of the decimal point. For instance, a value of .125" is a less precise value than .1250". I cannot quote the source of this axiom. I can tell you that I was instructed this way as an engineering undergraduate, and that in the 20+ years since then, I have never encountered an engineer, machinist, inspector, draftsman, or designer that disagreed with it. If ASME does not concur with this axiom, I would like to have that confirmed as well as the rational or specification that ASME believes governs the conveyance of the level of precision in a dimensional requirement.

If this axiom is true, then section 2.4 of ASME Y 14.5M is at the very least confusing (i.e. 12.2 does not mean 12.20.....0).

Related to the correct way to specify a level of precision for a dimensional requirement is the correct way to inspect the part for conformance to that requirement. A second axiom that I have applied through out my professional experience is that one should inspect a technical requirement with an instrument that is more precise than the requirement as specified. This "incremental precision" is generally accepted to be 1 order of magnitude. The one caveat being that this would not be done if 1 order of magnitude exceeded the "state of the art" in measurement capability. This axiom is (was) documented at one time in the military specifications (MIL-STD-45662, I believe). Regardless, it is certainly logical that if one wishes to evaluate a requirement specified to .000" that it should be evaluated with an instrument capable of resolving .0000", otherwise the measured value of the third decimal place is too uncertain as to be considered reliable.

The third element of my question is what one does with the value measured in the "incremental precision position". The generally accepted practice that I have encountered is to round the "incremental precision position" up or down to the level of precision as specified by the designer (as represented by the number of decimal places in the dimensional requirement). This third piece to my question is the part that seems to be most at odds with ASME Y 14.5M section 2.4. When the standard states "All limits are absolute. .....To determine conformance with limits, the measured value is compared directly with the specified value and any deviation outside the specified limiting value signifies nonconformance with the limits", an interpretation could be that the "generally accepted practice" of rounding the "incremental precision position" in any direction is wrong. This phrase in the standard would imply that a dimensional requirement of .125 +/- .005" (specified value of .125", specified limiting value of .130") should be rejected for an as measured value of .1301", or for that matter .130001". Is that true? Is this .0001" or .000001" what ASME means when you say "outside the specified limiting value"?

This is the fundamental problem that I am requesting assistance in understanding, can one legitimately round the least significant digit (s) of measurements to the number of decimal places as specified by the designer, which for the sake of this question is the defined level of precision for the requirement, without violating the specified limiting value?

As a final illustration, consider the following: A hole diameter is specified to be .121+/- .001". The part is measured with an instrument that is capable (and calibrated) of resolving .00001", with the results being that the part measures .12210". Should the part be accepted? If the answer is no, then should it be accepted if the measured value is .12209"? Should it be accepted if the measured value is .12201"?

If your answer is that an instrument capable of resolving to .00000" is the wrong instrument, then what is the correct interpretation if the part were inspected with an instrument capable of resolving only .0000", and the measured value is .1220" when in reality the actual value is .12204". Is not the instrument rounding down? Does this not violate the intention of the standard?

I would greatly appreciate clarification on this issue. My company's intention is to unambiguously comply with internationally recognized standards, of which we consider ASME Y14.5M a critical element. Up until an internal difference of opinion resulted in a thorough research of the applicable standards (we are in the process of researching ANSI/IEEE 268 for additional guidance), we thought that we were doing so. After reviewing the standard it is not clear if we are or are not. I would submit that section 4 of ASME Y14.5M falls just short of defining the interpretation of limits in terms of the evaluation of parts for their conformance to those limits. This is the area that we are in need of clarification on.