Number of 'significant digits' in measurements

[57jeeper]

Hello to the group:

We have a metric drawing from a European medical company that has a dimension of 4.3 +/- 0.01 mm for a feature. We are measuring the dimension to .001 resolution on a CMM and rejecting for anything less than the low limit of 4.290 mm.
There is an arguement that we should only report to 2 decimil places because that is what the tolerance is expressed as.
This would mean that we would need .005 mm to trip the reading to the next lower 'significant digit' to show an out of tolerance condition.
That would mean anything within the range of 4.285 to 4.315 would read as acceptable if rounded to 2 decimil places.
My thinking is that this practice would add another 50% tolerance to the dimension which I don't believe is good practice.
My question is, is there a DIN or ISO specification that deals with this subject?
I've already checked ISO 2768-1&2 and 8015 with no success.

Thanks,

Mike

 

[Jim Wynne]

Re: Number of 'significant digits' question.

ANSI/ASME Y14.5M-1994 (I don't have the current version at hand at the moment) says this:

All limits​

​

​

​

are absolute. Dimensional limits, regardless
of the number of decimal places, are used as if​

they were continued with zeros.

I assume that later versions carry the same requirement. We've discussed this here at the Cove before. The general understanding is that the limits given are absolute and no rounding should be done if it changes the tolerance limit. 1.000001 is > 1.00.

 

[Michael_M]

Re: Number of 'significant digits' question.

Hello to the group:

We have a metric drawing from a European medical company that has a dimension of 4.3 +/- 0.01 mm for a feature. We are measuring the dimension to .001 resolution on a CMM and rejecting for anything less than the low limit of 4.290 mm. <snip>

Not sure what CMM or software you are using, but the CMM/Software I use will still report out of tolerance when viewing only 2 decimal points. For example, a dimension of 1.00 +/- .010. If I measure 1.010001, it will show 'out of tolerance' even if all I see is 1.01.

However, Jim is correct. The tolerances are absolute regardless of the number of decimal places.

 

[Statistical Steven]

Re: Number of 'significant digits' question.

ANSI/ASME Y14.5M-1994 (I don't have the current version at hand at the moment) says this:

All limits​

​

​

​

are absolute. Dimensional limits, regardless
of the number of decimal places, are used as if​

they were continued with zeros.

I assume that later versions carry the same requirement. We've discussed this here at the Cove before. The general understanding is that the limits given are absolute and no rounding should be done if it changes the tolerance limit. 1.000001 is > 1.00.

Jim, we have disagreed on this subject before and I understand the standard, but if the specifciation is 4.3 +/- 0.01 then why should I use a measurement system that reads to any more digits than 0.01?

 

[Michael_M]

Re: Number of 'significant digits' question.

Jim, we have disagreed on this subject before and I understand the standard, but if the specifciation is 4.3 +/- 0.01 then why should I use a measurement system that reads to any more digits than 0.01?

If you only use a device that can measure .01, then there will be only three numbers allowed; 4.2, 4.3, and 4.4. Further, there is no proof that the 4.2 and the 4.4 numbers are acceptable as they could be measuring 4.449 and 4.2 could be measuring 4.15. The total error is equal to the stated tolerance.

 

[Bev D]

Re: Number of 'significant digits' question.

Jim, we have disagreed on this subject before and I understand the standard, but if the specifciation is 4.3 +/- 0.01 then why should I use a measurement system that reads to any more digits than 0.01?

Steven - this is more than a mathematical exercise or 'what measuring device do I need' exercise. in the vast majority of manufacturing (aerospace, defense, automotive, semiconductors, etc., Jim's position is taken as correct - it's in the governing ANSI standard. (Those who debate this are typically only trying to weazle out of rejecting bad parts - not saying that you are, you are actually trying to understand the reasoning).

Here is the reason:

If the spec limit is correct then anything greater or less than the limit is to be rejected because it won't work. (or given stackups it won't work with at least some of it's mating parts.)

If the spec limit is wrong, the argument isn't about the tolerance, it's about changing the specification limits.
​

I know this sounds kind of harsh, but really, in my organization I do not tolerate this discussion as it is just 'lazy' engineering. If we have to argue about the tolerance then we must have parts at the limits. THAT is the issue to discuss and fix: make better parts. we are simply wasting time adn money discussing the tolerancing rounding question. it is diversionary from the real issue.

 

[Statistical Steven]

Re: Number of 'significant digits' question.

Steven - this is more than a mathematical exercise or 'what measuring device do I need' exercise. in the vast majority of manufacturign (aerosapce, defense, automotive, semiconductors, etc., Jim's position is taken as correct - it's in the governing ANSI standard. (Those who debate this are typically only trying to weazle out of rejecting bad parts - not saying that you are, you are actually tryign to understand the reasoning). <snip>

I understand the issue at hand...but the discussion of significant figures is not black and white.

If your spec is 4.3 +/- 0.10 then by the "standard" you can only have the following values 4.20 4.21 4.22 4.23 ... 4.40 if you use a gage the reads to 0.01. This would hold true if the spec was 4.3 +/-0.1 or 4.3 +/- 0.100. So if I understand you correctly, there is NO need to specify the extra trailing zero since they are assumed to be there.

But were are talking SIGNIFICANT figures so the specification sets at what point the digits are no longer significant. There is a difference between +/- 0.1 and +/- 0.10 in the first case a value of 4.19 is rounded to 4.2 and in specification in the second it is not rounded and is out of specification.

 

[Bev D]

Re: Number of 'significant digits' question.

I understand the issue at hand...but the discussion of significant figures is not black and white.



But were are talking SIGNIFICANT figures so the specification sets at what point the digits are no longer significant. There is a difference between +/- 0.1 and +/- 0.10 in the first case a value of 4.19 is rounded to 4.2 and in specification in the second it is not rounded and is out of specification.

I think you have nailed it: we are NOT talking about significant digits. that is a diversionary question. the real question is about parts being in spec or not in spec. significant digits are irrelevent...

most parts aren't actually within a teeny bit of the spec. ANd when you wrp in the measurement error inherent in system it is usually larger than the amount of the significant digit question. So, instead of wasting time debating it, we should redirect the converstaion to improving the process which is the big issue AND gets us away from the significant digits diversion.

while I enjoy a good academic argument about how many angels can fit on the head of a pin it doesn't bring me any revenue or ensure customer loyalty. that is why the ANSI standard is clear.

 

[Statistical Steven]

Re: Number of 'significant digits' question.

I think you have nailed it: we are NOT talking about significant digits. that is a diversionary question. the real question is about parts being in spec or not in spec. significant digits are irrelevent...

most parts aren't actually within a teeny bit of the spec. ANd when you wrp in the measurement error inherent in system it is usually larger than the amount of the significant digit question. So, instead of wasting time debating it, we should redirect the converstaion to improving the process which is the big issue AND gets us away from the significant digits diversion.

while I enjoy a good academic argument about how many angels can fit on the head of a pin it doesn't bring me any revenue or ensure customer loyalty. that is why the ANSI standard is clear.

We will agree to disagree on this topic. I understand the ANSI standard and its intention, but I cannot reconcile it with how specifications are set. It implies that a specification of +/- 0.1 and +/- 0.10 are the same. I do not agree with that assertion, but I appreciate the insight.

 

[Jim Wynne]

Re: Number of 'significant digits' question.

<snip>But were are talking SIGNIFICANT figures so the specification sets at what point the digits are no longer significant.<snip>

No. The number of decimal places used in the specification indicates only what tolerance to apply (if that's the way tolerances are specified), not when anything becomes insignificant. I've seen lots of drawings where there were specifications in two and three decimal places with a note indicating a global tolerance. It should never be assumed that a tolerance of 10.00 ?.010 means that anything greater than 10.010 is OK.

As for measurement device resolution, there's a reason for the 10-1 "rule."