How many measurement data decimal positions to report and be in conformance

Fralem · Jan 3, 2012

Does anybody know if there is a standard on how many positions to report after the decimal?

Here is the situation: We have a technical drawing that states a dimension of X.XXX'' +/- 0.005'' and we have conflicting opinions on the subject as to how many places after the decimal must be reported back on the inspection report. Some say that the drawing says three so report three, others say that an extra position should be reported. So in the example above should we report 3 or 4 positions after the decimal.

That then leads me to another question: Either of the possibilities above what then happens if the reported position is the fourth (X.XXXX'') and the fifth or sixth position when I run the inspection routine would make the part out of specs. Is the part no good or are these positions to far out to matter as the technical drawing states 3 places?

Please enlighten me

Frank

Statistical Steven · Jan 3, 2012

Re: How far must the mearsurements be reported and be in conformance

Fralem said:
Does anybody know if there is a standard on how many positions to report after the decimal?

Here is the situation: We have a technical drawing that states a dimension of X.XXX'' +/- 0.005'' and we have conflicting opinions on the subject as to how many places after the decimal must be reported back on the inspection report. Some say that the drawing says three so report three, others say that an extra position should be reported. So in the example above should we report 3 or 4 positions after the decimal.

That then leads me to another question: Either of the possibilities above what then happens if the reported position is the fourth (X.XXXX'') and the fifth or sixth position when I run the inspection routine would make the part out of specs. Is the part no good or are these positions to far out to matter as the technical drawing states 3 places?

Please enlighten me

Frank

Frank you measure to the 4th decimal place and report on the record to the 3rd decimal place.

For example:

If you measure 5.4568 you report 5.457 which is compared to the specification.

If you measure 5.4562 you report 5.456 which is compared to the specification.

Kingsld1 · Jan 3, 2012

If you are working to ANSI Y14.5 then technically any amount above or below the stated limit is out of tolerance.

Personally, I would prefer to have parts that were not pushing the edge of the tolerance, even though my samples were just barely in tolerance. Keeps the customer from finding that one sample part that is just out of spec. (Been on both sides of that)

Jim Wynne · Jan 3, 2012

Fralem said:
Does anybody know if there is a standard on how many positions to report after the decimal?

Here is the situation: We have a technical drawing that states a dimension of X.XXX'' +/- 0.005'' and we have conflicting opinions on the subject as to how many places after the decimal must be reported back on the inspection report. Some say that the drawing says three so report three, others say that an extra position should be reported. So in the example above should we report 3 or 4 positions after the decimal.

That then leads me to another question: Either of the possibilities above what then happens if the reported position is the fourth (X.XXXX'') and the fifth or sixth position when I run the inspection routine would make the part out of specs. Is the part no good or are these positions to far out to matter as the technical drawing states 3 places?

Please enlighten me

Frank

This question comes up surprisingly often. The question has a simple answer, though. If your upper tolerance limit is (for example) 1.005 and your measurement is 1.0051 (or 1.0050000...1, you have exceeded the upper tolerance limit, which is an expression of what size the thing must be in order to be considered acceptable.

In actual practice, how you go about dealing with this situation depends on factors not in evidence here, and the approach suggested by Statistical Steven is often used. Be assured, though, that there is no general industry standard that it's OK to exceed a maximum value or fall below a minimum.

pkost · Jan 4, 2012

Jim Wynne said:
In actual practice, how you go about dealing with this situation depends on factors not in evidence here, and the approach suggested by Statistical Steven is often used. Be assured, though, that there is no general industry standard that it's OK to exceed a maximum value or fall below a minimum.

As long as you have documented how you make and record the measurement and justify why it is effective.

AndyN · Jan 4, 2012

pkost said:
As long as you have documented how you make and record the measurement and justify why it is effective.

???? - are you suggesting that there's a documented work instruction (or similar) on how to make a measurement? What makes you suggest such a thing is necessary? If you'd share that with us so we can better understand.

AndyN · Jan 4, 2012

Kingsld1 said:
If you are working to ANSI Y14.5 then technically any amount above or below the stated limit is out of tolerance.

Personally, I would prefer to have parts that were not pushing the edge of the tolerance, even though my samples were just barely in tolerance. Keeps the customer from finding that one sample part that is just out of spec. (Been on both sides of that)

Good point about the parts 'pushing the limit' - plus, no-one has mentioned the impact on the observed measurement the device's resolution can have. The resolution should be 4:1 (as a minimum) or 10:1 (better) to allow this determination to be helpful. I think Steven came close, but not how you measure and report to those digits.

pkost · Jan 4, 2012

AndyN said:
???? - are you suggesting that there's a documented work instruction (or similar) on how to make a measurement? What makes you suggest such a thing is necessary? If you'd share that with us so we can better understand.

My mistake - it appears that I misread the quote.

I do however believe it is important to specify the accuracy required in measurement especially when extreme accuracy is required. It leads to less argument or confusion!

Jim Wynne · Jan 4, 2012

pkost said:
My mistake - it appears that I misread the quote.

I do however believe it is important to specify the accuracy required in measurement especially when extreme accuracy is required. It leads to less argument or confusion!

The question isn't about accuracy/precision per se, although it's something to consider when near the spec limits. The question, essentially, is whether 1.0051 is greater than 1.005, assuming accurate measurement.

Statistical Steven · Jan 4, 2012

AndyN said:
Good point about the parts 'pushing the limit' - plus, no-one has mentioned the impact on the observed measurement the device's resolution can have. The resolution should be 4:1 (as a minimum) or 10:1 (better) to allow this determination to be helpful. I think Steven came close, but not how you measure and report to those digits.

Andy...that is my point. The measurement system has to have at least 10:1 resolution, so why report out more digits than required by the specification? I take a hard line stance on signficant figures especially in the FDA regulated industry where reporting out to more or less significant figures is an audit finding.

How many measurement data decimal positions to report and be in conformance

Fralem

Statistical Steven

Statistician

Kingsld1

Involved In Discussions

Jim Wynne

pkost

AndyN

Moved On