Significant digits when reporting measurements

[vforge]

What is the accepted practice when reporting dimensional results with regard to significant digits and rounding?

For example, if a feature is toleranced as 5.00 +/- 0.25, should we report to 3 places (measurement is 5.252 = out of spec) or do we round down to 2 places and report 5.25 = in spec?

I'm new to quality, so trying to grasp the basics. Thanks.

[Bill Ryan]

Welcome to the Cove

Our normal "practice" is to report one digit further than the callout. We don't have that documented internally but I seem to remember seeing it as a "should" somewhere (I require it for any documentation I may be sending to our customer). You may be, momentarily, "fooling" the customer (and yourself) but if you take variability into account, you could end up shipping product which will cause a "no build" or loss of functionality.

Good luck!!

Bill

[Wes Bucey]

Originally Posted by Bill Ryan

Welcome to the Cove

Our normal "practice" is to report one digit further than the callout. We don't have that documented internally but I seem to remember seeing it as a "should" somewhere (I require it for any documentation I may be sending to our customer). You may be, momentarily, "fooling" the customer (and yourself) but if you take variability into account, you could end up shipping product which will cause a "no build" or loss of functionality.

Good luck!!

Bill

When I was in the high tech aerospace/medical machining business, we made it a policy to cover this issue in Contract Review with the customer before we ever cut metal.

As an aside, we always discussed the form, fit, and function aspects of a component we made for customers (everything custom to customer's design.)

We found it helpful to request tolerances as either
(+0.000 / -0.xxx) or (+0.xxx / -0.000),
instead of (+/-0.yyy),
depending on the mating part. We had a nice little set of toys to illustrate our point (male mating parts with dimension at maximum material condition and female mating parts the opposite.)

Sometimes the customer's design engineers would have an epiphany and modify ALL their drawings to show tolerances in our suggested method.

[Sam]

Normally you use an instrument that has a discrimination of ten times the tolerance or as Bill stated "one more place than the callout", which means you report the result to 3 places. Based on this I would say no you cannot round down.

[Graeme]

Originally Posted by vforge

What is the accepted practice when reporting dimensional results with regard to significant digits and rounding?

Practices vary, of course, and should conform to your customer's requirements. However, the applicable standard is:

ASTM E 29, Using Significant Digits in Test Data to Determine Conformance With Specifications.

Thist can be purchased (paper or electronic) from ASTM (www.astm.org).

According to the footnotes, this standard was first published in 1940 and reaffirmed in 1999.