Measurement system that has a Least Count of 1/10th of the tolerance



A particular company has to maintain the temperature during the process within 28 +/- 1 degree celsius and uses a temperature controller graduated in steps of 1 degree.

Can anyone tell me whether this is a non-conformance and why?

The external auditor says that a measurement system that has a least count of 1/10th of the tolerance should be used.

Craig H.


If you have a digital thermometer that reads out in 1 degree increments, the effective range of the spec is 28 +/- 1.5. That is because the instrument will not read 26 or 30 until the temperature of the product is +/- 1.6.

In effect, your process has gained 33% more leeway.

There may be other reasons I am not aware of.


The 10 to 1 rule (resolution should be 1/10 of your tolerance, minimum) is a rule of thumb, not a "standard" or requirement as far as I know.

I would be somewhat suspect of your measurement method if you have a +/- 1 degree specification. In addition to the readout you also have to consider the sensor itself (thermocouple, RTD, etc.) You also have to consider rounding error. If the readout is 29 you could be anywhere from 28.5 to 29.4 degrees - possibly .4 out of specification.

Is there a true need to hold this specification? If so, it sounds like you need a more precise method to measure. Not sure I would/could call it an audit nonconformance.


Craig H.

Although I would not be comfortable with the above situation, it would be difficult to write a nonconformance for it during an ISO 9000 audit, unless there are specific customer instructions that are not being followed. However, it would get mentioned as an observation.


The rule of thumb is 4 - 10 to 1. And yes it would be a nonconformance unless you have documention that justifies the results of the present instrument.

Ravi Khare

The rule of thumb while selecting the measurement system Least Count is 1/10.

Then there are other factors like R&R error, Linearity and Bias which result in your being able to measure within the tolerance zone in 5 or 6 distinct steps. The MSA Manual published by AIAG recommends that you should have 5 or more distinct data categories. Only then, you will be able to control and hold the parameter within central 20% of the bilateral tolerance zone.
Top Bottom