Determining a tolerance value for Measuring devices in-house inspection


Starting to get Involved
Hello All,
Hope everyone's doing good. I have a question here.
I am trying to understand this. This is an Inspection Record sheet for Vernier calipers against Gauge Blocks that I found in my Univ Library.
Everything is pretty straight forward except for the "Danger Zone" in the sheet.
Any idea on how did they come up with that value. I am not going to question their tolerance limit because, I think one can change it depending on the instrument capability, and manufacturing tolerances. This danger zone value? I tried to look into AIAG 10%; 10-30% concept but, I don't think this is Gage R and R at all. This must be very simple. I just couldn't get it. Thanks for the help in advance.
Kind Regards,


Last edited:
Elsmar Forum Sponsor

Jen Kirley

Quality and Auditing Expert
Staff member
Good day Manasa, I am sorry for the delay in this response.

This is an interesting tool.

I tried it and found that I needed at least 1 out of 5 (20%) error readings to get the Standard Deviation to rise above 1. I also noticed that I reached a higher Std Dev if my errors were on both sides instead of always being 1.0005. The calculations are happening via formulas in columns S and T, but I am not mentally equipped enough to check the math. And the Std Dev calculations only work on Outside Jaw.

It's clear enough that the "Danger Zone" is trying to help the user identify what could be decided as excessive variation, but I don't see any clarity in how the thresholds were set, especially given the generous tolerance (+/- .002"), or why std. dev of 1 was decided on as a "zone" boundary.

I would like the chart except it requires a very significant error to register anything (1.01 vs. the expected 1.001).

Put it all together and I must say with regret that the tool does not look very useful because:
  1. It does not have instructions for use, and
  2. There is no clear way to set the desired std. dev, and
  3. Std. dev is only calculated for Outside Jaw, and
  4. The chart needs to have greater discrimination to be of any use in analysis.


Starting to get Involved
Hi Jen,

Thanks for the response. I really appreciate the effort you have put into it. Yes, this chart needs a much clearer explanation. After looking into it for a long time, I figured this has something to do with the Simple Linear Regression Analysis; Least Squares Method, and then coming up with the equation they have. I am still not successful in understanding the danger zone they came up with. But, I think at least I am close to the concept. I am not great at Statistics. So, if anybody can figure it out. I will be grateful. Thank you.
Last edited:


Forum Moderator
Staff member
The following is my speculation for the rationale behind this spreadsheet. The standard Vernier caliper is typically marked every 0.001 inches. The danger zone appears to be set at a level equivalent to +/- 2 standard deviations (95% confidence) around the regression line. With 5 measurements, any measurable deviation (0.001") will flag a warning.


Starting to get Involved
Thanks, Miner. It makes a little bit more sense now. This may be due to my lack of complete understanding of the concept. But, why is the Linest function used?, and how is the SD value equal to 10000 x Linest value. Research: This link has other procedures but this SD value changes for every instrument. It is very confusing. I think it's better to find a way that makes my data acceptable rather than understanding these forms. We are a very small company trying for ISO. We are trying not to go to the unnecessary depth and perform gage R and R for instruments like vernier. What can I do with my Required Values (Gage Block Values), my Measured Values, and My tolerance value to say my data is acceptable.I can make sure each measurement is within tolerance. But, in my view is not good enough. Any suggestions?? I thought I could make use of the SD Concept if I understood that. But, it makes very less sense at this point. May be looking into the whole regression analysis in excel will help! not sure!Thanks.


Forum Moderator
Staff member
My personal recommendation is to toss these forms, develop something simple that you understand and move on. You don't want to have an auditor question the form and then stumble around trying to explain it.

LINEST is just a regression function, so that is no mystery. I thought the 10000 X SD was just shifting the decimal point (like expressing a dimension in tenths vs. a decimal), but it should not change from form to form.

You might want to review some of my MSA blogs. There is also a template I created for Linearity, Stability and R&R studies.


Starting to get Involved
Good Morning Miner,
Thanks much for your suggestion. Yes, I will toss those forms because the multiplier is just wrong I think. As you said, the idea is to shift the decimal place I believe but, it keeps changing even if the tolerance doesn't. For measuring tape, with +/-0.05" tol, Danger Zone is SD>4. I simply don't get it. I have gone on to refer to your template: Gage R&R Template AIAG 4th Edition is an amazing tool. Great work. Do u think just performing Stability and Linearity will do for Inspection of Measuring Equipment when looked at it from an Auditor's perspective?. I personally don't think we need more than that for what we do. But, finding the right balance between not enough and too much is a bit hard. I am a newbie as you can pretty much tell. I will look into it and try to understand the whole stats. I do have a template that I created which is very basic and I think for what we do just something like a Danger zone fits great. But, how do I come with that value with backed calculations and concept is tricky? But, if t critical and t bias is the way to go, I will learn them. I will attach the template just for an idea as to how too basic it is and why I am struggling for improvement. My superiors think going overboard is a waste. But, not enough is also waste. I should decide on one that works. Any thoughts will help.
Thank you.


Last edited:


Forum Moderator
Staff member
What are your responsibilities? At first, I thought you were quality, but from the tone of your last question, it sounds like you may be specifically calibration.

Auditors will normally ask about the following aspects of measurement systems analysis:
  • Bias
  • Linearity
  • Stability
  • Repeatability & Reproducibility
Bias and Linearity (bias over a range) should be addressed by a good calibration system. Auditors look at this closely.

Stability should be handled by a risk assessment to determine whether it is necessary. Auditors rarely ask about this one, but there is always a chance. I have only run into two situations where I needed such a study. One was where thermal expansion on product with tight tolerances was involved. The other involved heat buildup in electronic components. In most cases, stability is a non-issue, but you need a formal risk assessment to justify skipping it.

Repeatability and Reproducibility are specific to the gage, the part and the feature measured, and are not typically performed by the calibration group, but by quality or manufacturing engineering. Auditors also like to focus on this area. This is a difficult area to try to skip, but you can set up gage families (covered in my blog) to minimize the effort involved.


Starting to get Involved
Thanks for the prompt reply Miner. My job is Quality, but since we are starting from scratch, and the company being small (14 employees), we want to optimize the extent we take this. We are a custom fab shop (tubing). The work our company does is amazing. Most importantly, the machines were built by my employer, their repeatability, consistency, is amazing. I am fairly new trying to develop documentation from the scratch in a way that best suits our scale of operation and get an ISO eventually because, we do supply aircraft parts, automotive parts. What is funny is ISO in a way is present in this company and that is why the parts are amazing and the customers audit themselves and opt for doing business with us.Unfortunately, I haven't been involved in any of those formal big company auditings.Rest of the customers do not need them and are generous with their tolerances.
However, typically we are within 5 thou (linear), +/-0.1 degrees for angular dim.I get what you said about bias, linearity, stability and R & R (AIAG).But, like I said trick is to find the right balance. The instruments generally are inspected against the master gauge blocks / pin gauges / angle blocks regularly. Any significant difference found, the instrument is replaced. For every fixed interval of my employer decided years, the Master gauges are calibrated. Will this suffice the ISO requirement? Because the standard itself gives the liberty of choosing the depth that suits our customer base.It gets hard when everything should be met keeping everything as simple as possible. Like you said bias and linearity are very imp to say that the device is good. Here I am using this analysis to say my ME is good not my parts. Stability can be found if altered by frequent periodic checking of Bias, linearity. My standard value is set so, I will find my bias using my measured value. I will take 5 measurements covering the entire range of the instrument satisfying the linearity requirement. If you had a look at my template, these 2 are addressed. But, the only thing I need is a summed up value of my Measured Values (SD/any other statistical entity) to say yes it is good or it is bad.Right now individual measurements are checked if they are within tol.
Gage R and R at this time is not being looked into because these are still tiny steps forward. But, definitely should be looked into.
Documentation with evidence is all that is in the way of getting the cerification.
Last edited:


Forum Moderator
Staff member
I tried to come up with something fairly simple. The attached file shows how to use a 1-sample t-test to a standard. If the sample measurements are statistically different from the standard, cell D13 will turn red. If not, it stays green. You can incorporate this into your existing spread sheets. Or you can use a control chart set to the standard as in the stability study. If it shows signs of out of control behavior, you have a problem.


Thread starter Similar threads Forum Replies Date
J Determining SPC tolerance Statistical Analysis Tools, Techniques and SPC 21
I Determining Calibration Tolerance of a Measurement Device General Measurement Device and Calibration Topics 2
H Determining Calibration Tolerance for new Leak and Occlusion Equipment Measurement Uncertainty (MU) 2
E Determining Weighing / Counting Scale Tolerance - No manufacturer specification General Measurement Device and Calibration Topics 24
S Determining Cpk on a single sided tolerance dimension Capability, Accuracy and Stability - Processes, Machines, etc. 3
S Determining sample sizes for PQ Qualification and Validation (including 21 CFR Part 11) 4
M Determining if an Insulin Pen Testing Machine is a Medical Device? EU Medical Device Regulations 4
D Determining the the maximum number of reprocessing cycles of attachments CE Marking (Conformité Européene) / CB Scheme 2
R Determining Uncertainty from Gage R&R Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 1
A IATF 16949 4.3.1 - Determining the scope of the quality management system - supplemental IATF 16949 - Automotive Quality Systems Standard 9
D ISO 9001:2015 4.3 Determining the Scope of the QMS ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
B Determining SAT Offsets vs TUS Offsets per SAE AMS 2750E AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 0
B Determining sample size for device sterility Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
D Determining of sample size for 'Operational Qualification' AQL - Acceptable Quality Level 5
R Question on determining defective units - I am not recording fixture to part rejected Statistical Analysis Tools, Techniques and SPC 5
S Clause 8.2.2 Determining the requirements for products and services ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
D Determining Calibration Frequency schedule for items used in production Manufacturing and Related Processes 2
C Determining if Maintenance Contractor is an External Service subject to ISO 9001 Clause 8.4 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 43
S AS9100D PEAR - Examples for organization's method for determining process results? AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 5
J ISO 17025 Documented Procedure for 6.2.5 - Determining competency ISO 17025 related Discussions 4
V Determining FDA 820 (registration) vs ISO 13485 - Supplier gives us the kit ISO 13485:2016 - Medical Device Quality Management Systems 1
J ISO 9001 8.4.1 - Determining controls applied to externally provided processes ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 8
E Determining what is good and what is bad can be subjective - when is it a quality issue? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
F Determining what type of scrap to include in my internal PPM calculation Quality Manager and Management Related Issues 5
M Determining number of employees within the "Scope" of the QMS ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
A Determining the Scope of the QMS during Stage 1? Registrars and Notified Bodies 11
W Minor Audit Nonconformance Against Determining the scope of QMS IATF 16949 - Automotive Quality Systems Standard 12
D Determining Critical Components for conformity with IEC 60601-1 IEC 60601 - Medical Electrical Equipment Safety Standards Series 21
Q ISO 9001, section 4.3 Determining the scope of our QMS ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
R Determining Sample Size for Medical Device Component Validation Inspection, Prints (Drawings), Testing, Sampling and Related Topics 0
A What does, Determining the requirements related to products and services,mean IATF 16949 - Automotive Quality Systems Standard 1
A Determining Retention Period for Medical Device QMS documents Document Control Systems, Procedures, Forms and Templates 5
S Surveillance Sampling Test - Determining Sample Size Inspection, Prints (Drawings), Testing, Sampling and Related Topics 5
F ISO 9001:2015 4.3 - Determining the scope of the quality management system ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 7
B Standards Needed In House - Determining what standards are applicable Various Other Specifications, Standards, and related Requirements 3
W Determining Medical Device Classification in Mexico Other Medical Device Regulations World-Wide 5
K Determining Effect of Failure without a DFMEA (Design FMEA) FMEA and Control Plans 1
W Determining the Status and Importance of the Processes and Areas to be Audited Internal Auditing 7
T Determining Customer Requirements for the US Postal Service (USPS) IATF 16949 - Automotive Quality Systems Standard 4
Crimpshrine13 Rules of achieving and maintaining IATF recognition - Determining audit days IATF 16949 - Automotive Quality Systems Standard 2
R Developing procedure for Determining Company's Context And Interested Parties ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 53
W Determining Asset Life or Depreciation Life for M&TE General Measurement Device and Calibration Topics 1
N Reason for determining no adverse effect on reworked product ISO 13485:2016 - Medical Device Quality Management Systems 8
N Procedure for determining pinhole position in condom Inspection, Prints (Drawings), Testing, Sampling and Related Topics 1
alonFAI Determining PCBA Xray Test Sampling Size Statistical Analysis Tools, Techniques and SPC 1
S Determining sample size for inspection to achieve x% confidence re defects Misc. Quality Assurance and Business Systems Related Topics 10
S Determining Sample Size - AQL & LQ AQL - Acceptable Quality Level 10
R Determining the feasibility/scope for FDA IQ/OQ against a very simple application US Food and Drug Administration (FDA) 2
B Determining the "storage volume" (ISO 8536-8:2004 clause 6.14) Other Medical Device Related Standards 4
N Procedure/calculation for determining the foot pressure of dial thickness gauge General Measurement Device and Calibration Topics 10

Similar threads

Top Bottom