Argument for R&D Measuring Instruments to be included in the Calibration Program

[Graeme]

Does it matter if the measurement is wrong?

In my opinion ...

If the instrument is used to make a measurement that something or someone else relies on, it should be calibrated.

Let's say A performs an experiment and piblishes a report. B reads the report and wants to replicate the experiment, but gets results that are quite different. What are the possibilites?

• If B used calibrated instruments and A did not, then if B publishes and says A is mistaken it will cause greate embarassment (at least) to A. Even worse if B lists the instrument uncertainty and shows how much difference is outside that.
• If A used calibrated instruments and B did not, then if B publishes and says A is mistaken it will be B who gets embarassed.
• If neither A or B used calibrated instruments, nobody knows who is right or wrong. Then when C publishes yet another set of results, determined using calibrated instruments, then both A and B will be embarassed and half the people reading the reports will be confused.
• The best case is that A and B both use calibrated instruments. Any difference in their results can be evaluated, and any difference that is greater than the combined measurement uncertainty of the instruments can be targeted for further investigation.

Research is the basis for future development. If the data from research do not have the property of measurement traceability (which includes quantification of the uncertainty) then they are meaningless. Traceability comes from correctly using calibrated instruments. Without that, the future of the development project does not look too good.

Development is to turn research into marketable products. Products have attributes that must be defined and measured so that they can be manufactured repeatably. If development measurements are made using uncalibrated instruments, how can the design specifications be correct? If those are not correct then the manufacturing tooling and gaging will not be correct, which means the product will not be correct -- and the company has just wasted a whole lot of money.

From time to time I have heard people try to justify using uncalibrated instruments to set up new systems by saying that the final test is done with calibrated instruments so that will catch anything. Hmmm. That means you are catching a problem at the most expensive time, just before turnover to the customer, when the customer is probably watching the certification test, and now you have to rework the entire setup using calibrated tools anyway, and rework is a cost that can't be charged to the customer!

Anyway, I believe R&D tools & equipment are no different from any other. To quote a friend who is no longer here,

• Does it matter if the result of the measurement is wrong?
  • If it does matter, then calibrate the instrument.
  • If it doesn't matter, then why are you (wasting your time by) making the measurement?

(I added the bit in parentheses.)

Also, clause 7.6 references ISO 10012:2003 which contains further requirements as well as guidance on measurements. While you could decide to exclude areas from the measurement management sytem, you have to determine the associated risks and decide what to do about them. From the beginning of section 4, General requirements:

The organization shall specify the measurement processes and the measuring equipment that are subject to the provisions of this International Standard. When deciding the scope and extent of the measurement management system, the risks and consequences of failure to comply with metrological requirements shall be taken into account.​

The measurement management system consists of the control of designated measurement processes and metrological confirmation of measuring equipment (see Figure 2), and the necessary supporting processes.

The measurement processes within the measurement management system shall be controlled (see 7.2). All measuring equipment within the measurement management system shall be confirmed (see 7.1).

(BTW, this version replaces the old 10012-1 and 10012-2 standards.)

(Lots of people have said that this standard can be ignored because "it's just guidance". But how many "shall" items are there in those three paragraphs?)




​

​

 

[Icy Mountain]

He beat me!

I was going to quote Graeme's tag line. IMO, if you are going to measure anything during Design Verification or Design Validation, you ought to be using calibrated devices. ISO or TS aside, it's just plain common sense for R&D to know that their measurements bear a passing resemblance to reality.

I will grant one exception: I have a Senior Design Engineer that has brought me 3 different meters in the past 7 years. He dropped each on my desk and said: "I don't care what the sticker says, this meter is out of calibration". He was correct, all 3 times. In the same 7 years, not one other piece of equipment on his bench has been out of tolerance when it is recalibrated.

I would exempt his equipment but it would set a bad example for the other engineers that believe everything the meter tells them.

 

[MikeL]

Design Verification... definetly

Don't have an argument over devices used to qualify a product,

When we first started talking about R&D I was thinking of where concepts for products are developed. like taking a competitors car for a spin.

Any devices used for verification and validation need to be calibrated.

 

[David Hartman]

Don't have an argument over devices used to qualify a product,

When we first started talking about R&D I was thinking of where concepts for products are developed. like taking a competitors car for a spin.

Any devices used for verification and validation need to be calibrated.

Not to coninue beating a dead horse, but even in the situation you describe, wouldn't you want to know to what degree a competitor's car betters yours in any specific category? Look at Road & Track, Motor Trend, Car and Driver, Automobile Magazine, or any of the rest that road test vehicles regularly, would you even pay attention to the results of these road tests if you knew that the calibration of the test equipment may have changed from one vehicle to another (what value could you get from such a road test)?

How can you understand how much more sound insulation you need to add to your cars, if you don't know precisely how much quieter the competitions vehicle is to yours?

How about your competitions engines on a dyno, then comparing yours to it, should you even worry about whether the dyno is setup and calibrated the same way or not?

When performing comparisons, if the equipment is not properly calibrated, it's just subjective analysis.

 

[jrcook5]

I thought R&D measuring devices (Calipers, micrometers, etc.) should always be part of the controlled calibration system.
Dave

I have read and re-read this thread and even waited a day to respond due to the emotions this topic evokes. I realize that this is an informal discussion of relative issues and not to be taken as the final rulings of a governing body however, I think a lot of people including myself, read these discussions because of the years of experience and expertise of those replying to the topics and have come to rely upon the answers given as a valuable resource. Having said this, I must now convey my sense of frustration regarding the lack of agreement among the experts as it relates to the controlled calibration of not just R&D but all measuring devices. Image the level of despair experienced by a novice, to the world of calibration, trying to conform to the requirements when there isn't a consensus on what the requirements are.

Back to Dave's original statement "...I thought R&D measuring devices...should always be part of the controlled calibration system." To respond to the statement I think we must first answer what standard or requirement are we complying with and what exactly does the requirement mean? Secondly, define the differences between calibration and "controlled calibration". Are there times when performing a calibration is adequate without entering the device into a formal "control system"? Thirdly, we don't always know the best way, for other departments, to achieve the desired result.

I think most standards clearly state that only the devices that are used to demonstrate product conformance, to stated specifications, need to controlled, and therefore require periodic calibration. If this is the rule than it becomes very straightforward what devices require calibration. I am in strong disagreement with those that characterize this as meeting just the minimum or focusing on the least I have to do. I feel these statements demean the effort necessary to properly implement and maintain an effective measuring device control system. I am very confident that the company that took their final product specifications and insured that all the instruments used to perform these measurements were capable of the necessary accuracy and controlled them properly, would not be considered to be lagging anyone in this aspect of their performance.

A quality system that meets the requirements of a standard or regulation is not doing the minimum. I'm sure they could be doing a lot less by just implementing a system on paper that appears to meet the standard but continue to covertly do what has been good enough in the past. Complying with the standard or the requirements is building a foundation on which improvement can be made in the future. There is no shame in implementing a system which meets the standard and in the future making adjustments based on the risk/reward of the improvement.

To summarize, let's agree on what the standards require relative to measuring devices. Complying to the requirements is not a small accomplishment and finally 4 more months before we go fishing again.

 

[Hennie]

Why the debate about ISO requirements? Surely any measuring equipment that can affect product quality must be calibrated. Especially during R&D stage. After all, once a product is released for series production the info passed onto production and the supporting processes must be based on accurate info with regards to all aspects of product requirements. I will certainly not accept standards/specifications specified by R&D based on results achieved on measuring equipment not calibrated.
Secondly, what about MSA studies. I regard this as important. Surely you want correlation between R&D measuring equipment and shopfloor equipment.

 

[Jim Wynne]

I just went back and read through
this thread which you started, and I'll repeat here what I said there, with particular emphasis on the last sentence:

When considering the level of control you need, and calibration requirements, you have to consider not just erroneous measurements made in pass/fail production situations, but also in terms of protecting the customer from anything that might reasonably result in delays or disruptions. The goal should be to provide a steady flow of the processes required to meet the customer's schedule, which your company has presumably agreed to.

If this means that erroneous measurements made by development engineers in the pre-production stages could result in delays, then the devices should be calibrated and controlled. If the situation for a particular device is ambiguous, err on the side of customer protection.

Designing a gage control system isn't all cut-and-dried, as you've found out. If you depend solely on the standards for guidance in what you must do, you'll leave out things you should do.

I'll add this: the goal in building a quality system should not be meeting the standard, or achieving registration. The goal should be to develop a way of doing business that gives you the best chance of making money and satisfying customer requirements. We all have to work within the limits of our own employment situations, and that means that sometimes we have to cut corners or do something less than what we would do if we ruled the world. You have to do the best you can with what you've got, and if you're in a position that prohibits you from doing the maximum, it doesn't mean that what you can do is the only the minimum. We're all somewhere in between, so don't feel rained on.

 

[jrcook5]

Surely any measuring equipment that can affect product quality must be calibrated.

So our criteria is "any measuring equipment that can affect product quality"?
Please define "affect quality please".

 

[Helmut Jilling]

So our criteria is "any measuring equipment that can affect product quality"?
Please define "affect quality please".

I think this thread is starting to get a little strained. You may not like the positions we have stated, but JSW said it well, a couple posts back. Don't do this just for the standard, do it for your profit and your customers' benefit.

Please define "affect quality please". This is obvious. If a gage does not produce an accurate measurement, if that will in way any way affect negative results in the product you make, that "affects quality." Don't make it too difficult. I thin you understand the term, you just disagree with those who are applying it broadly.

To really meet the standard, you have to understand and meet the objective of the requirmeents, not just the technical letter of the law.

As JSW said, do this for you, and the customer, not ISO.

 

[jrcook5]

I think this thread is starting to get a little strained. You may not like the positions we have stated, but JSW said it well, a couple posts back. Don't do this just for the standard, do it for your profit and your customers' benefit.

Please define "affect quality please". This is obvious. If a gage does not produce an accurate measurement, if that will in way any way affect negative results in the product you make, that "affects quality." Don't make it too difficult. I thin you understand the term, you just disagree with those who are applying it broadly.

To really meet the standard, you have to understand and meet the objective of the requirmeents, not just the technical letter of the law.

I agree that we must implement policies that will benefit the company however we must also operate within the constraints of the particular governing standard or regulation and we must cleary define the must do's in the requirements. The statement, we must calibrate devices which may affect quality, begs the question, how do you define quality? Do you accepte Crosby's premise that quality is "conformance to the requirements"? I don't wish to be argumentative. I just would like a little more clarity regarding the issue of calibration. I certainly am not trying to make it difficult, quite the opposite, my hope is that calibration and the relative issues would be easily understood by all, those with years of experience and newcomers alike.