Calibration Intervals (Frequency) derived from Variables Data

I have been assigned the task of performing calibration interval analysis on two of our company’s products to see if the calibration interval can be extended. A visit to the Metrology Lab provided a copy of RP-1 from which I selected method S2 as being the most appropriate. Preliminary data analysis indicates the current calibration time is at the 95% reliability target, which negates any attempt at extending the calibration interval without further intervention. Rather than take my management bad news, I prefer to take them solutions. I want to lay out a scenario of what needs to change in order to extend the calibration time and let them make the business decision as to whether to implement or not. To accomplish this I decided I need to further analyze individual "variable" failures to get a better visual representation comparing trend, test tolerances, and confidence limits. I can fit a curve through data, calculate standard deviations, and confidence limits, but now I want confidence bands around a curve corresponding to 95% confidence and this confidence band is a function of time.

Enter “Calibration Intervals from Variables Data”

Calibration Intervals from Variables Data.pdf

Equation (14) is not working. The model is second order. (14) is returning a one row 2 column “variance” matrix that I can’t seem to pull the elements out of . And if I could get the elements out I have drawn a blank as to how to incorporate these elements into an equation to plot the confidence boundary vs. time.

I am looking for the solution from (14) to a confidence vs. time function OR an alternative set of equations “that work and are understandable” to calculate and plot confidence bands as a function of time.


Thanks

Re: Calibration Intervals from Variables Data

Didn't mean to start my own blog here. After staring at "Uncertainty Growth Estimation in Uncertainty Analyzer" this morning. I got the brainstorm to compromise with MRbar/d2 scaled to enclose 95% of a normal distribution as a graphical solution that may work. I have enough data points. Follow up research this pm on Rbar/d2 revealed "Uncertinty Analysis and Parameter Tolerancing" "Figure 4" "(Deviations from Nominal)" confirms that this may be possible. Will give the math a try next week.

Thanks

Re: Calibration Intervals from Variables Data

Interesting topic. The post got in at the start of the weekend when it's pretty slow here. Hopefully someone will pick up on it by or on Monday.

Also, by all means - Let us know what you come up with as well. We appreciate the information!

Re: Calibration Intervals derived from Variables Data

Make sure that you test for significance on your equation. Your scatterplot looks an awful lot like a shotgun blast, which does not visually indicate much of a relationship. The equation that you developed may be heavily influenced by a few points.

Re: Calibration Intervals derived from Variables Data

Thank you for the paper. I do laud the effort to empirically develop methods to improve quality.

As your second post suggests, I assume you have found your answer to your question.

Are you at a point of validating the method? I was greatly intrigued to see the author suggest not using weights. Without sound theoretical justification for the variables chosen and no weights, I wonder if consistency problems will develop in replication.

I would be intrigued to learn from the author why he/ she chose not to use weights in the regression analysis. Plausibly, rational discussion would merit the use of the chosen parameters, but the data may suggest otherwise. Too, it might be interesting to explore some additional predictors of appropriate calibration interval.

Again, thank you for the article. I enjoyed reading it. Please repost with your results, and any correspondence with the author.

Re: Calibration Intervals derived from Variables Data

Miner said:

scatterplot looks an awful lot like a shotgun blast, which does not visually indicate much of a relationship.

Click to expand...

That's what I was thinking, but not being an expert in the field I figure(d) I'd keep my mouth shut.

Yes - It's interesting. I've never seen anyone go this far (precise?) in determining calibration intervals.

Re: Calibration Intervals derived from Variables Data

Miner: The equation is heavily influenced by a few points. Least squares best fit does this. Its the ouliers that throw the prediction off. Which opens up a little of the next question. How do I deal with outliers. Every article I read has a different perspective. To predict take them out, but leave them in because they are the most intersting cases? Better yet analyze it both ways (which I don't really want to do).

BradM: I have not found the answer to my question, BUT I have an Rbar/d2 variation to try next week. Does least squares best fit return an average? Why not use least difference best fit?

Your observation about weighted vs nonweighted brought a flashback when I read it. I think the author doesn't use weights because he hasn't validated this method yet I also detected some hesitancy about validity of first versus second order fits. Notice my curve is second order, it beat out first order by .001.

Validation! I've never done a calibration interval analysis before so you get the opportunity to watch me struggle up the learning curve. I will do a lot of research and technique selection, crunch 100's of thousands of data points, (there are 313 variables/calibration and 270 As Found calibrations to analyze for the first product) make a lot decisions on what gives the most correct answer (make my best guess). Validation will come with time. As I gain experience the analysis method can and probably will be tweeked. Remember "my" analysis is a combination of RP-1 S2 and Intervals from Variables data. S2 is over glorified validated standard practice, the variable data intervals technique I suspect is not validated, but its theory sounds really good.

Marc: I don't see a shotgun blast, I see a normal distribution following the plotted curve of the mean. The shotgun blast x axis is "weeks", the y axis is dBm. The trend is -0.5 dBm drift downward in 100 weeks. The test tolerance is +-1dBm. +-1.96 standard deviations is equivalent to the Reliability Target of 95% (95% of As found product will be IN-TOLERANCE) and where it crosses the lower variable test tolerance is the estimate of calibration cycle time. I can eyeball this crossing at 80 weeks, the current calibration cycle is at 52 weeks.

I still want a mathematical technique to get the 1.96 standard deviation curve plotted around the mean, so I can get closer than eyeballing 80 weeks.

I get to do this with 313 variables/calibration. The Pass Fail of a S2 calibration is determined by the variable that goes OUT OF TOLERANCE first. The shot gun plot is one of the Variables that caused a S2 Fail.

I suspect I will find a 1:1 correlation between this variable and TCXO drift. ie the drift in the frequency standard is causing frequency shifts in all the mixers, filters, and IF bandwidths in our product and this is causing the output amplitude to roll off at -.5 dBm / 100 weeks. I will get around to checking for this correlation eventually.

So Marc, the data point in the lower right hand corner caused an entire S2 calibration to fail. Is it a valid data point or an outlier? Do I have a real S2 calibration Fail or can I change it to Pass?

My gut says its a valid data point and this calibration is a confirmed Fail.

Re: Calibration Intervals derived from Variables Data

Thank you!! What an excellent follow-up to our posts. This is such an interesting topic (at least to me). I'll "mull" over your post, and give you some more of my thoughts. Maybe since we're in the week now some others can chime in.

Please do keep us informed of your testing process.

Re: Calibration Intervals derived from Variables Data

Marc said:

Yes - It's interesting. I've never seen anyone go this far (precise?) in determining calibration intervals.

Click to expand...

Actually, the U.S. Navy at their Engineering Center in Corona has a department that has done only that type of work for 20+ years.....they have a staggering amount of data.....and I believe the Navy also has had a lot of input to RP-1.....

I couldn't download the information tonight, so I couldn't tell if it is taking more of a Monte Carlo approach (which is making a big come back in Metrology it appears), but is appears to have some similarities, based only on the write-up.

Hershal

Re: Calibration Intervals derived from Variables Data

Just out of curiosity, in what circumstances is such precision in determining calibration intervals necessary?