Validation of Predictive Models of Yield

Q

quick_silver

#1
hi,

The company hired the help of an outsider to come up with a predictive model for the yield in the manufacturing plant where i was assigned. Now we are using these models, but then i noticed that there were times that the actual is very far from the predicted values. My question is that how will i measure the effectiveness of the model using the new data set.
According to some of the articles i have read regarding this, they were talking about arriving the r square of the new data set. Is calculating the r square enough to determine the model validity as of this moment.
Now I'm asking for a guidance on how to really measure the model validity for the new data set.

Thanks in advance!
 
Elsmar Forum Sponsor

Bev D

Heretical Statistician
Staff member
Super Moderator
#2
Re: validation of predictive models

quicksilver - this is a great question. as George Box once said: all models are wrong; some are useful.

it would be helpful is you could add a few more specifics. for example, is the model addressing the total yield of the factory? or is there a specific model for each product or manufacturing line? or is it the yield of a single processing step? or individual characteristic?

is the model based only on historical values of the yield? or is it based on a combination of input factors and their influence on the yield?

it's also important for us to understand how the model is intended to be used. is a mmodel being used to determine what input settings will be used to guarantee a certain yield? or is it being used to understand when something in the process has changed and to direct you to take some type of corrective action?
 
Last edited:
Q

quick_silver

#3
Re: validation of predictive models

Thanks Ma'am..

1) we have six models for each product type which undergoes unique processing
2) the model was determined by using 2 yrs historical data, we have about 50 factors monitored that we believe have impact on the different yield. Now the 3rd party provider they run a correlation analysis and then from that they arrived the regression models. Actually we did a lot of iterations, because upon seeing the 1st model there are certain factors that the models says will have positive impact but violates human logic (it should be negative not positive). And the most acceptable model is the one that we are currently using
3) the importance of the model for us, is first from among the 50 factors we need the model to tell us which among the 50 have really the biggest impact to the yield and so we can focus more our action plans on how to control those factors.
 

Darius

Quite Involved in Discussions
#4
Re: validation of predictive models

they run a correlation analysis and then from that they arrived the regression models. Actually we did a lot of iterations, because upon seeing the 1st model there are certain factors that the models says will have positive impact but violates human logic (it should be negative not positive). And the most acceptable model is the one that we are currently using
There are some asumptions that have to be made when using regression, if the asumptions are not met can lead to weir conclusions. One of the most common mistake is to take the model as a lineal ones when the data isn't. If you think that the factor defies the logic; it could be noise/non significant for the model and almost sure that the factor is a value near 0.

If you are woried about the effect of the factor is not taken in account the way it should, you can make a chart of the factor against the difference between the real and the forecasted value. If some effect was not taken in account you will see some trend (a curve), showing that the efect was not taken the way it should. IMHO graphical methods show things that the numbers can't, you may have a non lineal trend but an r squared low because the lack of lineality.

from among the 50 factors we need the model to tell us which among the 50 have really the biggest impact to the yield and so we can focus more our action plans on how to control those factors.
As you may know the bigger the absolute of the factor the more impact on yield will have, but the relation will change if the variables that affect such variation are controled, you can have a strong relationship between a variable and another but if the variable is controled the impact will be negligible, it's like seeing a line from a micron, you will only see the noise (in the case of a line on the center, only black).

You ask about the use of R squared to determine wich model is right, if you are comparing a model result against the other IMHO is better to use the methods used in forecasting to determine wich gives a better results (RMSE, MAPE,MdAPE, GMRAE, MdRAE). IMHO Median Relative Absolute Error (MdRAE) is well protected against outliers.
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#5
well, 50 input factors is a LOT of factors. You said that they used 2 years of historical data? How many independent runs or lots does this represent? it is possible that the model is overfitted. This can give a decent r square value but will result in a model with very little actual predictive power in practice.

the only true way to validate the model is overtime: plot the predicted value vs the actual value for some time, say 20-30 runs (or lots)...
 
Q

quick_silver

#6
Re: validation of predictive models

Thanks Sir Darius. I did calculate the tracking signal of the six models. Accdg to my readings the ideal should be -1 to 5 but unfortunately i got many outliers. That's why i was looking of some other ways of checking the validity. How can i attached the excel file here?
 

Miner

Forum Moderator
Staff member
Admin
#8
There is a definite flaw in the predictive model.

If you follow BevD's suggestion to plot predicted value (Y) versus actual value (X) as well as Actual - Forecast (Y) versus actual value (X), you will see the anomaly clearly.

Ideally, predicted value plotted by actual value should be a line running at 45 degrees with scatter on either side, while Actual - Forecast versus actual value should be a horizontal line at zero with scatter above and below. This data show the reverse, which means there is a very fundamental problem with the equation. That is, its not predicting at all.
 
Q

quick_silver

#9
Thanks Miner.

So moving forward I should recommend that the predicted model should be revisited and create a new one.
My question would then be how can I measure if the model that will be created is a useful one without waiting for new data, is there a way? Because when we accepted the model, the acceptability criteria was on the model or iteration no. with highest r adj square and the model whose equation agrees with conventional wisdom.
we have already paid the company who did the modeling for us. I just realized that the payment should have been after the availability of the new data and that the new data proved that the predicted equation is really useful. It turns out it is a waste money after all. :(

Thanks in advance!
 
Last edited by a moderator:

Steve Prevette

Deming Disciple
Staff member
Super Moderator
#10
For monitoring whatever predictive model is used, I'd suggest running a control chart of the delta between the prediction and the actual. Then see if that delta is stable and predictable, and if so, evaluate if the error is acceptable or not. If there are trends, then there probably is something that is affecting actual production that the model is not taking into effect.
 
Thread starter Similar threads Forum Replies Date
S Rationalising the level of effort and depth of software validation based on risk ISO 13485:2016 - Medical Device Quality Management Systems 10
D Questions regarding process validation ISO 13485:2016 - Medical Device Quality Management Systems 6
Y We found out we have been using a equipment without validation for past 4 years Quality Manager and Management Related Issues 6
Z Is IQ necessary for laser marking validation? EU Medical Device Regulations 3
E 13485:2016, Sections 4.1.6, 7.5.6 and 7.6 - Validation of Software - Need some Advice please ISO 13485:2016 - Medical Device Quality Management Systems 2
A Validation of Forced Aeration Process ISO 13485:2016 - Medical Device Quality Management Systems 3
E Mentor for Test Method Validation (TMV) Design and Development of Products and Processes 2
M API 4F/7K/8C Design Package Validation Oil and Gas Industry Standards and Regulations 2
I ISO 2233:2000 Question - Medical Device Shipping/Transportation Validation Other ISO and International Standards and European Regulations 1
T Annual Validation as a detection mode on a PFMEA? FMEA and Control Plans 5
B TMV - Selection of TM's for Validation ISO 13485:2016 - Medical Device Quality Management Systems 5
S Forced ServiceNow validation - No change in our current user and functional requirements IT (Information Technology) Service Management 6
P Human Factors / Usability validation in the time of COVID Human Factors and Ergonomics in Engineering 9
C Template for Excel Validation Reliability Analysis - Predictions, Testing and Standards 5
M IT validation for a paper based MD repair company QMS ISO 13485:2016 - Medical Device Quality Management Systems 6
P Unrealistic Packaging Validation Sample Size 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 13
D Test summary report example for design validation wanted - ISO 13485 ISO 13485:2016 - Medical Device Quality Management Systems 1
M Is Validation of Plating Processes required and who is responsible? Qualification and Validation (including 21 CFR Part 11) 11
T ISO 13485 - Process validation at critical suppliers ISO 13485:2016 - Medical Device Quality Management Systems 7
K Software Validation for Measurement Tools used in Process Validation ISO 13485:2016 - Medical Device Quality Management Systems 2
Stoic Manual soldering processes - 100% verifiable, or always requiring validation? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 15
P Design verification driven by new equipment. How is this different than process validation? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 1
S Rees System Validation Qualification and Validation (including 21 CFR Part 11) 1
K PQ validation qualification - Asked to write a PQ protocol ISO 13485:2016 - Medical Device Quality Management Systems 6
Stoic Are any medical device companies using the 2011 FDA process validation guidance instead of GHTF/SG3/N99-10:2004? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 1
K Old medical devices -> 7.3.7. Design and development validation ISO 13485:2016 - Medical Device Quality Management Systems 1
S SOP for ISO 13485:2016 Quality related Software validation ISO 13485:2016 - Medical Device Quality Management Systems 9
Y Retrospective Validation - Class I device 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 10
S High voltage testing - ISO 17025 - 7.2.2 Validation of methods and 7.3 Sampling ISO 17025 related Discussions 3
M Production approval testing - Alternative ideas for Validation Reliability Analysis - Predictions, Testing and Standards 4
M Validation of two nearly identical products Other Medical Device Regulations World-Wide 5
J Requested Validation plan and reports Manufacturing and Related Processes 4
S Validation Records - Very young QMS Qualification and Validation (including 21 CFR Part 11) 2
M Test method validation - Is MSA (MSA1, MSA2, MSA3 and linearity) a good solution? Medical Device and FDA Regulations and Standards News 1
G Devices from IQ, OQ or PQ process to be used for verification, validation and summative? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 5
K ERP System Software Validation - ISO13485 2016 4.1.6 Design and Development of Products and Processes 8
W ASTM F1929 dye penetration test - Validation for in-house testing ISO 13485:2016 - Medical Device Quality Management Systems 13
Bev D Verification and Validation of Measurement Systems Misc. Quality Assurance and Business Systems Related Topics 0
Y Does Solidworks (2D/3D drafting modules) need validation? Other Medical Device and Orthopedic Related Topics 5
D Software validation in Medical Equipment Other Medical Device and Orthopedic Related Topics 20
K Validation of new machine (second machine of the same type) Pharmaceuticals (21 CFR Part 210, 21 CFR Part 211 and related Regulations) 2
A Literature review/HACCP validation of metal detection Food Safety - ISO 22000, HACCP (21 CFR 120) 0
Y Packaging validation for non-sterile Medical Equipment Other Medical Device Related Standards 1
A Our auditor told if we didn't have a patent we would have to do a validation or verification ISO 13485:2016 - Medical Device Quality Management Systems 6
N Design Verification & Process Validation - Statistical sample sizes Design and Development of Products and Processes 2
E Equipment Qualification - IQ/OQ per ISO 13485:2016 section 7.5.6 Process validation ISO 13485:2016 - Medical Device Quality Management Systems 7
Q Clean Line Validation ISO 13485:2016 - Medical Device Quality Management Systems 6
D 510K and Changes to Verification and Validation US Food and Drug Administration (FDA) 2
H EO Sterilization Validation - Sterility Testing and Load Configuration Other Medical Device Related Standards 1
C Looking for simple Software Validation IQ templates. Qualification and Validation (including 21 CFR Part 11) 4

Similar threads

Top Bottom