Validation of Predictive Models of Yield

Q

quick_silver

#1
hi,

The company hired the help of an outsider to come up with a predictive model for the yield in the manufacturing plant where i was assigned. Now we are using these models, but then i noticed that there were times that the actual is very far from the predicted values. My question is that how will i measure the effectiveness of the model using the new data set.
According to some of the articles i have read regarding this, they were talking about arriving the r square of the new data set. Is calculating the r square enough to determine the model validity as of this moment.
Now I'm asking for a guidance on how to really measure the model validity for the new data set.

Thanks in advance!
 
Elsmar Forum Sponsor

Bev D

Heretical Statistician
Leader
Super Moderator
#2
Re: validation of predictive models

quicksilver - this is a great question. as George Box once said: all models are wrong; some are useful.

it would be helpful is you could add a few more specifics. for example, is the model addressing the total yield of the factory? or is there a specific model for each product or manufacturing line? or is it the yield of a single processing step? or individual characteristic?

is the model based only on historical values of the yield? or is it based on a combination of input factors and their influence on the yield?

it's also important for us to understand how the model is intended to be used. is a mmodel being used to determine what input settings will be used to guarantee a certain yield? or is it being used to understand when something in the process has changed and to direct you to take some type of corrective action?
 
Last edited:
Q

quick_silver

#3
Re: validation of predictive models

Thanks Ma'am..

1) we have six models for each product type which undergoes unique processing
2) the model was determined by using 2 yrs historical data, we have about 50 factors monitored that we believe have impact on the different yield. Now the 3rd party provider they run a correlation analysis and then from that they arrived the regression models. Actually we did a lot of iterations, because upon seeing the 1st model there are certain factors that the models says will have positive impact but violates human logic (it should be negative not positive). And the most acceptable model is the one that we are currently using
3) the importance of the model for us, is first from among the 50 factors we need the model to tell us which among the 50 have really the biggest impact to the yield and so we can focus more our action plans on how to control those factors.
 
D

Darius

#4
Re: validation of predictive models

they run a correlation analysis and then from that they arrived the regression models. Actually we did a lot of iterations, because upon seeing the 1st model there are certain factors that the models says will have positive impact but violates human logic (it should be negative not positive). And the most acceptable model is the one that we are currently using
There are some asumptions that have to be made when using regression, if the asumptions are not met can lead to weir conclusions. One of the most common mistake is to take the model as a lineal ones when the data isn't. If you think that the factor defies the logic; it could be noise/non significant for the model and almost sure that the factor is a value near 0.

If you are woried about the effect of the factor is not taken in account the way it should, you can make a chart of the factor against the difference between the real and the forecasted value. If some effect was not taken in account you will see some trend (a curve), showing that the efect was not taken the way it should. IMHO graphical methods show things that the numbers can't, you may have a non lineal trend but an r squared low because the lack of lineality.

from among the 50 factors we need the model to tell us which among the 50 have really the biggest impact to the yield and so we can focus more our action plans on how to control those factors.
As you may know the bigger the absolute of the factor the more impact on yield will have, but the relation will change if the variables that affect such variation are controled, you can have a strong relationship between a variable and another but if the variable is controled the impact will be negligible, it's like seeing a line from a micron, you will only see the noise (in the case of a line on the center, only black).

You ask about the use of R squared to determine wich model is right, if you are comparing a model result against the other IMHO is better to use the methods used in forecasting to determine wich gives a better results (RMSE, MAPE,MdAPE, GMRAE, MdRAE). IMHO Median Relative Absolute Error (MdRAE) is well protected against outliers.
 

Bev D

Heretical Statistician
Leader
Super Moderator
#5
well, 50 input factors is a LOT of factors. You said that they used 2 years of historical data? How many independent runs or lots does this represent? it is possible that the model is overfitted. This can give a decent r square value but will result in a model with very little actual predictive power in practice.

the only true way to validate the model is overtime: plot the predicted value vs the actual value for some time, say 20-30 runs (or lots)...
 
Q

quick_silver

#6
Re: validation of predictive models

Thanks Sir Darius. I did calculate the tracking signal of the six models. Accdg to my readings the ideal should be -1 to 5 but unfortunately i got many outliers. That's why i was looking of some other ways of checking the validity. How can i attached the excel file here?
 

Miner

Forum Moderator
Leader
Admin
#8
There is a definite flaw in the predictive model.

If you follow BevD's suggestion to plot predicted value (Y) versus actual value (X) as well as Actual - Forecast (Y) versus actual value (X), you will see the anomaly clearly.

Ideally, predicted value plotted by actual value should be a line running at 45 degrees with scatter on either side, while Actual - Forecast versus actual value should be a horizontal line at zero with scatter above and below. This data show the reverse, which means there is a very fundamental problem with the equation. That is, its not predicting at all.
 
Q

quick_silver

#9
Thanks Miner.

So moving forward I should recommend that the predicted model should be revisited and create a new one.
My question would then be how can I measure if the model that will be created is a useful one without waiting for new data, is there a way? Because when we accepted the model, the acceptability criteria was on the model or iteration no. with highest r adj square and the model whose equation agrees with conventional wisdom.
we have already paid the company who did the modeling for us. I just realized that the payment should have been after the availability of the new data and that the new data proved that the predicted equation is really useful. It turns out it is a waste money after all. :(

Thanks in advance!
 
Last edited by a moderator:

Steve Prevette

Deming Disciple
Leader
Super Moderator
#10
For monitoring whatever predictive model is used, I'd suggest running a control chart of the delta between the prediction and the actual. Then see if that delta is stable and predictable, and if so, evaluate if the error is acceptable or not. If there are trends, then there probably is something that is affecting actual production that the model is not taking into effect.
 
Thread starter Similar threads Forum Replies Date
A Has anyone implemented the Adobe Acrobat Sign Validation Pack to be 21 CFR Part 11 Compliant? ISO 13485:2016 - Medical Device Quality Management Systems 1
C Test Method Validation - ISO Standards Qualification and Validation (including 21 CFR Part 11) 1
J API Q1 - 5.7.1.5 - Validation of Processes for Production and Servicing Oil and Gas Industry Standards and Regulations 4
A Validation Plastic Injection Molding Process protocol ISO 13485:2016 - Medical Device Quality Management Systems 5
M Use of statistical techniques for Process Validation ISO 13485:2016 - Medical Device Quality Management Systems 9
M Risk-based approach to Test Method Validation for Design Verification? US Medical Device Regulations 4
K Design: Verification Vs Validation And Validation Vs Transfer ISO 13485:2016 - Medical Device Quality Management Systems 19
C Medical Device Gamma Irradiation Validation per VDmax25 (ISO 11137) Qualification and Validation (including 21 CFR Part 11) 1
A Human Factor Validation Human Factors and Ergonomics in Engineering 4
B Verification and Validation of calculations (FEA) in OCTG components. Oil and Gas Industry Standards and Regulations 9
B Validation of FEA Analyses in Oil&Gas Industries. There are a lot of guidelines for other activities. There is a similar proposal for O&G? Design and Development of Products and Processes 0
K Analytical Method Qualification Vs Validation expectations ISO 13485:2016 - Medical Device Quality Management Systems 1
C. Tejeda Process validation of rework assembly methods (medical devices) Medical Device and FDA Regulations and Standards News 3
B Validation of design for valve api 6d 25 edition ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 0
P Validation for RUO (Research Use Only) ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
B Hi , everyone i need a procedure for validation of design prototype api 6d (valve manufacturing) Oil and Gas Industry Standards and Regulations 1
Ed Panek Validation of Signature Software (Off the shelf) US Medical Device Regulations 4
Q Combining Validation Protocol & Report into 1 template Document Control Systems, Procedures, Forms and Templates 4
K ISO 17025 Method Validation and Verification for Test Lab ISO 17025 related Discussions 4
S Environment Monitoring System Validation ISO 14001:2015 Specific Discussions 1
B Supplier Evaluation report - Validation required or not ISO 13485:2016 - Medical Device Quality Management Systems 3
M Cleaning Validation of components Manufacturing and Related Processes 2
P Validation Methods of Machine learning and Artificial intelligence Pharmaceuticals (21 CFR Part 210, 21 CFR Part 211 and related Regulations) 10
M How to respond to 483 validation finding we disagree with? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 33
I Cryogenic Container Closure Integrity for HCT/P, Validation Method US Food and Drug Administration (FDA) 0
G Mfr. Process Validation BEFORE Design Transfer? Other Medical Device and Orthopedic Related Topics 1
G Injection Molded Parts in Verification & Validation Other Medical Device and Orthopedic Related Topics 3
B Spreadsheet - Used for complaint investigation - Validation required or not ISO 13485:2016 - Medical Device Quality Management Systems 9
T Vacuum Heat Treatment Validation Manufacturing and Related Processes 1
M Root Cause and Corrective Action for CAPA's lacking validation/verification ISO 13485:2016 - Medical Device Quality Management Systems 19
M Software Validation SAP B1 for ERP ISO 13485:2016 - Medical Device Quality Management Systems 2
V Retrospective validation medical devices Qualification and Validation (including 21 CFR Part 11) 7
P Software validation for FPGA Software Quality Assurance 1
I Are suppliers required to hand over process validation reports? ISO 13485:2016 - Medical Device Quality Management Systems 20
N Computerized System Validation ISO 13485:2016 - Medical Device Quality Management Systems 12
M 3D Scanner Software validation ISO 13485:2016 - Medical Device Quality Management Systems 7
E Cybersecurity for Internal Tool Validation Medical Device and FDA Regulations and Standards News 1
B Transport Validation For Non-sterile Medical Devices ISO 13485:2016 - Medical Device Quality Management Systems 4
D Software Validation Question ISO 13485:2016 - Medical Device Quality Management Systems 10
G Pad Printing Validation OR Verification ISO 13485:2016 - Medical Device Quality Management Systems 4
A ETHYLENE OXIDE STERILIZATION VALIDATION Manufacturing and Related Processes 4
C. Tejeda Computer system validation approach for Minitab Statistical software Software Quality Assurance 11
D 8.5.1.2 Validation and control of special processes requirements for Heat Treat External Processor AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 4
S Performance Qualification and Process Validation ISO 13485:2016 - Medical Device Quality Management Systems 5
L ISO 11607-1 Packaging system validation Design and Development of Products and Processes 9
John C. Abnet ...validation of computer software ISO 13485:2016 - Medical Device Quality Management Systems 17
D Machine rebuilds versus process re-validation IATF 16949 - Automotive Quality Systems Standard 1
R Cloud-based SaMD Validation IEC 62304 - Medical Device Software Life Cycle Processes 8
G Process Validation Before/After Sterilization? Design and Development of Products and Processes 3
D Laboratory Refrigerator Validation ISO 13485:2016 - Medical Device Quality Management Systems 2

Similar threads

Top Bottom