SBS - The best value in QMS software

Validation of Predictive Models of Yield

Q

quick_silver

#1
hi,

The company hired the help of an outsider to come up with a predictive model for the yield in the manufacturing plant where i was assigned. Now we are using these models, but then i noticed that there were times that the actual is very far from the predicted values. My question is that how will i measure the effectiveness of the model using the new data set.
According to some of the articles i have read regarding this, they were talking about arriving the r square of the new data set. Is calculating the r square enough to determine the model validity as of this moment.
Now I'm asking for a guidance on how to really measure the model validity for the new data set.

Thanks in advance!
 
Elsmar Forum Sponsor

Bev D

Heretical Statistician
Staff member
Super Moderator
#2
Re: validation of predictive models

quicksilver - this is a great question. as George Box once said: all models are wrong; some are useful.

it would be helpful is you could add a few more specifics. for example, is the model addressing the total yield of the factory? or is there a specific model for each product or manufacturing line? or is it the yield of a single processing step? or individual characteristic?

is the model based only on historical values of the yield? or is it based on a combination of input factors and their influence on the yield?

it's also important for us to understand how the model is intended to be used. is a mmodel being used to determine what input settings will be used to guarantee a certain yield? or is it being used to understand when something in the process has changed and to direct you to take some type of corrective action?
 
Last edited:
Q

quick_silver

#3
Re: validation of predictive models

Thanks Ma'am..

1) we have six models for each product type which undergoes unique processing
2) the model was determined by using 2 yrs historical data, we have about 50 factors monitored that we believe have impact on the different yield. Now the 3rd party provider they run a correlation analysis and then from that they arrived the regression models. Actually we did a lot of iterations, because upon seeing the 1st model there are certain factors that the models says will have positive impact but violates human logic (it should be negative not positive). And the most acceptable model is the one that we are currently using
3) the importance of the model for us, is first from among the 50 factors we need the model to tell us which among the 50 have really the biggest impact to the yield and so we can focus more our action plans on how to control those factors.
 
D

Darius

#4
Re: validation of predictive models

they run a correlation analysis and then from that they arrived the regression models. Actually we did a lot of iterations, because upon seeing the 1st model there are certain factors that the models says will have positive impact but violates human logic (it should be negative not positive). And the most acceptable model is the one that we are currently using
There are some asumptions that have to be made when using regression, if the asumptions are not met can lead to weir conclusions. One of the most common mistake is to take the model as a lineal ones when the data isn't. If you think that the factor defies the logic; it could be noise/non significant for the model and almost sure that the factor is a value near 0.

If you are woried about the effect of the factor is not taken in account the way it should, you can make a chart of the factor against the difference between the real and the forecasted value. If some effect was not taken in account you will see some trend (a curve), showing that the efect was not taken the way it should. IMHO graphical methods show things that the numbers can't, you may have a non lineal trend but an r squared low because the lack of lineality.

from among the 50 factors we need the model to tell us which among the 50 have really the biggest impact to the yield and so we can focus more our action plans on how to control those factors.
As you may know the bigger the absolute of the factor the more impact on yield will have, but the relation will change if the variables that affect such variation are controled, you can have a strong relationship between a variable and another but if the variable is controled the impact will be negligible, it's like seeing a line from a micron, you will only see the noise (in the case of a line on the center, only black).

You ask about the use of R squared to determine wich model is right, if you are comparing a model result against the other IMHO is better to use the methods used in forecasting to determine wich gives a better results (RMSE, MAPE,MdAPE, GMRAE, MdRAE). IMHO Median Relative Absolute Error (MdRAE) is well protected against outliers.
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#5
well, 50 input factors is a LOT of factors. You said that they used 2 years of historical data? How many independent runs or lots does this represent? it is possible that the model is overfitted. This can give a decent r square value but will result in a model with very little actual predictive power in practice.

the only true way to validate the model is overtime: plot the predicted value vs the actual value for some time, say 20-30 runs (or lots)...
 
Q

quick_silver

#6
Re: validation of predictive models

Thanks Sir Darius. I did calculate the tracking signal of the six models. Accdg to my readings the ideal should be -1 to 5 but unfortunately i got many outliers. That's why i was looking of some other ways of checking the validity. How can i attached the excel file here?
 

Miner

Forum Moderator
Staff member
Admin
#8
There is a definite flaw in the predictive model.

If you follow BevD's suggestion to plot predicted value (Y) versus actual value (X) as well as Actual - Forecast (Y) versus actual value (X), you will see the anomaly clearly.

Ideally, predicted value plotted by actual value should be a line running at 45 degrees with scatter on either side, while Actual - Forecast versus actual value should be a horizontal line at zero with scatter above and below. This data show the reverse, which means there is a very fundamental problem with the equation. That is, its not predicting at all.
 
Q

quick_silver

#9
Thanks Miner.

So moving forward I should recommend that the predicted model should be revisited and create a new one.
My question would then be how can I measure if the model that will be created is a useful one without waiting for new data, is there a way? Because when we accepted the model, the acceptability criteria was on the model or iteration no. with highest r adj square and the model whose equation agrees with conventional wisdom.
we have already paid the company who did the modeling for us. I just realized that the payment should have been after the availability of the new data and that the new data proved that the predicted equation is really useful. It turns out it is a waste money after all. :(

Thanks in advance!
 
Last edited by a moderator:

Steve Prevette

Deming Disciple
Staff member
Super Moderator
#10
For monitoring whatever predictive model is used, I'd suggest running a control chart of the delta between the prediction and the actual. Then see if that delta is stable and predictable, and if so, evaluate if the error is acceptable or not. If there are trends, then there probably is something that is affecting actual production that the model is not taking into effect.
 
Thread starter Similar threads Forum Replies Date
mustomutlu Process Validation Final Report Other Medical Device and Orthopedic Related Topics 2
P Software verification and validation procedure IEC 62304 - Medical Device Software Life Cycle Processes 6
Y SaMD Verification and Validation SOP and Plan IEC 62304 - Medical Device Software Life Cycle Processes 3
C Spreadsheet Validation and Release Qualification and Validation (including 21 CFR Part 11) 2
M Packing Validation & Accelerated Aging Test ISO 13485:2016 - Medical Device Quality Management Systems 2
M Sterile packing validation tests to be performed and protocol Other Medical Device and Orthopedic Related Topics 1
S OEM full range calibrations vs Validation special test points Medical Device and FDA Regulations and Standards News 0
H Software Validation for FFS Packaging Machine Qualification and Validation (including 21 CFR Part 11) 1
Q ISO 13485 7.5.6 Validation - Off the shelf Software ISO 13485:2016 - Medical Device Quality Management Systems 3
M ERP / QMS related software standards for Validation IEC 62304 - Medical Device Software Life Cycle Processes 6
C ISO 19227 Validation Cost Other Medical Device Related Standards 2
A CNC Mill and Lathe - Validation Manufacturing and Related Processes 2
N Example for design and development planning,input,output,review,verification,validation and transfer Misc. Quality Assurance and Business Systems Related Topics 4
N Sterilization Protocol Change in Validation Process and further impacts ISO 13485:2016 - Medical Device Quality Management Systems 1
B Oracle Cloud ERP Validation during Quarterly Patch ISO 13485:2016 - Medical Device Quality Management Systems 1
D Software validation team Misc. Quality Assurance and Business Systems Related Topics 3
W LTPD, AQL, Ppk and Cpk validation sampling plan table Inspection, Prints (Drawings), Testing, Sampling and Related Topics 0
J Validation Sample Size for Tray Seal Qualification and Validation (including 21 CFR Part 11) 3
F AS9100 - Validation, FAIR's, ITAR and Sub-Contracting AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 3
R PCBA process validation Qualification and Validation (including 21 CFR Part 11) 2
A ISO 17025 - Methods validation and clients ISO 17025 related Discussions 3
S Sterilization validation after changing sterilization process provider Qualification and Validation (including 21 CFR Part 11) 3
B Sterilization Validation Plan Other Medical Device Related Standards 3
D Difference between Test Method Validation and Gage R&R Qualification and Validation (including 21 CFR Part 11) 18
T Laboratory Verification after validation ISO 17025 related Discussions 3
silentmonkey Rationalising the level of effort and depth of software validation based on risk ISO 13485:2016 - Medical Device Quality Management Systems 10
D Questions regarding process validation ISO 13485:2016 - Medical Device Quality Management Systems 6
Y We found out we have been using a equipment without validation for past 4 years Quality Manager and Management Related Issues 6
Z Is IQ necessary for laser marking validation? EU Medical Device Regulations 3
E 13485:2016, Sections 4.1.6, 7.5.6 and 7.6 - Validation of Software - Need some Advice please ISO 13485:2016 - Medical Device Quality Management Systems 3
A Validation of Forced Aeration Process ISO 13485:2016 - Medical Device Quality Management Systems 3
E Mentor for Test Method Validation (TMV) Design and Development of Products and Processes 2
M API 4F/7K/8C Design Package Validation Oil and Gas Industry Standards and Regulations 2
I ISO 2233:2000 Question - Medical Device Shipping/Transportation Validation Other ISO and International Standards and European Regulations 1
T Annual Validation as a detection mode on a PFMEA? FMEA and Control Plans 5
B TMV - Selection of TM's for Validation ISO 13485:2016 - Medical Device Quality Management Systems 5
S Forced ServiceNow validation - No change in our current user and functional requirements IT (Information Technology) Service Management 4
P Human Factors / Usability validation in the time of COVID Human Factors and Ergonomics in Engineering 17
C Template for Excel Validation Reliability Analysis - Predictions, Testing and Standards 6
M IT validation for a paper based MD repair company QMS ISO 13485:2016 - Medical Device Quality Management Systems 6
P Unrealistic Packaging Validation Sample Size 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 41
D Test summary report example for design validation wanted - ISO 13485 ISO 13485:2016 - Medical Device Quality Management Systems 1
M Is Validation of Plating Processes required and who is responsible? Qualification and Validation (including 21 CFR Part 11) 11
T ISO 13485 - Process validation at critical suppliers ISO 13485:2016 - Medical Device Quality Management Systems 7
K Software Validation for Measurement Tools used in Process Validation ISO 13485:2016 - Medical Device Quality Management Systems 2
Stoic Manual soldering processes - 100% verifiable, or always requiring validation? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 15
P Design verification driven by new equipment. How is this different than process validation? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 1
S Rees System Validation Qualification and Validation (including 21 CFR Part 11) 1
K PQ validation qualification - Asked to write a PQ protocol ISO 13485:2016 - Medical Device Quality Management Systems 6
Stoic Are any medical device companies using the 2011 FDA process validation guidance instead of GHTF/SG3/N99-10:2004? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 1

Similar threads

Top Bottom