Test Method Validation

shimonv

Trusted Information Resource
#1
Hi Colleagues,
I recently saw a proposal from a consulting firm to do test method validation for one former client of mine.
The plan looks like a very big project (19 TMV's) and would take months to complete.

My questions:
1. How do you determine when TMV is necessary? what has been your experience / best practice.
2. Can you avoid using fancy statistics (Gage R&R / ANOVA)? no offense intended for statisticians :)
3. Can you do one TMV report for a product line instead of multiple TMVs which would take much longer?

Thanks in advance.
Shimon
 
Elsmar Forum Sponsor

chris1price

Trusted Information Resource
#2
I worked for a large multinational who received a warning letter from FDA, the agency basically included TMVs, saying, "if your final inspection test isn't validated, you don't have a valid release method". The same for incoming materials and potentially for design verification tests. As a result, we spent 18 months remediating all inspections that affected product release (both variable and attribute) across 15 plants. This wasn't just at final QC, but could be in-process tests that were not repeated at a later stage.

We did Gauge R&R or attribute MSA for all the tests. It was easier to do it than argue for another method. If they don't already have it, invest in a copy of Minitab.

The point drummed home to me is that you are validating the test, not the product. If the same test is performed on different products, you should be able to show one validation is sufficient. This is especially true on tests for incoming materials. As an example we had tubing going into 100+ SKUs, but only one TMV to measure diameter, (or maybe one internal and one external TMV).

19 sounds like a lot, but if you have a good procedure and you set up templates for protocols and for reports, that can be reused for every test, it shouldn't be too burdensome.
 

yodon

Staff member
Super Moderator
#3
1. How do you determine when TMV is necessary? what has been your experience / best practice.
That's kind of a hard one to put bounds around since there are so many test types. If you ask the question "how do I know my results are valid?" it may give an idea. For example, if you have a test fixture, how do you assure that the upper and lower bounds are accepted and anything outside is rejected? Even if you have manual inspections, how do you get some level of assurance that defects are rejected?

Of course, the rigor should be risk-based.
 

William55401

Quite Involved in Discussions
#4
Great advice (as usual) from Yodon. "How do you know when necessary...?"

Your process FMEA should identify when process validation is needed for mfg process. Similarly, that FMEA must include test and inspection points. A final test would have a higher risk profile and trigger the need for validation vs a lite version of the test at the sub assembly level.

For one project (yes, triggered by FDA audit), our initial focus was to ensure the TMs were validated ASAP and then we went after the mfg processes. pFMEA was our prioritization tool.

HTH. Have fun. Enjoy the ride.
 

shimonv

Trusted Information Resource
#5
Thank you all. For the record, I am not interested in taking this project, but I am interested in better understanding the principles of TMV.

I get that it all hinges on pFMEA. So, could I state in a TMV plan that only risk with high severity (4 or 5), regardless of probability of occurrence, require test method validation?
To put it differently, how do I avoid doing TMVs for every type of testing, from incoming to outgoing, which is logical but very burdensome.

-Shimon
 

William55401

Quite Involved in Discussions
#6
I think you are on the right track. However, think about flow. Also, think about test results that may show up in your DHR as final acceptance criteria. Could you have a "3" that is an essential design output / a characteristic essential to function? Look at 820.30 (d). How do you track these outputs through your process? Most orgs have a control plan in place for these. Shorthand is EDO. Essential Design Outputs. At a minimum, I would ensure test methods are defendable (TMV in place) for each EDO measure.
 

Tidge

Trusted Information Resource
#7
I get that it all hinges on pFMEA. So, could I state in a TMV plan that only risk with high severity (4 or 5), regardless of probability of occurrence, require test method validation?
All test inspection methods require validation (if you want to claim how well you know they work).

I don't like the bolded proposal because you may have high severity risks that never occur, and/or low severity risks that always occur. There is also a negative (IMO) because if the ratings in your risk files are qualitative it will be VERY difficult to demonstrate how any given TMV (with associated study designs) consistently aligns with the application of the ratings in the risk files. Leveraging the risk files to determine a measure of effectiveness of controls (like test and inspection) implies that a quantitative assessment about the need for controls is understood. The TMV study designs are usually constructed based on the need to demonstrate a risk control at a certain level.

It is common that risk files are constructed with some sort of dual qualitative/quantitative scale, as soon as numbers show up it is possible to do 'back-of-the-envelope' calculations to see if the TMVs are supported by reasonable study designs or not. If there aren't any numbers in the RMF scales, then the TMVs are (very likely) completely subjective.

My personal opinion (based on some experience) is that 3rd party auditors are usually not that strong on "the maths", but if you invite such a person into this discussion by arbitrarily trying to leverage a qualitative scale from the risk files, you may be inviting all sorts of uncomfortable question... if not from the auditor but from whomever the auditor calls to discuss. Frankly, it would better to establish TMVs based on study designs (there are different types, of course) that guarantee a known confidence level, power, etc... and just plan to do them all. It is better to discover that you need fewer samples to support the TMV than it is to discover that a TM doesn't work like you thought it was supposed to.
 

shimonv

Trusted Information Resource
#8
Thank you Tidge for your thoughts.
So, in your opinion - all test inspection methods should be validated and the qualitative/quantitative values in the risk management file can be used to assess if the TMVs are supported by sufficient study designs.

I don't think it's practical. There must be a good balance between using training records as means of validation and TMVs at critical junction points (based on pFMEA) for added confidence.
 

Tidge

Trusted Information Resource
#9
Allow me to share an example: Suppose a manufacturer has some sort of rating for "detection" that is 1-to-5 on a "powers of ten" scale, such that

5 = 1-in-1 chance of 'escape'
4 = 1-in-10 chance of escape
3 = 1-in-100 chance of escape
2 = 1-in-1000 chance of escape
1 = 1-in-10000 chance of escape

The manufacturer implements training, and "conservatively" estimates that the training only allows 1-in-1000 chance of escapes, with no TMV, and assigns a rating of 2 to their inspection.

How hard is the 3rd party going to have to look before an NCR is found that should have been caught at inspection (but wasn't), and what is the story going to be at that point?

EDIT: I should add this question, because this is the 'easier'/'more appropriate' route for auditors: What will the manufacturer show the auditor to demonstrate that the manufacturer looked hard enough to determine that the rating=2 was appropriate.
 

shimonv

Trusted Information Resource
#10
Thanks Tidge,
There will always be risk and NCRs.
The art of QA is to find the most efficient and compliant method to reduce the risk to an acceptable level.
 
Thread starter Similar threads Forum Replies Date
D Difference between Test Method Validation and Gage R&R Qualification and Validation (including 21 CFR Part 11) 18
E Mentor for Test Method Validation (TMV) Design and Development of Products and Processes 2
M Test method validation - Is MSA (MSA1, MSA2, MSA3 and linearity) a good solution? Medical Device and FDA Regulations and Standards News 2
P Test Method Validation for Tensile Strength Tester Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 1
S Test Method Validation or MSA for Batch Manufacturing Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 1
A Test Method Validation on Elongation Machine General Measurement Device and Calibration Topics 0
J Attribute Test Method Validation 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 1
D Effect of Fixtures on Test Method Validation Design and Development of Products and Processes 1
S ANOVA and Test Method Validation for a Drug Delivery Device Statistical Analysis Tools, Techniques and SPC 4
S Medical Device Mechanical Test Method Validation Design and Development of Products and Processes 8
S Writing a Test Method Validation Procedure as well as Validation of Test Methods General Measurement Device and Calibration Topics 1
B Handling, Storage and Distribution - Package Validation Test Method Options? US Food and Drug Administration (FDA) 7
S Test Method Validation Help - New titration test method, not defined in the USP Qualification and Validation (including 21 CFR Part 11) 2
J Test method validation on raw materials, intermediates and final device Inspection, Prints (Drawings), Testing, Sampling and Related Topics 9
M Proper method to test flow of a regulator Inspection, Prints (Drawings), Testing, Sampling and Related Topics 1
T ISO 17025 Method Verification - One sided “F” test or two sided “F” test ISO 17025 related Discussions 1
A Test method for presence or absence of hexavalent chromium on a metal coating RoHS, REACH, ELV, IMDS and Restricted Substances 4
C Flatness Measurement Errors from a Datum - Test Method General Measurement Device and Calibration Topics 3
D ISO 17025 5.10.2(e) - Test Method ISO 17025 related Discussions 4
J International Standards governing Test Method Validations ISO 13485:2016 - Medical Device Quality Management Systems 1
C What is the most appropriate Bioburden Test Method for Surgical Gloves ISO 13485:2016 - Medical Device Quality Management Systems 10
L Find the right Test to fail method according to the standards of LV 124 FMEA and Control Plans 1
M Biological Monitoring Program Environmental Bioburden Test Method Needed ISO 13485:2016 - Medical Device Quality Management Systems 1
M MSA question on Cg and Cgk Calculation Method and Type of Test Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 1
S Looking for A Non-Destructive Crimp Test method to replace Pull Test Inspection, Prints (Drawings), Testing, Sampling and Related Topics 2
M Destructive VTMV Subgroups - A Destructive Variable Test Method to Validate Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 11
M How to validate a Dye Test method used to evaluate Package Integrity Pharmaceuticals (21 CFR Part 210, 21 CFR Part 211 and related Regulations) 8
T ISO 17025 Clause 5.4.2 - Calibration and Test Method Requirements ISO 17025 related Discussions 6
D ASTM 684/A684M-10a Bend Test Method Manufacturing and Related Processes 2
C Actual Standard Test Method for Alloy Chemical Analysis? Other Medical Device and Orthopedic Related Topics 3
S Understanding the Test Method of Defibrillation IEC 60601 - Medical Electrical Equipment Safety Standards Series 7
W Oven Leak Test Method - Integrity of the chamber to hold gas Qualification and Validation (including 21 CFR Part 11) 3
J Container Closure Integrity & Sterility Test - The Dye Immersion Method Quality Assurance and Compliance Software Tools and Solutions 16
J Easy Method to Test a Caliper - Good or bad? General Measurement Device and Calibration Topics 10
T Number of Test Samples - ASTM D2137 Method A 9.3.2 Various Other Specifications, Standards, and related Requirements 1
D Leak Test Method for Balloon Catheters EU Medical Device Regulations 2
P Sample Manufacturing Method for Ball Drop Test of Automotive Tempered Glass General Measurement Device and Calibration Topics 5
M ASTM F88 Standard Test Method for Seal Strength - Designation F88-09/M88-09 ver 15.1 Reliability Analysis - Predictions, Testing and Standards 6
G Definition ASTM D624 - Test Method for Tear Strength of Vulc. Rubber & Thermo. Elastomers Definitions, Acronyms, Abbreviations and Interpretations Listed Alphabetically 1
A XRF Analysis - "Worst Case" for an XRF test method? General Measurement Device and Calibration Topics 4
Q Looking for a Karl Fischer test method - Mettler DL37 KF Coulometer Inspection, Prints (Drawings), Testing, Sampling and Related Topics 1
A Test Method - DIN ISO 1133 for melt flow needed to verify supplier results Manufacturing and Related Processes 1
M Ambient vs. controlled test laboratory environment Test Method Evaluation Plan Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 1
B Test Method Qualification - Measurement Capability - What else is involved? Reliability Analysis - Predictions, Testing and Standards 1
S The essential criteria for inclusion in a test method Inspection, Prints (Drawings), Testing, Sampling and Related Topics 5
K Using an "obsolete" test method - TS 16949 Audit Non-Conformance General Measurement Device and Calibration Topics 13
S 17025 Flexible Scopes of Accreditation - Laboratory to 'self-assess' new test method ISO 17025 related Discussions 2
G Seeking: Powder Coating on Steel Test Method or Procedure Various Other Specifications, Standards, and related Requirements 10
J Old "outdated?" standards - Seeking an ASE J964 "grind lead test method" Other ISO and International Standards and European Regulations 6
Y Attribute Gage R&R - MSA 3rd Edition - 'Hypothesis Test Analysis - Cross Tab Method' Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 7

Similar threads

Top Bottom