How to find AQL and LQ levels from Product Risk Assesment Probabilities?

[qLience]

Hello everyone.

I am currently going to conduct a validation and I am a bit lost because normally I am just given a LQ level which I then should work from. But now I need to derive the AQL (ISO2859-1) / LQ(ISO2859-2) level from the Product Risk Assesment due to it is not a standard currently for the company I am a consultant for.

The situation is that I have a team of vision specialist team but they do not know when they have achieved their goals which have a attribute response (not good/good). I want to use LQ level due to we are doing validation which should present the worst batch where AQL is more average batch performance. But I want to first find my AQL level to find my LQ level.

I forexample then have one type a error where I know the following from my Product Risk Assesment:
- P_defect = 1.00E-02 : Probability of occurrence
- P_nd = 1.00E-03 : Probability of non detection
- Severity of harm = 2 (Results in temporary injury or impairment not requiring professional medical intervention.)

How do I find my AQL level based on my P_defect, P_nd and Severity and how can I change it if I want to save money on sampling but still fulfill requirements? And how do I convert the AQL level to a LQ level?

Best regards qLience

 

[NotMe]

The recommended method is to solemnly use the severity to determine the acceptance quality level. Which acceptance level to choose is up to the company's quality department. Only the company is able to balance the consumer risk and the business risk -- this is only valid, if no regulatory requirements exists, i.e. for sterilisation processes we are not allowed to select the consumer risk at will.

Unfortunately, I have to admit that we don't use the upper described method in our company. Instead, we use the so called risk priority number (RPN) to select the acceptable risk. The RPN method has flaws, see. e.g. this article.

 

[Bev D]

First your risk is determined by severity (forget the RPN number is it is gibberish and you only need the severity to determine how many defects you can tolerate.)

Secondly Remember that the AQL is the ACCEPTABLE defect rate that will be allowed to ship; you really need the RQL. LQ is so old fashioned it precedes my life…forget about it.

You can check out my spreadsheet that calculates sample sizes and OC curves for many different scenarios. But remember that these are for ongoing lot by lot inspections and are NOT for validation.

 

[NotMe]

To my knowledge the default method to validate a process (OQ, PQ) which yields a binary outcome (OK/NOK) is to define an acceptable failure rate (RQL) and an acceptable confidence level (gamma=1-alpha) and then to demonstrate that the process satisfies this failure rate. For many years we use this method to qualify our processes in the medical device industry and we never had any problems with auditors.

@Bev D: You state that this method is only for LOT by LOT inspection. Which other method do you use to validate a process, which yields a binary result (OK/NOK)?

 

[Bev D]

I’ve written about this many, many times.





AQL/RQL plans (including the confidence/reliability which is just an RQL plan) are by their very nature enumerative. They were first developed during the 2nd World War and formalized afterward in coordination with industry (i.e. negotiations more than statistics); therefore they have only a tenuous connection with statistical validity…They only tell you

1. about the lot in front of you
2. Are only as good as the representativeness of the sample (aka sampling frame)
3. At best can only tell you that the process doesn’t exceed some defect rate and
4. the defect rates used are horribly and embarrassingly large. 5% or even 1% are incredibly large in this day and age and for a critical medical device they are just plain abhorrent.

BUT the FDA accepts them because, well, that’s what has always been done. This is evident in the default use of 95% confidence and the use of the unthought about 95% probability level for any AQL since the default unthought about p value is 5%…very few people give any consideration to - if they remember or were even taught - what Fisher actually said about a 5% p value…


If you read my brief paper on validation testing you will see what I have used for the last 20+ years…

 

[qLience]

Hello Bev D and NotMe, Thank you very much for taking your time to spread some knowledge!

Could you take me through step-by-step how you determine AQL/RQL level from you product risk assesment?

I know what i have stated but i also know that my Severity is 2 (Marginal) and I know my batch size will be about 240, but is there a standard within medical device how much the accetable PPM must be based on the severity?

Score	Severity	PPM ??
5	Catastrophic
4	Serious
3	Critical
2	Marginal
1	Negligble

And how do i then determine whether my AQL level is sufficient compared to my probability for defect and probability of detection?

If you could reference some standards on your answer then this would be much appreciated!

Bev D your sampling plan generator looks extradinary but i do not exactly how to use it.

 

[NotMe]

Bev, I am not convinced that we are very far apart in our opinions. E.g. I learned that during the OQ we have to test the "worst-case" conditions for the critical characteristics. Thus, we pick those combinations of the input parameters (within the operating range), which are known to be worst for the product. We account for this worst-case scenario by allowing a higher failure rate for acceptance in the OQ then in the PQ. Thus, in the OQ we pick a lower (RQL, confidence) pair then in the PQ.

Regarding your comments about future products, I don't see how it is possible to ensure during validation that all future products are conforming. Instead, my perspective split into two:
1. During the validation (OQ, PQ) we demonstrate that the current process generates conforming products.
2. After the validation we ensure that the process stays in-control. This ensures that the "future products" are conforming with a "high" degree of certainty. However, as highlighted: This is AFTER the validation and not part of the validation.

In your article you state that you use the current knowledge of physics, material science, and/or chemistry to demonstrate that the product will be conforming. If this is not the situation I describe above (for OQ), could you elaborate on your procedure? Do you follow a procedure, or do you decide from process to process? How about the appropriate sample size for a OQ/PQ? Do you have a procedure describing the appropriate sample size and what do you use instead of a (RQL, confidence)-pair?

 

[NotMe]

@qLience: I would not use five different product risk scores, but only three "high, middle, low". If you want to document that some risks are currently believed to be negligible, then maybe add it as fourth score. However, you will never validate a process, which possesses a negligible risk -- why would you?

Instead of the ppm-value I argued above that I would use (RQL, confidence)-pairs. This is standard in the medical device industry, and everybody knows and understands them.

Your main concern is probably the values for the (RQL, confidence)-pairs. This is tricky, because this depends on your product. My trainer used the following example. Suppose you have two products: First, a clue, which stops the bleeding even if you lost a leg. However, the clue is toxic and the patient is going to die within the next five years. The second product is a hipp replacement. It is toxic as well, and the patient is going to die within the next five years. Although the product risk is equal for both products, the acceptance criteria are probably not equal.

And how do i then determine whether my AQL level is sufficient compared to my probability for defect and probability of detection?

As stated in my first post:I recommend to use only the severity.

A nice reference is Wayne Taylors book. It's extremely expensive, but I enjoyed it, because it presents a self-consistent concept. It only contains minor deviations from the general concept.

 

[Bev D]

Bev, I am not convinced that we are very far apart in our opinions. E.g. I learned that during the OQ we have to test the "worst-case" conditions for the critical characteristics. Thus, we pick those combinations of the input parameters (within the operating range), which are known to be worst for the product. We account for this worst-case scenario by allowing a higher failure rate for acceptance in the OQ then in the PQ. Thus, in the OQ we pick a lower (RQL, confidence) pair then in the PQ.

Regarding your comments about future products, I don't see how it is possible to ensure during validation that all future products are conforming. Instead, my perspective split into two:
1. During the validation (OQ, PQ) we demonstrate that the current process generates conforming products.
2. After the validation we ensure that the process stays in-control. This ensures that the "future products" are conforming with a "high" degree of certainty. However, as highlighted: This is AFTER the validation and not part of the validation.

In your article you state that you use the current knowledge of physics, material science, and/or chemistry to demonstrate that the product will be conforming. If this is not the situation I describe above (for OQ), could you elaborate on your procedure? Do you follow a procedure, or do you decide from process to process? How about the appropriate sample size for a OQ/PQ? Do you have a procedure describing the appropriate sample size and what do you use instead of a (RQL, confidence)-pair?

Many companies do not use the OQ/PQ approach and for some who do the OQ is not a very robust OQ - hence my paper and advice.
IF a company performs a real OQ as it was intended then the PQ can be run at near the nominal as one might do in production. For PQ we always had a requirement that 3 sequential lots must pass. For products that were not lot-based (larger diagnostic instruments) we would determine the supplier lot change points and require that 3 of these sequential lots changes constitute the PQ - of course we also had the luxury of releasing any instrument that passed all acceptance/release testing as each instrument was required to go through release testing. The test was the standard acceptance testing used in production - nothing special - so each lot was accepted or rejected on its own using an RQL based sampling plan. We also had SPC to ensure that things stayed stable or for early detection if something changed. Basically for instrument PQ we were just monitoring and reporting on results on a weekly basis until our PQ goals were met.

As for the concern regarding predicting future performance two things are true:
1) you cannot say that your process/product has been validated if you never sample form the extremes that you will in fact allow in the future. (This is of course covered by Deming as an analytic approach rather than an enumerative approach; but it’s also just plain common sense. )
2) OF COURSE, there are things that are not controlled by an organization during production such as suppliers compliance to specifications, and un-thought-of and therefor uncontrolled factors that change causing defects. This is why we have acceptance inspections and SPC…

We never had a “procedure” that covered specific sample size determination as it is not a cut-and-paste type of approach. We did cover the various concepts in-depth in our Quality Engineering training and our FMEA/Development training (that all scientists, engineers, project managers and their managers had to take). Determining sample sizes for new products and changes was the responsibility of the teams that lead the changes. These teams had an expert coach assigned to them to help guide the process. There was also a super expert board that reviewed the change assessments and control plans. OQ sample sizes were based on the challenge to the extremes and were therefore fairly small as you are usually in deterministic space rather than probabilistic space. But there are some times when the failure may be intermittent* even at the extremes and so larger sample sizes were required. Sample size determination was based on the estimation of differences and not RQL…while this leads to larger sample sizes they were much more effective. It is also useful to note that the AQL/RQL approach explodes pretty quickly at the types of defect rates we were dealing with (<<.5%). We also had many products that were ‘re-usable’* such as blood testers so our sample sizes were based on number of runs (or uses) as well as number of instruments…, In addition we had to be cognizant of the use conditions (the old stress-strength interaction) and ensure that the worst case conditions were also tested at the extremes of the allowable variation of the device itself. When testing changes to existing products/processes we used matched pairs or the current & new to keep sample sizes low.

The real issue here is that determining sample sizes is more than understanding how to navigate thru an AQL table or plugging numbers into the confidence/reliability formulas.

 

[Bev D]

Hello Bev D and NotMe, Thank you very much for taking your time to spread some knowledge!

Could you take me through step-by-step how you determine AQL/RQL level from you product risk assesment?

I know what i have stated but i also know that my Severity is 2 (Marginal) and I know my batch size will be about 240, but is there a standard within medical device how much the accetable PPM must be based on the severity?

Score	Severity	PPM ??
5	Catastrophic
4	Serious
3	Critical
2	Marginal
1	Negligble

And how do i then determine whether my AQL level is sufficient compared to my probability for defect and probability of detection?

If you could reference some standards on your answer then this would be much appreciated!

Bev D your sampling plan generator looks extradinary but i do not exactly how to use it.

First you cannot simply determine the AQL (don’t, just don’t) or RQL directly from the severity. The organization must determine the maximum defect rates they want to allow based on the market, regulatory requirements, competitive state and financial costs of failures. For example if your competitor has a .001% failure rate on a severity 3 item, I seriously doubt that the organization would allow a 1% failure rate…once the organization has determined the max allowable defect rates that is your RQL or your reliability for conf/reliability formulas. And the FMEA frequency/probability and detection rate have NOTHING to do with it. NOTHING.