Process Validation sample plans help

[geoli]

Hi, I am the only validation engineer at a medical device manufacturing company. My background has been facility validations, equipment validations etc. But I have now inherited the process validations to perform. Our procedures don't do a good job of explaining how to plan a PPQ from scratch and the people that might have known have long gone. So its just me trying to learn how to do this on the job. Unfortunately(or fortunately!) our customers are much less informed than I am and are unable to help so I have to guide them through process validation.

The process generates a maximum of 135 units (before QC checks). I will manufacture 3 batches for the PPQ, and I need to determine a sample size for two different attribute tests. One test is a visual inspection for different types of defects - binary pass/fail. The other is a type of mechanical testing. The unit either delaminates or not - pass/fail. A trial run of the process was completed: 41 units were manufactured and we have 10% failure rate for the visual inspection attribute test, and a 2% failure rate for the mechanical attribute test.

First I must determine the Confidence and Reliability for both CTQs. For this process I am using 95% confidence. The pFMEA I completed ended with RI level of 2 for the visual inspection test and the mechanical test, so both reliabilities were set to 95%.

My questions:

1. I think my next step is to determine an AQL and RQL for both CTQs.
   The AQLs I've seen thrown around are like 0.0001% all the way up to 10%. But never higher than that. I think the range I like is 0.1%, 0.15%, 0.25%, 0.4%, 0.65%, 1% . The advice I have read says that this number needs to be agreed by the customer, but what's stopping them picking the smallest number and never budging. I need to be able to recommend a suitable AQL with the appropriate drawbacks etc. Is the RQL just set to 5% because my reliability is 95% (i.e. 1-0.95)?
   
   
2. I know there's something called an OC curve. But I don't know how to make it. and how do I use it? Is it for determining how likely my process would pass the sample plan?
   
   
3. I know I need to use binomial distributions for my sample plan. But I don't really get how I determine the sample plan I pick? Do I pick a single sample plan or double sample plan? How do I determine the correct accept number and reject number? I have assumed a=0 r=1 which would give me a sample size of 58 at C=0.95 and R=0.95. As I am manufacturing 3 batches I would round to 60 and sample 20 from each batch (twice for both the CTQs). But as mentioned, I don't know if this is a suitable plan, or how to plot an OC curve to prove it.
   
   
4. And finally, at what point am I sampling for this PPQ. Do I sample after manufacturing but prior to nominal QA inspections - i.e. validating the performance of the manufacturing line. or do I sample after the QA inspections i.e. validating the performance of the process as a whole? I initially thought the former, but I would appreciate other's thoughts.

I feel like I vaguely understand the separate bits of planning this validation, but I don't know how to pull it altogether into a cohesive plan for a customer who understands this less than I do!

Thanks in advance

 

[Tidge]

There is a lot to unpack here, so I'll tackle the easy part: Basic math says if you want to be 95/95 (confidence/tolerance) that a defect isn't present in a sample, the smallest possible sample size for the hypothesis test is 59 units (with no defects observed). Seeing failures in 41 units indicates you are far from having achieved 95/95 levels of being defect free. This isn't the same thing as guaranteeing that consecutive lots (that pass this sampling) from this process will be 95% free of defects...

...OC curves are a different sort of beast. @Bev D can help explain a collaborated spreadsheet (XLS) here that can help you to understand how sampling plans interact with AQL and RQL (LTPD, UQL) and also how to set sample sizes based on known defect rates.

see here for link to spreadsheet

 

[Bev D]

It’s good that you are asking questions. Sampling plans are not some color by number app no matter what the google may tell you.

First: any sample plan must be applied to a single lot or batch - you cannot apply the sample plan across batches. The math is only for a single lot. Period. So forget that.

The link Tidge provided is to sample calculator for AQL, RQL and OC curves. Yes OC curves give you the relative probabilities of detecting a defect rate given a sample size and accept number. I think you will be surprised.

Next, sample plans ONLY work if the sample is RANDOMLY selected from the entire lot. Not the last pieces or the first pieces or the easiest to retrieve pieces.

Now these sample plans are intended for acceptance sampling - they will tell you (sort of) what the lot you sampled is doing but they are not predictive - which is what you want for validation is intended for. Typically validation plans have much larger sample plans to predict actual defect rates…up to 100%

Your Customer should tell you what the maximum defect rate they will accept. That is the RQL. BE sure you understand what he AQL and RQL actually mean…And sure they are going to say 0. This is modern quality. Unless your severity is low they are probably going to want zero defects - this is the dilemma of validation and acceptance sampling. Back in world war 2 it was OK to have 1 in 1000 grenades explode by themselves but not today. If your Customer doesn’t care then you will have to think about what your competitors are doing.

Lastly the hardest to accept - even for some statisticians - is that if your process settings don’t span the maximum allowable range of operation then you will not have anything predictive at all because you do not have a representative sample. And “PQ” is typically run at nominal, so you will be wrong even if you have no defects in the sample because the sampling frame is wrong.

Well that’s enough to start…

 

[geoli]

There is a lot to unpack here, so I'll tackle the easy part: Basic math says if you want to be 95/95 (confidence/tolerance) that a defect isn't present in a sample, the smallest possible sample size for the hypothesis test is 59 units (with no defects observed). Seeing failures in 41 units indicates you are far from having achieved 95/95 levels of being defect free. This isn't the same thing as guaranteeing that consecutive lots (that pass this sampling) from this process will be 95% free of defects...

...OC curves are a different sort of beast. @Bev D can help explain a collaborated spreadsheet (XLS) here that can help you to understand how sampling plans interact with AQL and RQL (LTPD, UQL) and also how to set sample sizes based on known defect rates.

It’s good that you are asking questions. Sampling plans are not some color by number app no matter what the google may tell you.

First: any sample plan must be applied to a single lot or batch - you cannot apply the sample plan across batches. The math is only for a single lot. Period. So forget that.

The link Tidge provided is to sample calculator for AQL, RQL and OC curves. Yes OC curves give you the relative probabilities of detecting a defect rate given a sample size and accept number. I think you will be surprised.

Next, sample plans ONLY work if the sample is RANDOMLY selected from the entire lot. Not the last pieces or the first pieces or the easiest to retrieve pieces.

Now these sample plans are intended for acceptance sampling - they will tell you (sort of) what the lot you sampled is doing but they are not predictive - which is what you want for validation is intended for. Typically validation plans have much larger sample plans to predict actual defect rates…up to 100%

Your Customer should tell you what the maximum defect rate they will accept. That is the RQL. BE sure you understand what he AQL and RQL actually mean…And sure they are going to say 0. This is modern quality. Unless your severity is low they are probably going to want zero defects - this is the dilemma of validation and acceptance sampling. Back in world war 2 it was OK to have 1 in 1000 grenades explode by themselves but not today. If your Customer doesn’t care then you will have to think about what your competitors are doing.

Lastly the hardest to accept - even for some statisticians - is that if your process settings don’t span the maximum allowable range of operation then you will not have anything predictive at all because you do not have a representative sample. And “PQ” is typically run at nominal, so you will be wrong even if you have no defects in the sample because is wrong.

Well that’s enough to start…

Thank you both of you for taking the time to reply! It's so nice that there are people out there that are willing to help someone at the start of their career.

I've downloaded the spreadsheet.

I used the basic sample size calculation you referenced @Tidge to get 59 samples at 95%/95%. Luckily a batch size is up towards 135 units. So this can be achieved.

In the spreadsheet on the Binomial RQL-AQL tab I placed 59 in cell C6.
For the AQL I used 0.1%. Looking over our historic process validations for other customers somewhere between 0.1% and 1% seems okay for attribute CTQs.
For the RQL a different customer sets it to 5% or 10% for attribute CTQs. So I've picked 5%.

For the P(acc) and P(Rej) I've set these to 95% and 95% - correct me if I'm mislead here. I did see the calculations: P(Rej) =1-BINOMDIST(c,n,p,TRUE) and P(Acc) =BINOMDIST(c,n,p,TRUE) but I wasn't sure what to plug into the equations.

The plan gives me a=0 r=1 single sampling plan.

SO I've played around with the other tabs, "OC Curves"
I have from the pilot runs: 10% failure rate for the visual inspection attribute test (I'll call PPQ-A), and a 2% failure rate for the mechanical attribute test (I'll call PPQ-B). So I now want to check if this sample plan is likely to pass or not. Placing 59 in cell C6 and then 0 in C7 (clearing the other cells). Ad then I want to look at the defect rates of 0.1 and 0.02 for PPQ-A and PPQ-B respectively.
It looks like a 100% chance of rejecting a lot for PPQ-A and ~70% of rejecting a lot (times 3? as I'm sampling 3 batches?) for PPQ-B.

So my conclusion here is that PPQ is not suitable for this manufacturing line? It would be a waste of time and money? 100% verification would be better until the failure rate is lower? But isn't that a bit harsh? 2% failure rate isn't that bad surely?

Am I on the right path?

 

[Bev D]

First 95/95 refers to 95% Reliability with 95% Confidence. This means you would detect a 5% defect rate 95% of the time that it was present. A lower defect rate would have lower probability of ’detection’ A higher defect rate would have a higher probability of ‘detection’. This is an RQL plan…the OC curve confirms this…

I don’t know what severity your mechanical defect has but yeah 2% defective is pretty bad unless the severity is trivial.

There is no sample plan that will magically validate a 10% defective process or even really a 2% defective rate. The manipulation of mathematical formulas doesn’t improve process performance.

 

[Tidge]

Have you first characterized the process using an OQ?

I've been adjacent to manufacturing processes where the "nominal" production settings where very poorly established and/or led to significant amounts of variation. I suppose the word "incapable" could have been invoked.

Specific to laminates... are these visual defects things like bubbles/contaminates throughout the entire surface or poor fiducial (edge) adhesion?

 

[geoli]

Have you first characterized the process using an OQ?

I've been adjacent to manufacturing processes where the "nominal" production settings where very poorly established and/or led to significant amounts of variation. I suppose the word "incapable" could have been invoked.

Specific to laminates... are these visual defects things like bubbles/contaminates throughout the entire surface or poor fiducial (edge) adhesion?

There are POQ studies in the works to look into raw material concentration and what those look like under nominal manufacturing conditions. I was banking on the PPQ to validate the "manufacturing conditions" (e.g. RPM, temperature etc.) as they are actually very sensitive; a small change will cause a very obvious problem which the operator can manually intervene. Each process run is very short and quite manual and there are in process scrapping/reworking.

So for that section of the process we are mainly relying on the validation of the equipment and then 100% verification of the process output i.e. QC of the final product. We wanted to use PPQ to determine if our process has minimal defects to reduce that sampling from 100% verification to a sampling plan like those found in ISO6001-1.

The delamination failure mode I referred to is a visual check for the adhesion of a layer of material from a different layer of material. A quick manual mechanical test is performed to verify that the two materials are adhered and do not delaminate from each other. Bubble and contaminants etc would fall under visual inspection test for different types of defects other defects are damage, incorrect design, quality of manufacture etc.

Sorry for the vagueness - obviously I'm under NDA and can't talk about specifics

First 95/95 refers to 95% Reliability with 95% Confidence. This means you would detect a 5% defect rate 95% of the time that it was present. A lower defect rate would have lower probability of ’detection’ A higher defect rate would have a higher probability of ‘detection’. This is an RQL plan…the OC curve confirms this…

I don’t know what severity your mechanical defect has but yeah 2% defective is pretty bad unless the severity is trivial.

There is no sample plan that will magically validate a 10% defective process or even really a 2% defective rate. The manipulation of mathematical formulas doesn’t improve process performance.

Unfortunately the severity is rated high in the pFMEA. But this has been very eye-opening as to what constitutes a bad yield! I totally agree, I don't like manipulating these formulae to get what I want. I shall propose that PPQ is not suitable and that we should continue with 100% verification.

Thank you for your help