AQL for Effectiveness Check - What AQL tiers to use on this type of sampling?

[JeantheBigone]

Good day Covers!
We're looking at doing an effectiveness check to see if a procedure is being followed.
Does anyone have a rule of thumb to share on what AQL tiers to use on this type of sampling?

 

[Bev D]

AQL inspection plans are not intended for this type of assessment.

a better focus is to think about how you actually assess compliance.

some examples:
Are you going to watch the "operators"? then your sample size needs to be 'long enough' that the operators lose that 'I'm being watched' behavioral bias. And you'll need to watch everyone who performs the process - a sample won't really suffice in this case.
Do you have some electronic rail that can be monitored? (Like electronic batch records) then all you can assess a large number without observation bias. just make sure you hit all of the operators at least twice.

A few other considerations: sample size isn't as important as the representativeness of the observations.

Your observations should be spread over time: operators will often comply early after training and drift off course over time.
Are there conditions that make compliance more difficult? such as volumes, looming deadlines, first shift managerial chaos over third shift calm, different organizational cultures that support or denigrate compliance (R&D vs Operations for example).
The important thing is observe or assess during the most stressful times and not the easy times.

 

[Eredhel]

Can you give any more details on what you're checking specifically?

 

[Mike S.]

AQL inspection plans are not intended for this type of assessment.

It depends. If the check is to detect a large percentage of defects or lack thereof, for example to see if the assemblers are adding a new clamp to an assembly that did not have this clamp in the past, it could be fine.

If you are trying to detect a small percentage of defects (i.e. to determine if a process change eliminated a 2% defect rate) an AQL check is not likely to be effective.

 

[JeantheBigone]

Sorry!

I should have been clearer that this is an effectiveness check on a quality process procedure, not a production procedure.

Without going into too much detail, we want to do an effectiveness check using records that should have been signed off on by certain authorized persons. If the record was appropriately approved, then the record "passes." If the individual performing the work did not get the proper approval, the record "fails."

 

[Al Rosen]

Sorry!

I should have been clearer that this is an effectiveness check on a quality process procedure, not a production procedure.

Without going into too much detail, we want to do an effectiveness check using records that should have been signed off on by certain authorized persons. If the record was appropriately approved, then the record "passes." If the individual performing the work did not get the proper approval, the record "fails."

FDA uses this sampling plan for records.

 

[Bev D]

If you are looking at the compliance of a single person - that it appears you suspect of ‘wrong doing’ - why wouldn't You look at all of them over some period? This doesn’t sound like a situation where any ‘defects’ are allowed and no sampling plan can handle that - just look at all of them.

Again you might think about the time period to check for some conditions that might result in this individual’s non-compliance.

 

[JeantheBigone]

If you are looking at the compliance of a single person - that it appears you suspect of ‘wrong doing’ - why wouldn't You look at all of them over some period? This doesn’t sound like a situation where any ‘defects’ are allowed and no sampling plan can handle that - just look at all of them.

Again you might think about the time period to check for some conditions that might result in this individual’s non-compliance.

I'm not sure why it appears we suspect one single person of "wrongdoing." This does not describe the situation at all.

This involves a procedure which applies to multiple individuals in one department who had become accustomed to not getting approval from a different department for certain documents. They were made aware of the requirement and given training last year and we want to do an effectiveness check. In the past year there were over 700 such documents, and given the nature of the documents (which I prefer not to divulge) if they have not been signed off on, we do not consider it "critical" but only "major." What I was asking for was an appropriate AQL.

This is not shift work, does not involve operators standing at a machine, and the only person who have to comply are all in the same department.

 

[Bev D]

my apologies - your description of the nature of not getting the appropriate approvals and the use of the term certain led me to use the air quoted 'wrong doing'. no intention to imply illegal or immoral acts. wrong doing in the sense of not doing the required thing....it is always difficult to discern the nature of vague situations.

so 700 documents. the AQL 'tiers' are related to how critical the defect is. as you know this requires an understanding of the probability of missing the non-compliance vs the effect of the non-compliance should it occur. and all AQL testing has a probability of missing some level of non compliance. This isn't a straightforward thing for the AQL tables as they were negotiated AND they are based on the lot/population size. neither of which result in statistically valid estimates of the probability of missing the non-conformance. This is exacerbated by the fact that non-homogeneity of the non-conformance renders any such estimates unreliable. human behavior is notoriously non-homogenous. it is quite possible that for a short time after the training they all complied. as time goes on and the causal element of what prompted them to non-compliance recurs they are more likely to not comply. as time goes on and they do not get caught they will be tempted to increase their level of non-compliance. Another consideration is that AQL sampling is not intended for one time use - it's probabilities are based on a continual stream of sampling...so again I would advise that AQL sampling is not the appropriate tool for this job. *I* would sample 100% of the documents form a few time periods when they are most likely to have not complied, or from when you have other reasons to suspect that they didn't comply.

However, since you have stated that the non-compliance is a major: in your mind which tier is appropriate for an AQL based sample? then you must determine what defect level is ACCEPTABLE because that is the defect level that drives the sample size within your selected tier.

 

[Mike S.]

Sounds like you may want to audit the process a couple different times. As Bev said, you need to weigh how much work (sampling/auditing) you want to do vs. how much risk you can tolerate.

For example, based on the “rule of 3’s” (p=3/n) if you inspected 50 pieces (samples) and found 0 defects you could say you are 95% sure you have </= 6% defects in the lot. Is that acceptable risk? Only you can say.

If you inspected only 20 and found 0 defects you could say you are 95% sure you have </= 15% defects in the lot.

If you inspected 100 and found 0 defects you could say you are 95% sure you have </= 3% defects in the lot. And so forth….