Determination of Sample Sizes when Auditing

An oldie but goodie...

How do you determine sample size when auditing records?

How do you determine sample size when auditing a process across multiple production lines?

How do you determine sample size when asking employees how their company quality policy?

What statistically valid method do you use to determine various sample sizes?

Marc said:

An oldie but goodie...

How do you determine sample size when auditing records?

How do you determine sample size when auditing a process across multiple production lines?

How do you determine sample size when asking employees how their company quality policy?

What statistically valid method do you use to determine various sample sizes?

Click to expand...

Marc,

The sample has to be large enough for the auditee and the audit client to accept the audit findings as fact.

And the findings or records always include the sample size.

But, like you, I'm looking forward to the answers in this thread.

John

There is no general requirement for audit samples to be statistically valid. That could be prohibitive in many cases. Auditors look for objective evidence. Regardless of the sample size, if an auditor looks at n records and finds no nonconformity, there is no assurance that record n+1 wouldn't be a bad one.

I am asking because like many here, my auditing experience goes back quite a few years. The other thread got me to thinking about how I *really* determine sample size in various situations whether it be records or even how many employees to ask the old "What does your company quality policy mean to you and how does it affect what you do?" which we have been asking people for years.

I have always been at odds with this aspect of auditing because it has always been something you typically have to do "on the fly" since until you are in the middle of doing an audit you often don't know things such as how many purchase orders there will be.

That said, how do you come up with an audit plan and how does "on the fly" determination of sample size affect the iplanned audit schedule.

I know the old "what ever you and your client are comfortable with", not to mention aspects such as "the risk" level (etc.), but again - How do you determine that?

I will say I think audit plans are better today than say 10 or 15 years ago. It has evolved into a more defined planned audits. I'm not trying to make a big deal out of this. I'm simply curious as to how others handle determining sample sizes in various situations.

Jim Wynne said:

There is no general requirement for audit samples to be statistically valid. That could be prohibitive in many cases. Auditors look for objective evidence. Regardless of the sample size, if an auditor looks at n records and finds no nonconformity, there is no assurance that record n+1 wouldn't be a bad one.

Click to expand...

I agree there is no requirement, or at least none that I am aware of. If I have 5000 records (as an example), is a sample size of 100 significant enough to even warrant sampling any more than a sample size of 10 - Or of not bothering to take a sample at all.

I think we all know that in any sampling plan other than 100% there is no assurance that record n+1 wouldn't be a bad one. That is a given.

Audit sampling of management system processes basically needs to provide confidence that the sample REPRESENTS/IS REPRESENTATIVE the item(s) being assessed. In most cases, sampling is determined by (non-scientific) professional judgement of the auditor(s) at hand. However, the latest version of ISO 19011(2011) in Appendix B also talks about Statistical sampling

B.3.3 Statistical sampling
If the decision is made to use statistical sampling, the sampling plan should be based on the audit objectives and what is known about the characteristics of overall population from which the samples are to be taken.

— Statistical sampling design uses a sample selection process based on probability theory. Attribute-based sampling is used when there are only two possible sample outcomes for each sample (e.g. correct/incorrect or pass/fail). Variable-based sampling is used when the sample outcomes occur in a continuous range.

— The sampling plan should take into account whether the outcomes being examined are likely to be attribute-based or variable-based. For example, when evaluating conformance of completed forms to the requirements set out in a procedure, an attribute-based approach could be used. When examining the occurrence of food safety incidents or the number of security breaches, a variable-based approach would likely be more appropriate.

Click to expand...

OK - Determination by "professional judgement". That's pretty open?

From IMDRF (International Medical Device Regulators Forum)

Guidelines for Regulatory Auditing of Quality Management Systems of Medical Device Manufacturers ? Part 2: Regulatory Auditing Strategy -

https://www.imdrf.org/docs/ghtf/fin...-guidelines-for-regulatory-auditing-part2.doc

Auditors may select samples based on factors which are most likely to affect the safety. In planning quality management system audits (see also section 6.6 Audit Planning), auditors need to consider many factors (e.g., the scope of the audit, the classification of the medical device(s), the complexity of the medical device(s), the intended use, applicable regulatory requirements, results of prior audits, etc.). Within each subsystem, sampling may need to occur in order to evaluate the effective implementation of the particular subsystem (and related subsystems). Tables 1 and/or 2 depicted in Appendix 1 may be used in determining appropriate statistical sample sizes.
.

I think that the type of record/evidence you are looking at also dictates the sample size e.g. if you were checking some purchase orders you may take 5 or 6 examples whereas if you were checking construction project files you most likely won't have the (or the will) to check so many.

Remember also that we are auditing processes now so it may be that we take just a small number of samples and follow them through the system - it takes time.

Colin has a good point - and another "dimension" can be the "scope" of the process records. By that I mean, not just a quantity, but the type of record. If we take purchasing, there may be raw materials purchased, services (calibration, maintenance,) and out sourced value-added processes (machining, painting) and each has it's own specific requirements. This adds further dimension to what is chosen - take 5 - 7 of each and you've got a pretty large task on your hands!

However, if you take a "risk based" approach, you'll spend less time looking at suppliers and products/services which have never been a problem, and focus on the new or changed supplier(s), or the new/changed product spec. etc and so on. Now that's a sample...