Internal Audit Sampling Plans - Determining Internal Audit Sample Size

[chaosweary]

So I am doing an internal assessment and pull 5 training records to verify compliance. After the audit my manager asks me how did I come up with pulling 5 records, how did you determine not to pull more or less? I said the number was 10% of the personell working in the area, no other logic. Is there any sample plans for how you pull samples internal audits? I thought about using the mil standard 105E or ANSI Z1.4 but those are really for product acceptance sampling, what sample plan or reference can I start using to get him off my back as to the logic of the "determination" of the sample plan?

 

[AndyN]

Well I guess..........

it depends on the purpose of the audit. Since there are a number to choose from, it'd be helpful to know what that was.

Personally, unless you are trying to replicate a 3rd party audit, as in before registration/surveillance, I can't see why you'd (randomly) want to see training records:mg: What are you trying to establish?

Have you had a change in personnel? In such a case you'd be better off finding which specific people haven't had training to perform their assignments.

I'm not surprised that you were challenged on the sampling. I'm thinking the manager was (covertly) challenging the whole purpose of the audit. Using a sampling plan is of no consequence to you and most of all, him. You probably don't have a 'focus' for the audit that management would endorse, so until the planning addresses an issue that they see as value added, the audits will miss vital opportunities.

By way of example; there was a company which always had product quality issues during the summer. They employed local 'interns' to cover for vacation working. Because they chose to audit 'an element a month' (QS-9000 standard), they missed '4.18' every year (one year completely, then by 1 month the next year.....). Shouldn't they have audited the training process when it was implemented to see what was going on?? (i.e. how effective it was).

So, my advice is look at your overall audit scheduling, not the sampling. Don't be random, that's an external audit technique and not for an internal auditor.

Andy

 

[Helmut Jilling]

Andy, I completely disagree! Chaosweary was entirely correct. Of course he is supposed to audit training records as a linkage to shop floor personnel. It is one of the expected linkages between the shop processes and the training process. That is the whole point of this "process approach" to connect the linkages. Or, are you looking at a different angle?

As to what "sampling plan" was followed, that is up to the auditor. You can randomly select some, and 5 of 50 is fine. Or, I tend to select based on who I interviewed on the floor. Those that demonstrate some weakness are worth selecting their records.

 

[chaosweary]

Not a choice

The audit purpose is to ensure all processes that affect quality meet the intent of the ISO/TS 16949 Automotive Quality Technical Specification and the company's Quality Manual.
Part of my internal audit training is to verify all employees have had training according to their job responsibilities (that affect quality) and the training is logged in our training system. Auditing for training, corrective/preventive action, records, measurement & process improvement etc are all basics of every single audit I do, so I am not sure where your coming from. I don't work for QA that samples products I work for the corporate internal audit organization

The problem is that I cannot find any formal guidance on sample size other than for product audits. I know the boss is just challenging me to find a quantitative answer, but I have to say that I can't find a reference (staring at the ASQ quality audit handbook right in front of me). I wonder if I should throw in the towel and say, "I don't know, how many records should I sample?" and then say, "how did you come up with that sample size?" I think Andy is not understanding the intent. As an internal auditor I audit everything a third party auditor audits and much much more!

 

[RoxaneB]

There really is no set guideline on sample size. Some people like 10%. Some like more...some like less. What matters is that:

• The Auditor explains to the Auditee that sampling is not an exact science and is conducted in a finite period of time with finite resources;
• The Auditor has properly used the method of sampling to be confident is his/her audit conclusions; and,
• The sampling can be verified (i.e., records looked at are documented so that the evidence can be traced).

I agree with looking at the training records but for sampling and to show an audit trail, I usually leave it to the end of audit when the team has compiled a list of auditees (and positions). With HR we then look at the required training for those positions and verify that they are required to be trained on their job and that they have indeed been trained...that whole qualified and competent thing.

 

[Bill Pflanz]

When I was studying for my Certified Quality Auditor exam, I tried to find a sampling plan method for auditing. All the references to sampling were very vague and none used any formal sampling methods. The use of 5 samples appears to have its roots in financial auditing.

Dennis Arter is well known in the auditing field and in his book on quality audits he did provide some guidelines. He felt the sample size should depend on how sure you wanted your conclusion to be correct and how many occurrences there are. The problem is that you do not know how many defects there are. He then went on to say that 30 to 100 samples are sufficiently precise but also said that being statistically precise is not valuable if management doesn't understand it. He also said that concluding that you have a major problem based on 5 samples is not valid either due to its lack of preciseness. The practical issue with the number of samples that Arter suggested is getting the audit done in a reasonable time and cost.

The real response should have been that you were using a sample of 5 but it does not confirm either a problem or lack of problem. Just because you did not find one does not mean there was not a problem. If you did find one by taking just 5 samples, the manager should consider that it is more than an isolated incident and the process should be at least reviewed in more detail. Now if you find 3 or more problems with just 5 samples than everyone should agree that there is a likely problem.

Bill Pflanz

 

[ralphsulser]

Or, I tend to select based on who I interviewed on the floor. Those that demonstrate some weakness are worth selecting their records.

I think this is the better approach because you have met and interviewed the employees, and you have formed an opinion their competentcy, then look at the training records to follow the audit trail.

 

[Helmut Jilling]

.... As an internal auditor I audit everything a third party auditor audits and much much more!

absolutely!

 

[chaosweary]

Thanks!

The real response should have been that you were using a sample of 5 but it does not confirm either a problem or lack of problem. Just because you did not find one does not mean there was not a problem. If you did find one by taking just 5 samples, the manager should consider that it is more than an isolated incident and the process should be at least reviewed in more detail. Now if you find 3 or more problems with just 5 samples than everyone should agree that there is a likely problem.

Bill Pflanz

I think I will use this explanation (in my own words). I am glad there is a lot of subjective material out there on this one. It give some breathing room as to the explanation.

 

[Greg B]

Personally, unless you are trying to replicate a 3rd party audit, as in before registration/surveillance, I can't see why you'd (randomly) want to see training records:mg: What are you trying to establish?Andy

Andy, I cannot agree with this at ALL. Whilst I do not audit everything an external auditor would audit I think it is part of any full process audit that I establish that everyone is competent in their positions. I audit the Process not the Standard (I leave that to the 3rd party) and by process I mean All the Inputs and Outputs and everything involved in the process steps. I look from a business angle and look at waste, efficiency etc. Personally I don't care if the auditee is a bit put out by my randomness and I never set a figure or percentage.
In general, I don't agree that we require a formalised sampling number. We are not there to Pass or Fail as in a QC setting. We are there to ensure that the business processess are efficient and effective. We do this by obeservation, questioning and the the collection of facts.