Okay, this one is bugging me!

Our Quality Manual has a statement under internal audit procedures:

Quality Manager and/or designee are to review a statistically sound sample of documents and records within the scope of the process audited to ensure compliance to the standard.

It further requires that the sampling plan be detailed. We are not conforming to our own requirement on this, nor have we been for some time.

Is there anyone that follows a similar rule? Could someone give some advice on how to prove that the sampling plan is sufficient or even what type of sampling plan should be employed? We have no definition of what type of sampling plan will be applied...I am not expert in statistics, I get by with SPC, Capability, and such.

Any advice is appreciated. :frust:

Ben:
Remove the statement from the procedure! There is no effective method for (statistically) sampling documentation. Indeed, if this is the methodology for doing audits, then the focus is incorrect, ineffective and based on external audit techniques. Internal audits shouldn't being done this way.

The only suitable method is to take a look at contemporary records etc. This will give management a view on the system as currently practiced, not something which happened (randomly) some time ago.

Furthermore, if the procedures are telling the auditor to do this, then perhaps looking onto their training is required too. Techniques such as this was how auditors were trained way back in the past. Has anyone brought their practices up to date?

Andy

Our quality manual does not specify such things, as they are explained/described in the actual procedure for internal audits. Perhaps a revision to the quality manual might be considered? I would suggest either a change to the quality manual to reflect current practice or revise the current practice to comply with the quality manual. :2cents:

Can't help but wonder, though, why would a company write the quality manual in such a prescriptive manner?

From the perspective of both auditor and auditee, I have seen more valid findings from a few samples that dig deep (especially into the handoff between departments) than I have seen from a large number of samples that go shallow.

I agree with the earlier comment about removing this statement from the manual. A 3rd-party auditor will be more concerned with the effectiveness of your audits than in their statistical validity. That is, are you finding the weaknesses in the system? If you never find an issue, is your auditing effective? No system is perfect.

Ben:
Remove the statement from the procedure!
I think this is probably good advice, but when you say,
There is no effective method for (statistically) sampling documentation.
I have to disagree, and can even envision a situation where it might be helpful, but only after an audit. For example, say that there is a requirement for some record to be signed by a particular person, and the audit reveals a single instance of an unsigned document. Furthermore, there are many such records, and there is a need to know whether the single example found in the audit is an outlier, or evidence of a systemic problem. In such an instance, a rational sampling plan would be in order, perhaps.

Actually GStough you bring up a very good point and one that we're working to remedy. The Quality Manual has become quite archaic, and rather than describe procedures for key processes it has been used, in the past, as a kind of collection of the procedures themselves. Our goal is to break these out into appropriate operational procedures, tier 2, and simplify the QM into a more concise document.

I'm thinking the section on sampling was not intended to mean an exact statistical technique should be applied, but rather that an appropriate sampling shall be performed. I wanted to see, however, if anyone did indeed utilize statistical analysis to validate the number of documents reviewed in internal audits.

Andy, I agree, and yes, someone is looking into the procedure, the Quality Manager and I. I was brought on board here as an internal auditor about 7 months ago and have found many of the procedures need to be revised or completely rewritten before any real results can be obtained from internal auditing, and so, we are pursuing that task at a slow and often frustrating pace.

Quality Manager and/or designee are to review a statistically sound sample of documents and records within the scope of the process audited to ensure compliance to the standard.

If you want to do this, there are three parts. First is to identify the entire population of what you mean by "documents and records". And then each item in the population must receive a unique indentifier.

Second, you must decide how big of a sample you want to take at what rate. For example, you may choose to sample two documents and records per week.

Third, you need to decide are you going to do a one pass random sample (review the records in random order, but remove a record from consideration from being sampled again until the entire population is been sampled once) or a pure random sample where every item has an equally likely chance of being pulled in each sample, even if it was just sampled last time.

I find the best way to do this is to list all of the unique identifiers in a column in Excel, assign a random number in the next column corresponding to each member of the population, then flip a coin. Heads - sort the records ascending, Tails- descending, then take the top records from the sort until you get the sample size you decided upon.

I have followed these posts, and they speak much wisdom. However, I still have a dumb question on my part....

If as an auditor, you don't utilize a "normal" sampling plan (confidence level, sample size, etc.), what reasonable approach do you take towards sampling?

If there are about 1200 instruments, show me the records on these 3 (5, 11, 1, etc.) pieces of equipment.

Do auditors develop a rule of thumb, internal auditing practice guidelines, gauge it by how much time, etc.? Sometimes internal auditors are less-experienced, and will not have the experience/knowledge that you guys have.

I simply don't understand what you would embrace for consistency and representation if you didn't utilize a plan. I totally agree that the Statistical validity is substantially less important than the integrity/quality of the audit. However, # of samples could yield the difference between a one-time problem, and a more serious problem, due to it's repeatability.

Again, I'm not in disagreement; I just need some guidance.

Again, I'm not in disagreement; I just need some guidance.

Probably the best consideration is what do you want for an average cycle time for the population to be looked at completely? Once a year? Once a decade? The rate you want to cover the population is related to the relative risk. If there was an undetected problem, how long could you go without having some more significant failure? And that must be balanced with the time and resources it takes to do the audits.

Such good responses so far.

When planning a sampling plan for assessing document control, you can ask:

1. What is the desired achievement--what degree and type of control is important?

2. What problem, if any, should be monitored?
2a. Are there any groups that should get special attention?

3. What's the population?

4. What are my resources--can I meet these expectations?

Two more thoughts occur to me.

1. The word Statistical is often misunderstood to think it always means an x-bar and r or equivalent type of control monitor.

2. A person may act on such a misunderstanding, and expect some elegant sampling and measurement system when a histogram/Pareto chart would do.

So, take care what you promise in those specs. Be ready to show, if and when asked, how a sampling scheme is decided, the response (plans and actual when it exists) if results are unsatisfactory, and a followup to see if it has worked. Nowhere do we see a requirement to lay all this stuff out in a quality manual or process document. It's a process/system auditing approach.

I have followed these posts, and they speak much wisdom. However, I still have a dumb question on my part....

If as an auditor, you don't utilize a "normal" sampling plan (confidence level, sample size, etc.), what reasonable approach do you take towards sampling?

If there are about 1200 instruments,

Do auditors develop a rule of thumb, internal auditing practice guidelines, gauge it by how much time, etc.? Sometimes internal auditors are less-experienced, and will not have the experience/knowledge that you guys have.

I simply don't understand what you would embrace for consistency and representation if you didn't utilize a plan. I totally agree that the Statistical validity is substantially less important than the integrity/quality of the audit. However, # of samples could yield the difference between a one-time problem, and a more serious problem, due to it's repeatability.

Again, I'm not in disagreement; I just need some guidance.This isn't a dumb question at all.

Instead it shows a potential opportunity in the audit program. How are audits being designed to assess performance and how are the results being used?

It's different from compliance. It requires someone to pay attention to trended data and notice if there's a problem spot, and to support a suspicion that imrovement is in order. Who does that?

There are two approaches:

1. A system-wide document/records audit

2. Sampling during each of the other audits.

#1 may make comparison easier, but it could be superficial.
#2 could provide the deep look needed to uncover opportunities (not just nonconformances) in areas, but comparing them could be messy.

If you opt for #2, consider using a spreadsheet that gathers scores across the system. Have auditors look for the same types of aspects as they go out on regular audits and enter the results as audits are completed. The results can be tracked over time for improvement, and "hot spots" could be more easily detected. Also, if a study is made of what kinds of documents/records are being audited, gaps or unneeded clumping can be observed and corrected.

I'm developing a tool to score FMEAs. When I'm done I will post it as an attachment.

This thread is fascinating. The subject of sampling crops up time and time again (if not here, certainly in training courses etc.)

There is no doubt that it is very difficult to define what qualifies as a 'reasonable' sample of whatever the auditor is looking for. A balance has to be struck between taking too few and, therefore, being superficial and wasting time (and possibly alienating the auditee) by looking at too many samples.

From my previous post, my assertion is that 'sampling' in the way described in the OP, is an out-moded audit tactic, based firmly on external auditing practices. Internal auditors shouldn't be emulating the same tactics - and certainly not arriving at samples by randomly selecting them!

What keeps coming to mind is why is the auditor doing the audit? Is it to determine if the system/process etc. is in compliance? Is it to determine if an action has been effectively taken to correct some issue?

By having a clear audit task, (we can call this the 'scope' and 'criteria' to be technically correct) the sample size becomes easier to determine. The sample is going to be determine by the circumstances which surround the implementation of the 'focus' of the audit - compliance, corrective action etc.

In many cases, this is time based, so the samples be based on the evidence which can be generated in the time taken to implement the system/process/corrective action etc. Of course, this may mean, just looking at one sample!! After all, it is what it is!

It all comes back to the basics - why are we auditing?

Andy

It all comes back to the basics - why are we auditing?
Andy

We should be auditing in order to verify that the system is in compliance with the requirements, and not to find problems. If, however, evidence is found that the system might not be in compliance, rational methods should be used to determine the facts. A single nonconformity might be evidence, but it's not conclusive evidence without rational data to support the contention.

Do auditors develop a rule of thumb, internal auditing practice guidelines, gauge it by how much time, etc.? Sometimes internal auditors are less-experienced, and will not have the experience/knowledge that you guys have.Indeed, excellent discussion. When I teach auditing classes, I emphasize the aspect that the sample used by the auditor to reach a conclusion must be representative of the process/activity being assessed. That should include:

effectivity date of the command media (if any) controlling the process. I mean, if a procedure was effected June 1st, 2006, you should not expect that records and evidence before that date to be in line with the procedure at hand.
criticality & risk of items being assessed: since you do have time constraints to perform the audit, focus on issues of significance. For example, if you are assessing if your purchase orders are clearly identifying the requirements associated with the product being bought, focus on critical products and don't waste your time if the toilet paper was clearly identified.
Like Jim said, if you do find a discrepancy, you need to consider expanding your sample to determine if the problem is systemic or something that it was totally isolated. This has an implication about your decision making process, as an auditor, how that would be reported. For example, if I am auditing a customer service call center, which receives 2,500 calls a day, and I notice one single "ticket/case" not adequately entered in the database, it would be ludicrous to expect true corrective action in this case.
Especially for third party audits, I strongly suggest the auditor to let the auditee know BEFORE s/he starts digging, what would be the approximate sample size s/he will be basing his/her conclusion on. For example, if I am assessing the accuracy of travelers used for production planning, I would tell the Manufacturing Engineer something like: I will "randomly" look at 15 of these travelers. The reason for that is so I don't give the auditee a wrong impression that I am digging for problems until I found something to write up.
Don't ignore what you know, hear and see. If I am auditing for preventive maintenance and I go to the shop floor, and I see a CNC machine which seems to be in disarray, albeit operational, why should I ignore that situation? I would definitely focus on that machine to see what is going on. Yes, I might be biased by the visual clues, but the goal of auditing is to see if the system is operational and effective. If I have (what seems to be) a clear sign of system failure, I should not ignore it.In my view point, there is no scientific/statistically valid method for sample size determination when looking for evidence in management system audits. But as I said before, the sample should be representative. This is an area that brings tremendous variation in auditor performance. Some auditors are extremely thorough, and sometimes, mis-perceived as nit-pickers. Other auditors are very superficial, reaching conclusions after examination of one or two pieces of evidence. In the Aerospace ICOP world, we have a constant battle to improve the auditor skills for sampling determination.

In answer to Brad's question, I believe that, over time, good auditors develop their skills concerning sampling determination. In my opinion, it will always be a judgement call.

Thanks to all for your added insight/wisdom. It really is beneficial having a source to pull different viewpoints and perspectives on things. Especially those "But we've always done it that way" items.