Audit Sampling methods - Discovery Sampling Defined and Discussed

[Govind]

I came across with this topic called "Discovery Sampling" in the ASQ CQA Body of Knowledge in the past. Also I saw again today in another (financial) audit related article. This triggered few questions in my mind. I decided to put infront of this knowledgeable Quality community and ask for their feedback.

Discovery Sampling is defined as:
" a procedure for determining the sample size required to have a stipulated probability of observing at least one occurrence when the expected population deviation rate is at a designated level."

My questions are:
1. Do you sample more than one document, record in a given process while auditing?
2. Do you random sample documents and records based on " a scientific sampling Plan" ?
3. Have you ever used this "discovery Sampling" method for your Quality System auditing?
4. If anyone using this approach, any pros and cons? practical? cost effective?

Iam not aware of any registrar using this approach.

Since there is no thread on this subject from this forum, I would appreciate your feedback on this topic.

Thanks,
Govind.

 

[C Emmons]

ok, I will bite. I am in the transportation industry, and the internal auditor for 19 facilities. I have several different documents which are sampled and in some cases the same document is audited to check different processes. I have found that be setting my sample size as a "days worth" it allows me to get a good across the board sample. For example: Our drivers are required to turn in confirmation of their deliveries at the end of the day. A "Delivery Receipt". This receipt serves as a clear inspection by the customer, time of arrival, departure, pcs delivered etc. By sampling a days worth it allows me to view a sample of all drivers at a particular site and discover if there may be an issue related to just one specfic driver etc.

 

[Randy]

The ASQ dribble above is just that...dribble.

You need to base your sampling (we're talking ISO system auditing here) on being able to provide the objective evidence of conformance to whatever the applicable requirement is and not to identify a pre-concieved/pre-determined deviation. Good gosh...we should never go into an audit with a pre-disposition for finding the deviations, we look for conformance. We're not looking for "scientific sampling" (a horrendous process at best). Sampling has to take into consideration things like the size of the organization, the volume of documentation available, the time available, the scope of the audit and others. In the end adequate sampling is going to have to depend upon the "competence" of the auditor and the physical factors of the audit itself.

 

[The Taz!]

Thanx!

Thanx Randy. . . I was biting my tongue reading this thread. . .

and I think the right word is Drivel

 

[Rob Nix]

Arrrrrggghhhh!

You took the words right out of my mouth, Randy. I'll bet the framers of "discovery sampling" never did an hour of auditing in their lives. The expertise of the auditor is what determines how much documentation is audited. There are TOO MANY VARIABLES to establish "sampling" rules. Sigh!

Like Taz repeated, DRIVEL!

 

[RoxaneB]

Like Taz and Rob said, you took the words right out of my mouth, Randy! I had to read this twice and adjust the contact lenses to make sure I wasn't mis-reading.

1. Do you sample more than one document, record in a given process while auditing?

Yes and yes. To sample just one means nothing...depending on the item and time constraints, three to five samples of a document (or a record) in an area tends to be my norm.

2. Do you random sample documents and records based on " a scientific sampling Plan" ?

Please clarify "scientific sampling plan." I mean, it's not like I thow suction darts at the monitor to determine which documents/records to look at, but I also do not allow the auditee to select the documents/records for me to look at...I'm don't play the "pick the best and hide the rest" game with people.

3. Have you ever used this "discovery Sampling" method for your Quality System auditing?
4. If anyone using this approach, any pros and cons? practical? cost effective?

No, I have not used and if it is truly meant for an auditor to keep digging until they find evidence of a nonconformity, I will not use it.

Pro - Theoretically, I guess, over time there would never be any discrepancies in the system.
Con - Digging and digging and digging....doesn't make audits very auditee-friendly now, does it?
Pro - Ummm....
Con - Reactive instead of proactive
Pro - Still thinking...
Con - When does it end? What if you've been sampling documents for 3 hours and found nothing...do you keep going?
Pro - *gone to get a coffee*
Con - Okay...you've reviewed 763 documents. One of them had a mismatching Revision Number and corresponding hard copy. Now what? Do you honestly issue a finding 0.13% of nonconforming documentation? Is this a value-added activity to do so?

Practical - No...all of the time and resources focusing on finding errors instead of improvement.
Cost Effective - see above

Govind, I hope we don't sound like we're bashing the idea (okay...maybe we are saying our opinions much stronger than usual), but based on the definition of discovery sampling has us auditing to discover problems instead of auditing to discover ways of improvement.

Are we misinterpreting the defintion?

 

[J Oliphant]

what about intuition

I trust my intuition a little more than this.

why- Your purpose is to do the efficient survey of the organization's compliance to requirements. As an auditor, it always seemed that possible issues betrayed themselves in little unscientific (and sometimes unrecordable) observations. A book that is unorganized, things that aren't signed, things that the auditees didn't seem to bring up or remember when asked basic questions about procedures. one minor defect, can lead to another.

now with your sampling technique, you'd spend a non-value added time trying to discover if the issue is signficant rather than going from one issue to another trying to characterize the breakdown on quality.

The registrars auditors all seemed gifted at sifting for clues and part of there rigorousness was the fact that they dived beyond the initial offerings of procedure and manual that was undoubtably perfected by QA. That is to my mind their power.

Dennis arter called them "sniffs" and advocated that auditors spend and unscientifically large amount of time looking at documentation that may indicate the larger problem.

I think that sampling plans would then make auditors overall, less penetrating, enlightening and valuable.

Such a sampling plan might be more useful in analyzing your own dept.

 

[Al Dyer]

Auditors are supposed to be trained and certified. That said, auditors are people that need to have experience in some type of business setting. There are many here that have "general" business sense and know when they are being given a line of bologna and asked to swallow geese!!!

An auditor uses his/her accumulated knowledge to ascertain whether a company is following the system that "THEY" initiated.

In my 23 odd years I have only conflicted with 1 auditor, and he was not happy when he gave our company a "major" for an unrsesolved CA. He was upset because I brought up the fact that he was the person that closed the CA and then tried to give us the gig (in front of the entire management team).

Oh well,
Al...

 

[Govind]

It is very interesting to know the feedback from various professionals from their perspective. This is the right kind of interaction I was expecting while I posted in this forum.

Randy made a good point, of course, known to all auditors (as audit 101), auditing is a process of looking for conformance not non-conformance. C.Emmons offered his logic for sampling using “day” duration, which works effectively for their system. Roxane, Iam glad you took the time to answer point by point. You were constructive in making your point.
I agree with Taz and Rob emphasizing on the experience of the auditor.

My known application of Sampling to an audit in general:
Sarbanes- Oxley audit uses Sampling methods. Sample size is based on the number of transactions made during the given period. Acceptance number is 0. i.e even if there is one non-conformance, the entire transactions during that period is verified for compliance and corrective action is taken for the failed incidence. I guess since SOX is a regulation, this kind of strict procedure is fallowed.

I got back to this posting only few minutes back as I had to dig up some Audit reference materials I had during my CQA preparation.

There is a whole section in Audit handbook about “Audit Sampling”. The author acknowledges the fact that many auditors use Non-statistical sampling methods to keep the cost down. At the same time, some auditors say that Sampling is the way to get the attention from management.
Non-scientific or non-statistical sampling includes methods like “judgemental Sampling”, "Haphazard Sampling" and "Block Sampling". Statistical Sampling includes "Random Sampling" and "Systematic Sampling".
Without getting approval from the author, I will not be able to reproduce more information, but the reference would be Page 237 to 254 Audit Handbook. J.P Russell.
He summarizes that both methods have their place in Quality auditing, as mentioned by our fellow members, auditors should use appropriate method based on the information required.

Example: If the Senior Management wants to know how well a discrete process is working? "Processing of an order". By using scientifically valid sampling methods with adequate sample size, one can say that the order is correctly processed with X % of confidence. This catches management attention rather than saying out of 5 samples taken, there were 2 minor observations. (Similar reference Page 241).
Page 252 and 253 provides whole list of pros and cons.
Page 50 talks specifically about Discovery method as to this method to be used when we know that there is a problem but not located yet.

Next, Management Audit by Allan. J. Sayle suggests the application of Statistical Sampling for Compliance audits. In page 260, he provides guidelines when 6 to 10 documents were sampled (Non statistical). Also what to do for different scenario.

Next Quality Council of Indiana primer Page VII-27 whole chapter on Discovery Sampling with an example to process audit. The reference is dated 1955 from Nation convention transactions of ASQ!

So getting back to my own questions:

1. Do you sample more than one document, record in a given process while auditing?

Yes, we should definitely sample more than one. Thumb rule of 6 to 10 as a practice if no specific confidence level of a process need to be reported.
2. Do you random sample documents and records based on " a scientific sampling Plan" ?
Irrespective of whether the sample size is scientifically derived, we should take random sampling ( or equivalent methods without bias).
3. Have you ever used this "discovery Sampling" method for your Quality System auditing?
From what I see from the 8 postings, no one seem to use scientific sampling, leave alone discovery sampling C.Emmons method is similar to block or cluster sampling. I have not used statistical sampling in Auditing so far.
4. If anyone using this approach, any pros and cons? Practical? Cost effective?
JP Russell in his book provides limited pros and cons in Page 50.

I still welcome more replies on this thread.

Nutshell there is no direct answer to whether Sampling should be used in Quality auditing or not. As usual, the popular answer from the experts, “it depends”.

Regards,
Govind.

 

[Govind]

Please clarify "scientific sampling plan." I mean, it's not like I thow suction darts at the monitor to determine which documents/records to look at, but I also do not allow the auditee to select the documents/records for me to look at...I'm don't play the "pick the best and hide the rest" game with people.

I think this is a good practice for an auditor to pick the sample document, records by themselves instead asking the auditee to pick the documents, records of their choice.

When the auditor starts to develop trust on the auditee, they may do this occasionally. But auditor selection eliminates any bias. Sampling should also cover a range of "implementation period" and variables like different equipment, operator, shifts,etc.

Thanks for that good point Roxane.

Govind.