Audit Sampling methods - Discovery Sampling Defined and Discussed

Govind

Super Moderator
Staff member
Super Moderator
#1
I came across with this topic called "Discovery Sampling" in the ASQ CQA Body of Knowledge in the past. Also I saw again today in another (financial) audit related article. This triggered few questions in my mind. I decided to put infront of this knowledgeable Quality community and ask for their feedback.

Discovery Sampling is defined as:
" a procedure for determining the sample size required to have a stipulated probability of observing at least one occurrence when the expected population deviation rate is at a designated level."

My questions are:
1. Do you sample more than one document, record in a given process while auditing?
2. Do you random sample documents and records based on " a scientific sampling Plan" ?
3. Have you ever used this "discovery Sampling" method for your Quality System auditing?
4. If anyone using this approach, any pros and cons? practical? cost effective?

Iam not aware of any registrar using this approach.

Since there is no thread on this subject from this forum, I would appreciate your feedback on this topic.

Thanks,
Govind.
 
Elsmar Forum Sponsor
C

C Emmons

#2
ok, I will bite. I am in the transportation industry, and the internal auditor for 19 facilities. I have several different documents which are sampled and in some cases the same document is audited to check different processes. I have found that be setting my sample size as a "days worth" it allows me to get a good across the board sample. For example: Our drivers are required to turn in confirmation of their deliveries at the end of the day. A "Delivery Receipt". This receipt serves as a clear inspection by the customer, time of arrival, departure, pcs delivered etc. By sampling a days worth it allows me to view a sample of all drivers at a particular site and discover if there may be an issue related to just one specfic driver etc.
 

Randy

Super Moderator
#3
The ASQ dribble above is just that...dribble.

You need to base your sampling (we're talking ISO system auditing here) on being able to provide the objective evidence of conformance to whatever the applicable requirement is and not to identify a pre-concieved/pre-determined deviation. Good gosh...we should never go into an audit with a pre-disposition for finding the deviations, we look for conformance. We're not looking for "scientific sampling" (a horrendous process at best). Sampling has to take into consideration things like the size of the organization, the volume of documentation available, the time available, the scope of the audit and others. In the end adequate sampling is going to have to depend upon the "competence" of the auditor and the physical factors of the audit itself.
 
R

Rob Nix

#5
Arrrrrggghhhh! :frust:

You took the words right out of my mouth, Randy. I'll bet the framers of "discovery sampling" never did an hour of auditing in their lives. The expertise of the auditor is what determines how much documentation is audited. There are TOO MANY VARIABLES to establish "sampling" rules. Sigh!

Like Taz repeated, DRIVEL!
 

RoxaneB

Super Moderator
Super Moderator
#6
Like Taz and Rob said, you took the words right out of my mouth, Randy! I had to read this twice and adjust the contact lenses to make sure I wasn't mis-reading.

Govind said:
1. Do you sample more than one document, record in a given process while auditing?
Yes and yes. To sample just one means nothing...depending on the item and time constraints, three to five samples of a document (or a record) in an area tends to be my norm.

Govind said:
2. Do you random sample documents and records based on " a scientific sampling Plan" ?
Please clarify "scientific sampling plan." I mean, it's not like I thow suction darts at the monitor to determine which documents/records to look at, but I also do not allow the auditee to select the documents/records for me to look at...I'm don't play the "pick the best and hide the rest" game with people.

Govind said:
3. Have you ever used this "discovery Sampling" method for your Quality System auditing?
4. If anyone using this approach, any pros and cons? practical? cost effective?
No, I have not used and if it is truly meant for an auditor to keep digging until they find evidence of a nonconformity, I will not use it.

Pro - Theoretically, I guess, over time there would never be any discrepancies in the system.
Con - Digging and digging and digging....doesn't make audits very auditee-friendly now, does it?
Pro - Ummm....
Con - Reactive instead of proactive
Pro - Still thinking...
Con - When does it end? What if you've been sampling documents for 3 hours and found nothing...do you keep going?
Pro - *gone to get a coffee*
Con - Okay...you've reviewed 763 documents. One of them had a mismatching Revision Number and corresponding hard copy. Now what? Do you honestly issue a finding 0.13% of nonconforming documentation? Is this a value-added activity to do so?

Practical - No...all of the time and resources focusing on finding errors instead of improvement.
Cost Effective - see above

Govind, I hope we don't sound like we're bashing the idea (okay...maybe we are saying our opinions much stronger than usual), but based on the definition of discovery sampling has us auditing to discover problems instead of auditing to discover ways of improvement.

Are we misinterpreting the defintion?
 
J

J Oliphant

#7
what about intuition

I trust my intuition a little more than this.

why- Your purpose is to do the efficient survey of the organization's compliance to requirements. As an auditor, it always seemed that possible issues betrayed themselves in little unscientific (and sometimes unrecordable) observations. A book that is unorganized, things that aren't signed, things that the auditees didn't seem to bring up or remember when asked basic questions about procedures. one minor defect, can lead to another.

now with your sampling technique, you'd spend a non-value added time trying to discover if the issue is signficant rather than going from one issue to another trying to characterize the breakdown on quality.

The registrars auditors all seemed gifted at sifting for clues and part of there rigorousness was the fact that they dived beyond the initial offerings of procedure and manual that was undoubtably perfected by QA. That is to my mind their power.

Dennis arter called them "sniffs" and advocated that auditors spend and unscientifically large amount of time looking at documentation that may indicate the larger problem.

I think that sampling plans would then make auditors overall, less penetrating, enlightening and valuable.

Such a sampling plan might be more useful in analyzing your own dept.
 
A

Al Dyer

#8
Auditors are supposed to be trained and certified. That said, auditors are people that need to have experience in some type of business setting. There are many here that have "general" business sense and know when they are being given a line of bologna and asked to swallow geese!!!

An auditor uses his/her accumulated knowledge to ascertain whether a company is following the system that "THEY" initiated.

In my 23 odd years I have only conflicted with 1 auditor, and he was not happy when he gave our company a "major" for an unrsesolved CA. He was upset because I brought up the fact that he was the person that closed the CA and then tried to give us the gig (in front of the entire management team).

Oh well,
Al...
 

Govind

Super Moderator
Staff member
Super Moderator
#9
It is very interesting to know the feedback from various professionals from their perspective. This is the right kind of interaction I was expecting while I posted in this forum. :applause:

Randy made a good point, of course, known to all auditors (as audit 101), auditing is a process of looking for conformance not non-conformance. C.Emmons offered his logic for sampling using “day” duration, which works effectively for their system. Roxane, Iam glad you took the time to answer point by point. You were constructive in making your point. :agree1:
I agree with Taz and Rob emphasizing on the experience of the auditor.

My known application of Sampling to an audit in general:
Sarbanes- Oxley audit uses Sampling methods. Sample size is based on the number of transactions made during the given period. Acceptance number is 0. i.e even if there is one non-conformance, the entire transactions during that period is verified for compliance and corrective action is taken for the failed incidence. I guess since SOX is a regulation, this kind of strict procedure is fallowed.

I got back to this posting only few minutes back as I had to dig up some Audit reference materials I had during my CQA preparation.

There is a whole section in Audit handbook about “Audit Sampling”. The author acknowledges the fact that many auditors use Non-statistical sampling methods to keep the cost down. At the same time, some auditors say that Sampling is the way to get the attention from management.
Non-scientific or non-statistical sampling includes methods like “judgemental Sampling”, "Haphazard Sampling" and "Block Sampling". Statistical Sampling includes "Random Sampling" and "Systematic Sampling".
Without getting approval from the author, I will not be able to reproduce more information, but the reference would be Page 237 to 254 Audit Handbook. J.P Russell.
He summarizes that both methods have their place in Quality auditing, as mentioned by our fellow members, auditors should use appropriate method based on the information required.

Example: If the Senior Management wants to know how well a discrete process is working? "Processing of an order". By using scientifically valid sampling methods with adequate sample size, one can say that the order is correctly processed with X % of confidence. This catches management attention rather than saying out of 5 samples taken, there were 2 minor observations. (Similar reference Page 241).
Page 252 and 253 provides whole list of pros and cons.
Page 50 talks specifically about Discovery method as to this method to be used when we know that there is a problem but not located yet.

Next, Management Audit by Allan. J. Sayle suggests the application of Statistical Sampling for Compliance audits. In page 260, he provides guidelines when 6 to 10 documents were sampled (Non statistical). Also what to do for different scenario.

Next Quality Council of Indiana primer Page VII-27 whole chapter on Discovery Sampling with an example to process audit. The reference is dated 1955 from Nation convention transactions of ASQ!

So getting back to my own questions:

1. Do you sample more than one document, record in a given process while auditing?

Yes, we should definitely sample more than one. Thumb rule of 6 to 10 as a practice if no specific confidence level of a process need to be reported.
2. Do you random sample documents and records based on " a scientific sampling Plan" ?
Irrespective of whether the sample size is scientifically derived, we should take random sampling ( or equivalent methods without bias).
3. Have you ever used this "discovery Sampling" method for your Quality System auditing?
From what I see from the 8 postings, no one seem to use scientific sampling, leave alone discovery sampling:) C.Emmons method is similar to block or cluster sampling. I have not used statistical sampling in Auditing so far.
4. If anyone using this approach, any pros and cons? Practical? Cost effective?
JP Russell in his book provides limited pros and cons in Page 50.

I still welcome more replies on this thread.

Nutshell there is no direct answer to whether Sampling should be used in Quality auditing or not. As usual, the popular answer from the experts, “it depends”. :)

Regards,
Govind.
 
Last edited:

Govind

Super Moderator
Staff member
Super Moderator
#10
RCBeyette said:
Please clarify "scientific sampling plan." I mean, it's not like I thow suction darts at the monitor to determine which documents/records to look at, but I also do not allow the auditee to select the documents/records for me to look at...I'm don't play the "pick the best and hide the rest" game with people.
I think this is a good practice for an auditor to pick the sample document, records by themselves instead asking the auditee to pick the documents, records of their choice. :agree1:

When the auditor starts to develop trust on the auditee, they may do this occasionally. But auditor selection eliminates any bias. Sampling should also cover a range of "implementation period" and variables like different equipment, operator, shifts,etc.

Thanks for that good point Roxane.

Govind.
 
Thread starter Similar threads Forum Replies Date
F Product audit sampling plans IATF 16949 - Automotive Quality Systems Standard 3
T ISO 13485: 2016 Internal Audit - Is sampling on projects allowed? ISO 13485:2016 - Medical Device Quality Management Systems 6
P Random Sampling vs. Thorough Review - Records to be Reviewed - Audit Finding Inspection, Prints (Drawings), Testing, Sampling and Related Topics 13
C Internal Audit Sampling Plans - Determining Internal Audit Sample Size Internal Auditing 45
E Internal Process Audit Sampling using MIL-STD-105-E - 500 operators on 3 shifts Inspection, Prints (Drawings), Testing, Sampling and Related Topics 4
Q ISO 9001-2015 Internal audit finding Internal Auditing 12
lanley liao How to understand this words that the planning of internal audit shall take into consideration the results of previous audits? Oil and Gas Industry Standards and Regulations 10
P Audit check for IT company (ISO 9001) ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 7
D Supplier audit Medical Device and FDA Regulations and Standards News 2
M Go Live With New ERP System before Recertification Audit General Auditing Discussions 6
A Add MDSAP to Internal Audit Schedule Medical Device Related Regulations 0
A Define timeline for Major and Miner Audit finding General Auditing Discussions 4
J IATF 16949 Internal Audit question - Auditor's responsibility Internal Auditing 6
A API Monogram audit review process Oil and Gas Industry Standards and Regulations 4
S IATF 16949 Internal Audit Example IATF 16949 - Automotive Quality Systems Standard 7
B Remote IATF 16949 audit preparation General Auditing Discussions 10
M AS9100D Registrar pre-audit requirements AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 15
I Do I need to sign off my annual audit calendar? Internal Auditing 2
R AS9100D internal audit checklist or ISO 9001 2015 to AS9100 D AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 2
M Nice and simple invitation email to an audit kickoff meeting Internal Auditing 1
M IATF 16949 - Audit of Remote Location/Support Site and IT IATF 16949 - Automotive Quality Systems Standard 4
Q IATF audit - Root Cause Analysis results IATF 16949 - Automotive Quality Systems Standard 5
Q Self-assessment audit information Quality Management System (QMS) Manuals 6
Le Chiffre Online training available for ISO/IEC 17021-1: Requirements for bodies providing audit and certification of management systems Training - Internal, External, Online and Distance Learning 3
xfngrs NIOSH Audit for N95 respirators US Food and Drug Administration (FDA) 1
Sidney Vianna IATF 16949 News Risked Based Audit Day Calculation IATF 16949 - Automotive Quality Systems Standard 2
T AS9100 audit due to facility move AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 4
M ISO 13485:2016 internal audit checklist Medical Device and FDA Regulations and Standards News 5
A Internal Audit Questions ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 11
B SAP Audit trail Periodic Review EU Medical Device Regulations 1
N What are the software audit and control steps Reliability Analysis - Predictions, Testing and Standards 2
R Gap Audit Aerospsace and Rail QMS Quality Manager and Management Related Issues 0
salaheddine96 Internal audit planning Internal Auditing 2
A MDSAP Audit Questionnaire Medical Device and FDA Regulations and Standards News 7
H Obligations as a contract manufacturer during an MDSAP audit ISO 13485:2016 - Medical Device Quality Management Systems 5
M What are the basics of Medical Device Single Audit Program (MDSAP)? ISO 13485:2016 - Medical Device Quality Management Systems 7
Sidney Vianna IATF 16949 News Update on the IATF CARA Project (“Common Audit Report Application”) - 12/2020 IATF 16949 - Automotive Quality Systems Standard 1
R ISO 17025 vertical audit checklist wanted Document Control Systems, Procedures, Forms and Templates 2
Z Rapid audit template for plastic parts manufacturing process Manufacturing and Related Processes 12
M ISO 9001 Major Nonconformance Internal Audit Schedule/COVID-19 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 18
S Quality Audit Training Activities Quality Manager and Management Related Issues 2
G IATF Rules for COVID 5th revision - Re-certification audit timing IATF 16949 - Automotive Quality Systems Standard 3
E MDR internal audit Internal Auditing 1
Ed Panek Remote Audit GOTOMEETING thoughts Coffee Break and Water Cooler Discussions 22
B ISO 9001 - "Remote Audit Fee" Registrars and Notified Bodies 13
L IATF external audit virtual (remote) IATF 16949 - Automotive Quality Systems Standard 13
M Audit Criteria Training Materials Internal Auditing 1
K New supplier audit as per V3.1 by French Automotive OEM General Auditing Discussions 2
S Complexity Rating - CB adding another audit day for "high complexity" AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 6
U Internal Auditor not trained but done Audit for some process Nonconformance and Corrective Action 5

Similar threads

Top Bottom