Definition Performance Based Auditing - What is the definition?

D

dashmen

#1
Hi everyone. It's my first post . I visit this forum almost every day for half a year and I think the forum is thoughtful and a great place to understand "quality". :read:
Now my question is, what's your understand of the "performance based auditing"?
 
Elsmar Forum Sponsor
W

WALLACE

#2
Welcome Dashmen,
You'll find lots of invaluable resources at the Cove.
Dashmen, what is your definition of a performance based audit?
Your answer shall allow the Covers to get a baseline on your understandings and interpretations.
Wallace.
 
D

dashmen

#4
Jennifer, I am studying the article. Thanks for your links.

Wallace, the situation is, one customer ask us have an audit on one of their suppliers, they give the checklist and ask that the audit must be a "performance based audit". I reviewed the checklist and feel that seems like the "process audit" . So I want some opinion and can ready for the audit.
 

Jen Kirley

Quality and Auditing Expert
Staff member
Admin
#5
dashmen said:
Jennifer, I am studying the article. Thanks for your links.

Wallace, the situation is, one customer ask us have an audit on one of their suppliers, they give the checklist and ask that the audit must be a "performance based audit". I reviewed the checklist and feel that seems like the "process audit" . So I want some opinion and can ready for the audit.
This is as I suspected.

One of the criticisms I have read of current registration standards (both ISO and the specialty standards that resemble its structure) is that audits of systems per these standards are based on evidence that the system is being followed. How effective and with what value are not questions for the auditor--the auditee is only responsible to show the system is functioning within specified parameters.

This has invited soul searching after problems such as the Ford-Bridgestone debacle, where tires blew on Explorer vehicles under certain condisions and people asked, "Why wasn't this caught before people died from it? Why didn't the audit process catch this problem?"

The basic answer is that registrars donot have the right to demand that performance data be included. Test results, design changes that result, etc. are considered sensitive, confidential data and can be kept secret.

Industry regulators have been the standard answer for this need for oversight, but they may be too few, thus cannot provide the kind of coverage necessary for effective oversight.

Those who favor less government intervention assert that free market forces can fill the gap; let flawed suppliers be exposed by their customers, and either be forced to change or suffer from shrinking market share. In other words, let the customer's demand for quality rule over practices, not just enforcement of law.

The performance audit seems to be an attempt to meet this objective.
However, you still wouldn't have the powers of a regulatory auditor. The majority of references I found, including the second link, involves government audits. Even when done internally, government entities would likely find it much easier to exert authority in the audit process than the free market auditor could expect.

If you feel the checklist does not address performance, you may want to negotiate that because it would still be very easy to hide poor performance behind graphs and charts if you are not allowed to dig for data that supports their claims of effectiveness. Your customer may or may not know all this, and the supplier may also understand it, or not.

It may be up to you to decide how approach drawing the parameters of this audit, negotiate the extent you will be held responsible for its results, and ensure the auditee will cooperate by supplying the data that shows performance, and not just that they are following the system.

Anyone else? I am ready to hear if I am on the mark or not, or if I am missing something.
 
D

D.Scott

#6
Jennifer - I'm not sure where you are coming from when you say auditors can't verify the effectiveness of the process.

ISO 19011:2002 (applicable to all organizations needing to conduct internal or external audits of quality and/or environmental management systems) says in the introduction: "verifying the effective implementation of an organization's quality and/or environmental policy".

16949 states: "..... and maintain a quality management system and continually improve its effectiveness ....". How can an auditor determine compliance to this requirement if they are not allowed to verify that effectiveness?

Am I missing something or were you meaning something else?

Dave
 

Jen Kirley

Quality and Auditing Expert
Staff member
Admin
#8
D.Scott said:
Jennifer - I'm not sure where you are coming from when you say auditors can't verify the effectiveness of the process.

ISO 19011:2002 (applicable to all organizations needing to conduct internal or external audits of quality and/or environmental management systems) says in the introduction: "verifying the effective implementation of an organization's quality and/or environmental policy".

16949 states: "..... and maintain a quality management system and continually improve its effectiveness ....". How can an auditor determine compliance to this requirement if they are not allowed to verify that effectiveness?

Am I missing something or were you meaning something else?

Dave
My understanding is that in performance audits, the auditor can look at performance data and determine the system's effectiveness.

The complaints I have read are that auditors normally may not be able to see actual data like regulatory and accounting auditors do. Such data can be claimed off-limits due to competetive sensitivity. This has been the excuse of why the Bridgestone/Ford tire problems went on as long as they did.

I suspect it is why there is such lingering cynicism regarding ISO...is the standard really doing all it can to ensure quality? Many have argued it is not.

It has, according to the complaints, required auditors to be satisfied with the auditees' own assessments, including testing, internal audits, customer feedback and so on. Activities based on results of those assessments, including CARs, continuous improvement and design changes can be promoted as "proof" that the system is effective. But these show the system is functioning, not its effectiveness or value or performance (as I use the term "performance", that is). It does not "prove" quality, just raises the comfort levels.

The bulk of the sources I found regarded government entities. I am not sure how much the performance audit is being used in private or publicly owned companies.

Now I could be wrong about all this, but I base my comments more on long-term reading than my own, frankly more limited exposure to the practice. If I am wrong, I'd like to find out because I am here to learn too!
 
H

haifeng

#9
Jennifer Kirley said:
My understanding is that in performance audits, the auditor can look at performance data and determine the system's effectiveness.

The complaints I have read are that auditors normally may not be able to see actual data like regulatory and accounting auditors do. Such data can be claimed off-limits due to competetive sensitivity. This has been the excuse of why the Bridgestone/Ford tire problems went on as long as they did.

I suspect it is why there is such lingering cynicism regarding ISO...is the standard really doing all it can to ensure quality? Many have argued it is not.

It has, according to the complaints, required auditors to be satisfied with the auditees' own assessments, including testing, internal audits, customer feedback and so on. Activities based on results of those assessments, including CARs, continuous improvement and design changes can be promoted as "proof" that the system is effective. But these show the system is functioning, not its effectiveness or value or performance (as I use the term "performance", that is). It does not "prove" quality, just raises the comfort levels.

The bulk of the sources I found regarded government entities. I am not sure how much the performance audit is being used in private or publicly owned companies.

Now I could be wrong about all this, but I base my comments more on long-term reading than my own, frankly more limited exposure to the practice. If I am wrong, I'd like to find out because I am here to learn too!
I agree with ur idea. That's situation is very popular in China. Most of the auditor conduct the audit according to the standard requirements, although according to 2000 EDITION, the audit should be conducted along the process of the auditee, but the actural performance datas are very difficult to confirm and verify. So, I think, in the future, a kind of combined audit will appear that's combine with the accounting audit and the quality system audit and other related audits. It's called by someone "the internal auditor."
 
D

dashmen

#10
Thanks all your replies. :applause:

I am thinking this issue: Does effective process=>effective outcome? or Does effective outcome=>effective process?
And how we define performance within an quality audit? Does it mean product performance or system performance or process performance? Maybe all of them?

This article (http://www.nalga.org/qrtly/art97d1.html ,thanks for Jennifer's link,I dig into it) gave me some inspiration,especially the last sentence:"the procedures (conversion activities) are only the piece of the equation that facilitates the achievement of management's goals."

More opinion are welcome.
 
Thread starter Similar threads Forum Replies Date
M Informational Several US FDA draft guidances, including some specific device guidances for the Safety and Performance Based Pathway Medical Device and FDA Regulations and Standards News 0
M Informational USFDA final guidance – Safety and Performance Based Pathway Medical Device and FDA Regulations and Standards News 0
M Informational USFDA Final Guidance – Safety and Performance Based Pathway Medical Device and FDA Regulations and Standards News 2
M FDA News USFDA – Safety and Performance Based Pathway Guidance Medical Device and FDA Regulations and Standards News 0
D QMS Performance Based on how many NCRs were generated month to month Nonconformance and Corrective Action 8
P Measurement of Performance based on Internal Audit Reports Quality Tools, Improvement and Analysis 12
Q Time Based, Measurable Mid-Year Performance Goals Quality Manager and Management Related Issues 3
Z Consequences for Poor Performance based on a Monthly Scorecard - Actions? Supplier Quality Assurance and other Supplier Issues 15
P Definition Dock-to-Stock Process based on Historical Quality Performance of a Part and Supplier Definitions, Acronyms, Abbreviations and Interpretations Listed Alphabetically 8
C QMS Performance Trends - Based on fixed or flexible periods - Clause 5.6.1.1 IATF 16949 - Automotive Quality Systems Standard 6
D What is 'sampling based on performance'? 7.4.3.1 Incoming Product Quality Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
D Essential performance and EMC immunity testing IEC 60601 - Medical Electrical Equipment Safety Standards Series 4
G What does performance specification include? US Food and Drug Administration (FDA) 1
B Handling lower detection limits for SPC and process performance Statistical Analysis Tools, Techniques and SPC 1
M IVDR and Performance Evaluation Plan CE Marking (Conformité Européene) / CB Scheme 2
C Where to draw the line for "sufficient evidence" to verify safety/performance of a device? CE Marking (Conformité Européene) / CB Scheme 2
K IEC 62304 - Functional and performance requirements for SOUP items IEC 62304 - Medical Device Software Life Cycle Processes 2
K EU MDR Annex 1 Chapter III: Information in the Instructions for Use-23.4 (e) the performance characteristics of the device; EU Medical Device Regulations 1
Q ISO 9001 8.5.1 - Control of production and service performance ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
W IEC 60601 - Essential performance c.2.34 IEC 60601 - Medical Electrical Equipment Safety Standards Series 2
S A clinical performance evaluation study with an IVD product as Investagional Use product - Clinical Monitor requirements 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 4
D Performance of high shear mixer (or rapid mixing granulator Qualification and Validation (including 21 CFR Part 11) 4
M Do you need an Applicable general safety and performance requirements Checklist? EU Medical Device Regulations 2
D Performance specification as a Risk Control Measure, EN 14971 ISO 14971 - Medical Device Risk Management 7
G Defining performance metrics for DFMA implementation Design and Development of Products and Processes 2
B Four Key Performance Indicators for Continual Improvement (Dec. 3 2019) [Deleted] Training - Internal, External, Online and Distance Learning 2
F 5520A High Performance Multi-Product Calibrators General Measurement Device and Calibration Topics 0
shimonv IEC 60601-1 Essential Performance - Is the signal accuracy specification an essential requirement? IEC 60601 - Medical Electrical Equipment Safety Standards Series 4
D Performance Qualification per GHTF Guidance Other Medical Device Related Standards 12
M Informational US FDA Final Guidance – Coronary, Peripheral, and Neurovascular Guidewires – Performance Tests and Recommended Labeling Medical Device and FDA Regulations and Standards News 0
O Performance Measurement ISO 9001: 2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
O Monitoring performance - How do I determine performance measurement basis within my organization? Misc. Quality Assurance and Business Systems Related Topics 4
M Informational MDCG 2019-9 Summary of safety and clinical performance A guide for manufacturers and notified bodies – August 2019 Medical Device and FDA Regulations and Standards News 0
H How to prepare Performance Qualification (PQ) for autoclave General Measurement Device and Calibration Topics 2
M Performance Standard for Balloon Trocars Other Medical Device Related Standards 2
M Informational TGA Consultation: Proposed changes to medical device essential principles for safety and performance Medical Device and FDA Regulations and Standards News 0
rezayatmand IEC 60601-2-18 Medical electrical equipment - Part 2-18: Particular requirements for the basic safety and essential performance of endoscopic equipmen IEC 60601 - Medical Electrical Equipment Safety Standards Series 2
qualprod Ineffective follow up of people performance - Audit Nonconformance ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
P Is there a counterpart to the General Safety and Performance Regulations for the USA? Other US Medical Device Regulations 2
D Summary of safety and clinical performance in GSPR MDR EU Medical Device Regulations 2
M Informational EU draft act – Single-use medical devices – safety and performance requirements for reprocessing Medical Device and FDA Regulations and Standards News 0
B Essential Performance of a Cone Beam Computed Tomography Device IEC 60601 - Medical Electrical Equipment Safety Standards Series 3
M Informational USFDA final guidance – Recommended Content and Format of Non-Clinical Bench Performance Testing Information in Premarket Submissions Medical Device and FDA Regulations and Standards News 0
M Informational USFDA draft guidance – Technical Performance Assessment of Quantitative Imaging in Device Premarket Submissions Medical Device and FDA Regulations and Standards News 0
M Informational USFDA – Radiological Health Regulations; Amendments to Records and Reports for Radiation Emitting Electronic Products; Amendments to Performance Stand Medical Device and FDA Regulations and Standards News 0
M Characterization Testing - NO acceptance criteria, no minimum performance requirement Inspection, Prints (Drawings), Testing, Sampling and Related Topics 1
D IEC 60601-1 - Essential performance doesn't make sense IEC 60601 - Medical Electrical Equipment Safety Standards Series 10
M Product Performance Claims Other Medical Device Related Standards 4
P Performance evaluation (IVDD Annex VIII) & Design and Development Validation Studies EU Medical Device Regulations 1
C How frequent to conduct Clinical / Performance Evaluation? CE Marking (Conformité Européene) / CB Scheme 4

Similar threads

Top Bottom