Are KPIs a requirement to determine Process Effectiveness?

Buckyb

Involved In Discussions
#1
I have a concern that I want to share with the forum. There is a potential problem brewing in aerospace auditing that subscribes to the notion that KPIs must be developed by organizations in order for the auditor to evaluate the effectiveness of the process being assessed. To not have KPIs or inadequate KPIs automatically results in "process not effective".

There are some (names and CBs withheld) that interpret AS9101D 4.1.2.5f to mean measurement is required. I would argue that the whole paragraph must be taken in context, not just item f. 4.1.2.5 states,"The audit team should evaluate, as appropriate, that processes:" are or have what is described in items a through h. There is no shall and "as appropriate" means just that. 4.1.2.5f can't be taken to mean measure as the only means as that is not the intent of the paragraph.

AS9100C Clause 4.1e. states "monitor and measure where applicable" and AS9101D states in Note 1 that KPIs are often used. There is no "shall" in these narratives either.

Also, AS9100 Clause 8.2.3 states that "the organization shall apply suitable methods for monitoring and, where applicable, measurement of the quality management system processes." The key is "where applicable"; it doesn't read monitor and measure.

While it is true that many organizations develop KPIs they are not required by AS9100. In fact AS9100 does not have "Key Process Indicators or KPI" anywhere in the document. AS9101D states KPIs twice; once in 4.1.2.5 Note 1 and on the PEAR Form Instructions item 11. Nowhere in any Standard does it state that KPIs are required to evaluate effectiveness but rather they are simply one of the means to measure effectiveness. KPIs aren't and can't be developed for every process and the choice to develop and use them must rest with organizational rationale and top management decisions. The Auditor's role must be to assess the process and determine its effectiveness; KPIs are only one means to do so.

I am not on this soapbox advocating no KPIs. I believe there has to be a mix of methods to measure process effectiveness and it can't be only KPIs. I hear rumors that this guidance/direction is coming from above in spite of what is in print in the Standards. If this is true it is a sad day for those who worked so hard to provide us Standards that we in the trenches try to live by.
 
Elsmar Forum Sponsor

Jen Kirley

Quality and Auditing Expert
Staff member
Admin
#3
Re: Are KPIs a requirement to determine Process Effectiveness ?

I had thought that KPI means "Key Process Indicator," which means a measurement of something important. Maybe I'm naive but I thought it was just an acronym for what the standard is asking us for: a means to know how well things are going before the customer does. No need to look for the acronym in the standard, just develop a measure of what's important.
 

Big Jim

Super Moderator
#4
Re: Are KPIs a requirement to determine Process Effectiveness ?

I had thought that KPI means "Key Process Indicator," which means a measurement of something important. Maybe I'm naive but I thought it was just an acronym for what the standard is asking us for: a means to know how well things are going before the customer does. No need to look for the acronym in the standard, just develop a measure of what's important.
Actually KPI means Key Performance Indicator. When the CBs, and I assume those above the CBs, started pushing hard on the process approach in early 2005, many automatically substituted Process for Performance, presumably because that is what they were being taught to use them for. This came from the business world, and evolved from MBOs (Management by Objectives).
 

Big Jim

Super Moderator
#5
Re: Are KPIs a requirement to determine Process Effectiveness ?

Although it is true that nowhere in the AS9100C standard (or the ISO 9001:2008 standard that it is derived from), are Key Performance Indicators mentioned by name.

It is there is spirit though in several places. It is in 4.1 where you are told to determine the methods to ensure control of the processes, where you are told to monitor them, and where applicable, measure them.

It is in 5.4.1 where it tells you that you need to have quality objectives, that they need to be measurable, and that they need to be established at relevant functions and levels throughout the organization. What do you think those relevant functions and levels might be? Processes maybe?

It is in 8.2.3 where it specifically says that you will monitor, and measure when applicable, your processes. This is somewhat of a repeat of 4.1, including the part about taking correction and corrective action as appropriate.

And here is the really big one. Where it tells you what topics you need to have measurements for (no where appropriate here). It is in 8.4 where it tells you that you need to determine, collect, and analyze data that helps you determine the health of your quality management system and evaluate where improvements can be made. It goes on to tell you four specifics topics to apply this analysis of data to.

I have no problem with companies finding other ways to monitor their processes. The problem is that most that don't use KPI/quality objectives to monitor their processes don't have another way, and simply whine about KPI/quality objectives don't need to be the way it is done.

Field 11 on the PEAR used by AS9101D asks for the auditor to describe how the auditee monitors that process. It suggests it may be KPI/quality objectives, but doesn't require it. That is the place for the auditee to explain what is different. (Field 15 for AS9101E).

And why this fuss about AS9101D anyway. It's gone. It has been replaced by AS9101E. It is my understanding that all AS91XX audits performed after 1/1/15 are to be audited using AS9101E.
 

Buckyb

Involved In Discussions
#6
Re: Are KPIs a requirement to determine Process Effectiveness ?

I'm not saying there shoudn't KPIs available to help the organization determine how they are doing; my point is that it should not be up to the CB or IAQG/AAQG to edict that organizations must have them in order to satisfy audit requirements when neither Standard reqiures them. You say KPIs are there in spirit but I suggest once you have CBs or Quality Groups making the decisions for the organization on what is measurable and what is not, you have stepped over the line and are then auditing by interpretation. IT is the organizations decision on how to manage their quality system and defend their QMS accordingly. I agree that if effectiveness can't be determined then an NC is in order. But don't put the pen to paper just because KPIs are not used. AS9101D or E makes no difference. In fact AS9101E is somewhat worse in that it makes the assumption "KPIs are used to identify an organization's progress towards achieving its performance goals." I'm not talking about 5.4 where goals/objectives "shall be measurable". This is "processes effectiveness" where processes are montitored and "where applicable measured". Again, both Standards state: "shall indicates a requirement". I see no shall; so I don't believe it is fair to evaluate effectiveness via KPIs alone. AS9101E paragraph 4.1.2.4 doesn't state that KPIs shall....
 

Big Jim

Super Moderator
#7
Re: Are KPIs a requirement to determine Process Effectiveness ?

I'm not saying there shoudn't KPIs available to help the organization determine how they are doing; my point is that it should not be up to the CB or IAQG/AAQG to edict that organizations must have them in order to satisfy audit requirements when neither Standard reqiures them. You say KPIs are there in spirit but I suggest once you have CBs or Quality Groups making the decisions for the organization on what is measurable and what is not, you have stepped over the line and are then auditing by interpretation. IT is the organizations decision on how to manage their quality system and defend their QMS accordingly. I agree that if effectiveness can't be determined then an NC is in order. But don't put the pen to paper just because KPIs are not used. AS9101D or E makes no difference. In fact AS9101E is somewhat worse in that it makes the assumption "KPIs are used to identify an organization's progress towards achieving its performance goals." I'm not talking about 5.4 where goals/objectives "shall be measurable". This is "processes effectiveness" where processes are montitored and "where applicable measured". Again, both Standards state: "shall indicates a requirement". I see no shall; so I don't believe it is fair to evaluate effectiveness via KPIs alone. AS9101E paragraph 4.1.2.4 doesn't state that KPIs shall....
Did you miss this part of my post?

"Field 11 on the PEAR used by AS9101D asks for the auditor to describe how the auditee monitors that process. It suggests it may be KPI/quality objectives, but doesn't require it. That is the place for the auditee to explain what is different. (Field 15 for AS9101E)."

or this?

"I have no problem with companies finding other ways to monitor their processes. The problem is that most that don't use KPI/quality objectives to monitor their processes don't have another way, and simply whine about KPI/quality objectives don't need to be the way it is done."

What was required in AS9101D was that you needed 12 months of data going into registration. You still need data to fulfill the requirements in 8.4 even if you don't use KPI to monitor your processes. I always had a problem with 12 months worth being a requirement as it reached beyond what AS9100C had on the topic. Thankfully, that requirement was removed from AS9101E. What they are looking for now is a reasonable amount to show you are actually doing it, usually two or three months worth.

Learn what the standard says, both AS9100C and AS9101E, and be ready to defend yourself, but remember that if you don't use KPI, you must use something to monitor EACH process.

If anyone reading this has an alternative method I would invite them to post it.
 

Buckyb

Involved In Discussions
#8
Re: Are KPIs a requirement to determine Process Effectiveness ?

Big Jim,

I get it totally and we are in agreement. I am very knowledgeable on what the Standards state.....But my point is that the CBs are to audit accordingly. I know of one example already where the CB has directed their auditors to write an NC if KPIs do not exist. We both agree that all aspects of a process must be assessed for effectiveness and organizations have the right to defend their processes. I'm saying in this case there is no defense; no KPIs = NC.
 

RoxaneB

Super Moderator
Super Moderator
#9
Okay, I'm going to proudly wear the dunce cap and ask..."How does an organization demonstrate process effectiveness without a KPI?" An example of how to apply this would be greatly appreciated.
 

Big Jim

Super Moderator
#10
Re: Are KPIs a requirement to determine Process Effectiveness ?

Big Jim,

I get it totally and we are in agreement. I am very knowledgeable on what the Standards state.....But my point is that the CBs are to audit accordingly. I know of one example already where the CB has directed their auditors to write an NC if KPIs do not exist. We both agree that all aspects of a process must be assessed for effectiveness and organizations have the right to defend their processes. I'm saying in this case there is no defense; no KPIs = NC.
There is ALWAYS a defense. It is called an appeal. If satisfaction cannot be found through the CB's appeal process, it can be appealed through the AB that is on their certificate.

Since this is a requirement of ISO 9001, the IAQG has no authority to interpret the requirement, through the use of the AS9101 or any other method.

Back to what I ended up with before though, you MUST have a viable method of monitoring the process if you are not measuring it. Don't even dream of getting into a ruckus with this without being able to demonstrate an effective manner of monitoring that process.

I hear a lot of theorists complain about this, but none of them have been able to provide an example.
 
Thread starter Similar threads Forum Replies Date
C Calculating your KPIs ISO 13485:2016 - Medical Device Quality Management Systems 7
J KPIs or Metrics to Measure a New Complaint Handling Process 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 5
qualprod Proposed Kpis ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
P Received a minor for not having good measureables/goals. Need help with KPIs. IATF 16949 - Automotive Quality Systems Standard 52
K IATF16949 inputs/outputs & KPIs Internal Auditing 3
Q Corrective Action - Assigning KPIs to Non-Guilty Processes ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
Q Suggested KPIs for a Manufacturing Production Business ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 1
Q How to align a Business Strategy to Operative KPIs ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 7
Q KPIs - Lead and Lag Indicators - ISO 9001:2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
A If 4.1 is a process, how to assign KPIs for this process? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 12
J Lean Appropriate KPIs for a Distribution Centre Lean in Manufacturing and Service Industries 1
P New to AS9100 Rev. C - Setting KPIs and FAI Reports AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 14
S Process Map and defining KPIs Misc. Quality Assurance and Business Systems Related Topics 5
K Can you Suggest KPIs for a Business Process Improvement Team? Pls. Benchmarking 4
R Document Control Person - Setting KPIs, Development Plan, and Career Path Quality Manager and Management Related Issues 6
R Key Process Indicators (KPIs) for ISO 9001:2000-certified Service Organization ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
R KPIs (Key Performance Indicators) for Telecom Operators TL 9000 Telecommunications Standard and QuEST 1
B Is there a necessity of two sets of KPIs for Delivery Process Quality Manager and Management Related Issues 15
M First Draft of Policy, Objectives and KPIs - Your feedback appreciated ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 10
I Are KPIs (Key Performance Indicators) considered as Quality Objectives? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 7
A Aircraft Maintenance Quality & Safety KPIs - Industry Trend or Database EASA and JAA Aviation Standards and Requirements 3
A Aircraft Maintenance Quality & Safety KPIs - Industry Trend or Database Quality Manager and Management Related Issues 2
R Quality for Telco Companies - Quality in Customer Experiences & KPIs Quality Tools, Improvement and Analysis 5
0 Problem with setting up KPIs via Excel Pivot Table Misc. Quality Assurance and Business Systems Related Topics 6
Q Training KPIs (Key Process Indicators) Training - Internal, External, Online and Distance Learning 8
A Department KPIs - Plant, Machineries and vehicles (PMV) ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 1
Jim Wynne "Business Coach" Touts Basing Pay on KPIs Philosophy, Gurus, Innovation and Evolution 43
S Differences Between KPIs and Objectives/Targets ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 11
S Do Key Performance Indicators (KPIs) really make a difference in the QMS? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
D Key Performance Indicators (KPIs) in the Construction Industry ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
G KPIs required by TOP Management for strategic planning Statistical Analysis Tools, Techniques and SPC 2
D Key Performance Indicators (KPIs) book - Reviews and suggestions wanted Statistical Analysis Tools, Techniques and SPC 9
S KPIs (Key Performance Indicators) and Risk Management Process (ISO14971:2007) ISO 14971 - Medical Device Risk Management 5
B Defining Physical Asset Maintenance KPIs (Key Performance Indicators) Benchmarking 6
Ajit Basrur Customer Satisfaction Survey Content - Which KPIs employed? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
J Key Performance Indicators (KPIs) for Management Review process Management Review Meetings and related Processes 37
W Misinterpretation of requirement acceptable as root cause? Problem Solving, Root Cause Fault and Failure Analysis 19
S ECG Cable Banana to Snap or Tab electrode Adapters -- ISO 10993 Requirement Other Medical Device Related Standards 0
D IATF 16949 Requirement for CMMI in a Global Company Elsmar Cove Forum Suggestions, Complaints, Problems and Bug Reports 0
W Regarding Requirement for Hi pot & Leakage Current test Station Set-up IEC 60601 - Medical Electrical Equipment Safety Standards Series 6
L Stage 2 audit - Requirement for 3 months of records ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
M Requirement to manufacture under a quality management system EU Medical Device Regulations 4
A Risk Number for each software requirement IEC 62304 - Medical Device Software Life Cycle Processes 7
T ISO 17025:2017 requirement 5.7.b. about maintenance the integrity of the management system ISO 17025 related Discussions 1
I Laboratory Fridge / Freezer Inventory - Requirement(s) ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 6
A Several sections of traditional 510(K) have the same or similar requirement of content. US Food and Drug Administration (FDA) 4
S Requirement to Conduct New Shelf-life Testing? (re-do testing for design change) EU Medical Device Regulations 3
C Compliance with ISO 17025 requirement 8.4.2 - Controls - Records recovery ISO 17025 related Discussions 4
Q User Requirement Specification for HR (Human Resource Management System) Manufacturing and Related Processes 1
P Obstruction alarm requirement ISO 80601-2-12 Other Medical Device Related Standards 1

Similar threads

Top Bottom