Are KPIs a requirement to determine Process Effectiveness?

[Buckyb]

I have a concern that I want to share with the forum. There is a potential problem brewing in aerospace auditing that subscribes to the notion that KPIs must be developed by organizations in order for the auditor to evaluate the effectiveness of the process being assessed. To not have KPIs or inadequate KPIs automatically results in "process not effective".

There are some (names and CBs withheld) that interpret AS9101D 4.1.2.5f to mean measurement is required. I would argue that the whole paragraph must be taken in context, not just item f. 4.1.2.5 states,"The audit team should evaluate, as appropriate, that processes:" are or have what is described in items a through h. There is no shall and "as appropriate" means just that. 4.1.2.5f can't be taken to mean measure as the only means as that is not the intent of the paragraph.

AS9100C Clause 4.1e. states "monitor and measure where applicable" and AS9101D states in Note 1 that KPIs are often used. There is no "shall" in these narratives either.

Also, AS9100 Clause 8.2.3 states that "the organization shall apply suitable methods for monitoring and, where applicable, measurement of the quality management system processes." The key is "where applicable"; it doesn't read monitor and measure.

While it is true that many organizations develop KPIs they are not required by AS9100. In fact AS9100 does not have "Key Process Indicators or KPI" anywhere in the document. AS9101D states KPIs twice; once in 4.1.2.5 Note 1 and on the PEAR Form Instructions item 11. Nowhere in any Standard does it state that KPIs are required to evaluate effectiveness but rather they are simply one of the means to measure effectiveness. KPIs aren't and can't be developed for every process and the choice to develop and use them must rest with organizational rationale and top management decisions. The Auditor's role must be to assess the process and determine its effectiveness; KPIs are only one means to do so.

I am not on this soapbox advocating no KPIs. I believe there has to be a mix of methods to measure process effectiveness and it can't be only KPIs. I hear rumors that this guidance/direction is coming from above in spite of what is in print in the Standards. If this is true it is a sad day for those who worked so hard to provide us Standards that we in the trenches try to live by.

 

[Golfman25]

Re: KPIs a Requirement to Determine Process Effectiveness

This is the old make sure all the boxes are checked approach. Process + KPI = check. Do it like everyone else and don't have to think.

 

[Jen Kirley]

Re: Are KPIs a requirement to determine Process Effectiveness ?

I had thought that KPI means "Key Process Indicator," which means a measurement of something important. Maybe I'm naive but I thought it was just an acronym for what the standard is asking us for: a means to know how well things are going before the customer does. No need to look for the acronym in the standard, just develop a measure of what's important.

 

[Big Jim]

Re: Are KPIs a requirement to determine Process Effectiveness ?

I had thought that KPI means "Key Process Indicator," which means a measurement of something important. Maybe I'm naive but I thought it was just an acronym for what the standard is asking us for: a means to know how well things are going before the customer does. No need to look for the acronym in the standard, just develop a measure of what's important.

Actually KPI means Key Performance Indicator. When the CBs, and I assume those above the CBs, started pushing hard on the process approach in early 2005, many automatically substituted Process for Performance, presumably because that is what they were being taught to use them for. This came from the business world, and evolved from MBOs (Management by Objectives).

 

[Big Jim]

Re: Are KPIs a requirement to determine Process Effectiveness ?

Although it is true that nowhere in the AS9100C standard (or the ISO 9001:2008 standard that it is derived from), are Key Performance Indicators mentioned by name.

It is there is spirit though in several places. It is in 4.1 where you are told to determine the methods to ensure control of the processes, where you are told to monitor them, and where applicable, measure them.

It is in 5.4.1 where it tells you that you need to have quality objectives, that they need to be measurable, and that they need to be established at relevant functions and levels throughout the organization. What do you think those relevant functions and levels might be? Processes maybe?

It is in 8.2.3 where it specifically says that you will monitor, and measure when applicable, your processes. This is somewhat of a repeat of 4.1, including the part about taking correction and corrective action as appropriate.

And here is the really big one. Where it tells you what topics you need to have measurements for (no where appropriate here). It is in 8.4 where it tells you that you need to determine, collect, and analyze data that helps you determine the health of your quality management system and evaluate where improvements can be made. It goes on to tell you four specifics topics to apply this analysis of data to.

I have no problem with companies finding other ways to monitor their processes. The problem is that most that don't use KPI/quality objectives to monitor their processes don't have another way, and simply whine about KPI/quality objectives don't need to be the way it is done.

Field 11 on the PEAR used by AS9101D asks for the auditor to describe how the auditee monitors that process. It suggests it may be KPI/quality objectives, but doesn't require it. That is the place for the auditee to explain what is different. (Field 15 for AS9101E).

And why this fuss about AS9101D anyway. It's gone. It has been replaced by AS9101E. It is my understanding that all AS91XX audits performed after 1/1/15 are to be audited using AS9101E.

 

[Buckyb]

Re: Are KPIs a requirement to determine Process Effectiveness ?

I'm not saying there shoudn't KPIs available to help the organization determine how they are doing; my point is that it should not be up to the CB or IAQG/AAQG to edict that organizations must have them in order to satisfy audit requirements when neither Standard reqiures them. You say KPIs are there in spirit but I suggest once you have CBs or Quality Groups making the decisions for the organization on what is measurable and what is not, you have stepped over the line and are then auditing by interpretation. IT is the organizations decision on how to manage their quality system and defend their QMS accordingly. I agree that if effectiveness can't be determined then an NC is in order. But don't put the pen to paper just because KPIs are not used. AS9101D or E makes no difference. In fact AS9101E is somewhat worse in that it makes the assumption "KPIs are used to identify an organization's progress towards achieving its performance goals." I'm not talking about 5.4 where goals/objectives "shall be measurable". This is "processes effectiveness" where processes are montitored and "where applicable measured". Again, both Standards state: "shall indicates a requirement". I see no shall; so I don't believe it is fair to evaluate effectiveness via KPIs alone. AS9101E paragraph 4.1.2.4 doesn't state that KPIs shall....

 

[Big Jim]

Re: Are KPIs a requirement to determine Process Effectiveness ?

I'm not saying there shoudn't KPIs available to help the organization determine how they are doing; my point is that it should not be up to the CB or IAQG/AAQG to edict that organizations must have them in order to satisfy audit requirements when neither Standard reqiures them. You say KPIs are there in spirit but I suggest once you have CBs or Quality Groups making the decisions for the organization on what is measurable and what is not, you have stepped over the line and are then auditing by interpretation. IT is the organizations decision on how to manage their quality system and defend their QMS accordingly. I agree that if effectiveness can't be determined then an NC is in order. But don't put the pen to paper just because KPIs are not used. AS9101D or E makes no difference. In fact AS9101E is somewhat worse in that it makes the assumption "KPIs are used to identify an organization's progress towards achieving its performance goals." I'm not talking about 5.4 where goals/objectives "shall be measurable". This is "processes effectiveness" where processes are montitored and "where applicable measured". Again, both Standards state: "shall indicates a requirement". I see no shall; so I don't believe it is fair to evaluate effectiveness via KPIs alone. AS9101E paragraph 4.1.2.4 doesn't state that KPIs shall....

Did you miss this part of my post?

"Field 11 on the PEAR used by AS9101D asks for the auditor to describe how the auditee monitors that process. It suggests it may be KPI/quality objectives, but doesn't require it. That is the place for the auditee to explain what is different. (Field 15 for AS9101E)."

or this?

"I have no problem with companies finding other ways to monitor their processes. The problem is that most that don't use KPI/quality objectives to monitor their processes don't have another way, and simply whine about KPI/quality objectives don't need to be the way it is done."

What was required in AS9101D was that you needed 12 months of data going into registration. You still need data to fulfill the requirements in 8.4 even if you don't use KPI to monitor your processes. I always had a problem with 12 months worth being a requirement as it reached beyond what AS9100C had on the topic. Thankfully, that requirement was removed from AS9101E. What they are looking for now is a reasonable amount to show you are actually doing it, usually two or three months worth.

Learn what the standard says, both AS9100C and AS9101E, and be ready to defend yourself, but remember that if you don't use KPI, you must use something to monitor EACH process.

If anyone reading this has an alternative method I would invite them to post it.

 

[Buckyb]

Re: Are KPIs a requirement to determine Process Effectiveness ?

Big Jim,

I get it totally and we are in agreement. I am very knowledgeable on what the Standards state.....But my point is that the CBs are to audit accordingly. I know of one example already where the CB has directed their auditors to write an NC if KPIs do not exist. We both agree that all aspects of a process must be assessed for effectiveness and organizations have the right to defend their processes. I'm saying in this case there is no defense; no KPIs = NC.

 

[RoxaneB]

Okay, I'm going to proudly wear the dunce cap and ask..."How does an organization demonstrate process effectiveness without a KPI?" An example of how to apply this would be greatly appreciated.

 

[Big Jim]

Re: Are KPIs a requirement to determine Process Effectiveness ?

Big Jim,

I get it totally and we are in agreement. I am very knowledgeable on what the Standards state.....But my point is that the CBs are to audit accordingly. I know of one example already where the CB has directed their auditors to write an NC if KPIs do not exist. We both agree that all aspects of a process must be assessed for effectiveness and organizations have the right to defend their processes. I'm saying in this case there is no defense; no KPIs = NC.

There is ALWAYS a defense. It is called an appeal. If satisfaction cannot be found through the CB's appeal process, it can be appealed through the AB that is on their certificate.

Since this is a requirement of ISO 9001, the IAQG has no authority to interpret the requirement, through the use of the AS9101 or any other method.

Back to what I ended up with before though, you MUST have a viable method of monitoring the process if you are not measuring it. Don't even dream of getting into a ruckus with this without being able to demonstrate an effective manner of monitoring that process.

I hear a lot of theorists complain about this, but none of them have been able to provide an example.