Friends!
On the first regular audit by the Registrar of our ISO 9001:2000 certified system the auditor noticed that main performance indicators for anaslysis of our Management Review process were not defined properly. :mg:
These are our indicators we showed him: number of Planned actions; executed actions vs. planned, %; Actions executed on time, %; Revised issues.
Could you please advise which else performance indicators can we assess for our Management Review process.
Thank you in advance! :thanx:

the auditor noticed that main performance indicators for anaslysis of our Management Review process were not defined properly.Not properly defined... Ok,what clause did he nail it with?

/Claes

Not properly defined... Ok,what clause did he nail it with?

/Claes

I suppose that clause 8 of ISO 9001:2000 standard.

He noticed that we could not properly analysed the effectiveness of Management Review for future improvement using such Performance indicators. :mad:

Maybe I'm confused. The standard speaks of measuring the effectiveness of the Quality Management System (QMS), and it also speaks of holding Management Reviews at planned intervals, but it never speaks of measuring the effectiveness of the Management Review.

A review is simply a review, "a retrospective critical evaluation". The purpose of the review is to go over how effective you have been in other areas of the QMS. The most it says about the review itself is that you look at the "status of management review action items" (ISO-9004 5.6.2), which it appears you are doing. I'd challenge them on that one.

I suppose that clause 8 of ISO 9001:2000 standard.Yes... That is what I supposed too (Or more specifically clause 8.2.3), but the auditor must know, which is why I asked. Is he auditing acc. to the standard in question, or acc. to his personal opinion? If he cannot nail his finding to a certain clause, I'm inclined to believe that he is voicing an opinion rather than auditing. He noticed that we could not properly analysed the effectiveness of Management Review for future improvement using such Performance indicators. Did he, now? If your MR produces output that helps the organization to improve (Fulfilling the requirements of clause 5.6.3) I would consider it to be on track, because that is what the MR is supposed to achieve. The question is: Are you happy with the MR output?

/Claes

Added later: Hullo Rob, I see that we are thinking along the same lines

Your auditor might be looking for indicators of:
Company established goals/objectives - performance data against these goals
Customer Satisfaction - CCRs, On-time Deliveries, etc.
Process performance - analysis of stated process goals, ie: scrap, cost of quality, PPM
Internal audits - results and actions

As Claes says, find out what clause and we may be able to better identify what he/she is looking for.

Dave

This reminds me of our recent discussion on whether sales growth was a 'quality objective' and how this might be audited - and I think the majority group conclusion was that an auditor really could not raise non-compliances against the content or type of 'quality objective'.

Hence I do not know how your auditor could conclude that the KPI's were not defined properly - what does he mean by properly ?

This reminds me of our recent discussion on whether sales growth was a 'quality objective' and how this might be audited - and I think the majority group conclusion was that an auditor really could not raise non-compliances against the content or type of 'quality objective'.

Hence I do not know how your auditor could conclude that the KPI's were not defined properly - what does he mean by properly ?

Your auditor should be evaluating whether your KPI tells YOU how your process is functioning. Also, who says there has to be a KPI for Management Review Process. That is only one acceptable method to evaluate effectiveness. Remember the tagline of one of our frquent poster (is it JSW05?) - not everything that can be counted should be counted...

Maybe I'm confused. The standard speaks of measuring the effectiveness of the Quality Management System (QMS), and it also speaks of holding Management Reviews at planned intervals, but it never speaks of measuring the effectiveness of the Management Review.

A review is simply a review, "a retrospective critical evaluation". The purpose of the review is to go over how effective you have been in other areas of the QMS. The most it says about the review itself is that you look at the "status of management review action items" (ISO-9004 5.6.2), which it appears you are doing. I'd challenge them on that one.

I agree with Rob's premise. However, I would add, many clients define management review as a process. If such, they would want to evaluate the effectiveness of this "process." However, that doesn't mean they have to define a KPI (what would it be?). There are many ways to evaluate process effectiveness. Clause 8.2.3 says monitor effectiveness or measure where applicable.

Friends, thank you all for your response!

I really agree with all of you, but anyway it was registered as our non-conformity and we have to improve it now. That is the point.
Our Registrar is one of internationally recognised and very qualified company and I can't have any doubt in their competence.
If you advice me how we can evaluate the effectiveness of Management Review (using KPI or other method) I would appreciate you.

I messed around a bit with your above post, Julia. You posted it in the Documentation Levels of QMS System - Please Review (http://elsmar.com/Forums/showthread.php?t=13389) thread, but I assume this is where it belongs, so I moved it.I really agree with all of you, but anyway it was registered as our non-conformity and we have to improve it now. That is the point.Right. First of all, if kpi's are to be used, I see nothing wrong with the ones you already mentioned (no of planned actions; executed actions vs. planned, %; Actions executed on time, %; Revised issues). Our Registrar is one of internationally recognised and very qualified company and I can't have any doubt in their competence.Actually, in the light of this NC I can, and I would probably not accept that NC. But: To me, the real question is how to turn this into something that is useful to you, and not only satisfies the auditor.

/Claes

Actually, in the light of this NC I can, and I would probably not accept that NC. But: To me, the real question is how to turn this into something that is useful to you, and not only satisfies the auditor.

/Claes
Yes, Claes, you're right.
So, we can't disregard officially registered NC and we have to improve it.
I have no idea (except what I mentioned as some KPI) how we can assess effectiveness of Management Review so that it would be our Corrective action for NC and to be useful for our company.

I'm really surprised that nobody assess Management Review. Is it possible?

I'm really surprised that nobody assess Management Review. Is it possible?Well, we do of course, it's just that we don't use KPI's to do it. We assess it in internal audits, mainly by having a look at agenda, input, records and results. Besides, the agenda changes every now and then acc. to the needs of the moment.

/Claes

Claes, I see your point.
So, if I show to auditor the results of Internal audit for Management responsibility including Management review - if it will be really enough for him?

I personally feel that assessing the effectiveness of the 'assessment' processes is going a little far, and when does one stop ?

If I assess the effectiveness of a process by internal audit, and then assess the effectiveness of internal audit, do I then need to assess the effectiveness of the effectiveness assessment of internal audit, and would I then need to assess the effectiveness of the assessment of the effectiveness of the assessment of the effectiveness of internal audit, and would I then............................

Have I made my point yet ?

So, if I show to auditor the results of Internal audit for Management responsibility including Management review - if it will be really enough for him?Um... I can only speak for myself, but If it was me: Yes. I have a feeling though, that your auditor may have a different opinion, and I don't want to get you into trouble. The best thing would have been (Yes, it's so easy to be wise in retrospect, I know, I know) to have contested the NC during the audit. It's much harder for him to back down now.Have I made my point yet ?Yes, Da, Si, Ja, Oui.. LMAO http://elsmar.com/Forums/images/smilies/lol.gif You have, in your usual softspoken way, and I agree.

/Claes

u can add following things (but not limited to same) in your agenda of meeting to satisfy your auditor

Follow up on the assignment of previous Management Review
Non Conformances and Trends
Customer Complaints
Sales Return
In-House Rejection
Business Position
Market Share
Supply Trends
Vendor Performance
Product Improvements
Human Resource Activities Report
Quality Objectives Status
TQM Activities Status

I think D. Scott and Kashifbutty2k are on the right track. The auditor seems to have identified a problem with some of the inputs to Management Review outlined in clauses 8.5.1, 8.4 and 5.6.2. JuliaSun should go through those clauses and ensure that appropriate analysis of data has been done to provide management with information that will aid their decision making and comply with the standard. The auditor may have been looking for evidence of management review of process and product metrics, and instead found that “performance indicators” all related to planning effectiveness and not to measurements of production processes and customer feedback.

I think D. Scott and Kashifbutty2k are on the right track. Absolutley.The auditor may have been looking for evidence of management review of process and product metrics, and instead found that “performance indicators” all related to planning effectiveness and not to measurements of production processes and customer feedback.That is entirely possible. The problem is that acc. to Julia he is asking for better main performance indicators for anaslysis of their MR process, which is something completely different...

One thing is blindingly obvious, however: Whatever it is he is after, he would have caused much less confusion if he had stated the std clause in question clearly. Now Julia (and the rest of us) have to resort to guessing.

/Claes

Yes... That is what I supposed too (Or more specifically clause 8.2.3), but the auditor must know, which is why I asked. Is he auditing acc. to the standard in question, or acc. to his personal opinion? If he cannot nail his finding to a certain clause, I'm inclined to believe that he is voicing an opinion rather than auditing. Did he, now? If your MR produces output that helps the organization to improve (Fulfilling the requirements of clause 5.6.3) I would consider it to be on track, because that is what the MR is supposed to achieve. The question is: Are you happy with the MR output?

/Claes

Added later: Hullo Rob, I see that we are thinking along the same lines

It's not just "Are YOU happy"! Is this a value added activity and does it enhance the QMS? Do you have data that's being presented that trends this information? ....I so often review MR meeting where I can feel that the managers are really just wondering..."What do I have to know and what do you want?"...On the other end I also see a lot of meetings that are AWESOME!

Welcome to the Cove, Jetlag :bigwave:It's not just "Are YOU happy"! Is this a value added activity and does it enhance the QMS?Er, yes... That is what I meant with the "If your MR produces output that helps the organization to improve...." part.

I agree that MR is a great tool when used correctly. Unfortunately, that is not always the case...

/Claes

Do we have a KPI to measure the effectiveness of Management Review?....no. We have over 400 KPIs at my site alone....this includes all levels from crew up to senior management (so naturally there are duplicate KPIs).

However, what we can show is the process for the outputs from Management Review and how they are used to improve the effectiveness of our Business Management System.

It's this wonderful little thing called an Action Item. During Management Review, recommendations are made and discussions are had. From that, a "To Do" list is formed. Any item on the "To Do" list equates to an Action Item. It could be something as simple as updating a Failure Treatment report that slipped through the cracks or something more resource-needy like develop a new database to streamline the Quality Planning and Engineering Assistance Request processes.

Something like the latter, would then get entered into our Routine and Improvement Management software and tracked as an official improvement project.

An issue like updating the record, would be addressed quickly and the Action Item log updated with the item closed at the next Management Review.

Effective....timely addressing of the issues, actions to a suitable magnitude taken, and all reviewed via an audit (be it internal and/or external).

Thanks to all for your replies. :agree1:
I have to say that we really have some problems with Management Review in our company.
First of all, our General Manager usually leads MR meetings and sometimes my agenda does not coinside with his. :argue:
I mean that he considers some actual problems and may not pay attention to all required Review Inputs such as Internal Audit Results and so on. The auditor noticed this during last regualr audit (it was another NC). Only after that the General Manager agreed that we need thoroughly discuss all inputs required by ISO standard.
I think that if we conduct the MR meetings as exactly required by the standard - it would be easy for us to assess the effectiveness of MR. :biglaugh:

...sometimes my agenda does not coinside with his. I understand (what a diplomatic way of expressing it :lol: ). It happens to all of us, of course. ...I mean that he considers some actual problems and may not pay attention to all required Review Inputs such as Internal Audit Results and so on. The auditor noticed this during last regualr audit (it was another NC). I suppose that NC was more "useful" than the one that prompted you to start this thread, then? Glad to hear it. I think that if we conduct the MR meetings as exactly required by the standard - it would be easy for us to assess the effectiveness of MR.Hear, hear... :applause:

Concerning your original question (The NC about KPI's for MR): I realize that we had little help to offer. Have you decided on how to handle it?

/Claes

Concerning your original question (The NC about KPI's for MR): I realize that we had little help to offer. Have you decided on how to handle it?

/Claes

Not yet. But I think that I will manage it anyway. The issue is not so easy for me. :mad: I have to think it properly. :bonk:

I have to think it properly. That sounds like a good idea. It usually is...:agree1:

/Claes

Greetings everyone.

We have just made the transition from 1994 to the 2000 version. During the audit, the auditor cited among others, that "continual improvement is not evident in all functions of the QMS (vs. 8.5.1)." Because of the dictatorial way with which she conducted the audit, we had sat in a stupor when she recited her findings during the closing meeting. We had prepared quality objectives for the major departments, revised our policy based on the objectives, created process maps, conducted management review and had minutes of the meeting, conducted IQA (citing among others that our IQA procedure is inadequate because there is no mention of open and closing meetings nor dicussed the purpose for these meetings), and indicated the need for more trainings on the implementation of preventive action programs. Now we do not know how to correct the non-conformance report. Do we have to make a procedure for continual improvement? :(

I would be vrey much grateful for your inputs.

You have to have a serious talk with your registrar's contract manager, and get some of those NCRs removed, if possible.

First of all, what evidence did she cite to prove that "continual improvement is not evident..."?

Secondly, there is no requirement in ISO9K2K for the internal audit (IQA?) process to have opening and closing meetings.

Thirdly, there is no minimum amount of training required for "implementing preventive action programs".

In all fairness, I do not know from your post what you DO have to support those general areas. Perhaps they weren't described sufficiently by your organization or worded properly by the auditor. Maybe you could explain here the following (and we can go from there): How do you track improvement activities? How do you communicate internal audit results? What is included in your training program?

Greetings everyone.

We have just made the transition from 1994 to the 2000 version. During the audit, the auditor cited among others, that "continual improvement is not evident in all functions of the QMS (vs. 8.5.1)." Because of the dictatorial way with which she conducted the audit, we had sat in a stupor when she recited her findings during the closing meeting. We had prepared quality objectives for the major departments, revised our policy based on the objectives, created process maps, conducted management review and had minutes of the meeting, conducted IQA (citing among others that our IQA procedure is inadequate because there is no mention of open and closing meetings nor dicussed the purpose for these meetings), and indicated the need for more trainings on the implementation of preventive action programs. Now we do not know how to correct the non-conformance report. Do we have to make a procedure for continual improvement? :(

I would be vrey much grateful for your inputs.


It does sound a little inappropriate from your decription. However, for each nc the auditor issued, they had to give you a Corrective Action which identifies the particular requirement and clause you failed to meet. Review these, and ask the auditor to explain any you do not agree with, or do not understand. If the auditor's request is not supported by the requirement, then there should be some discussion to rescind or revise the particular nc. Generally, this can be done in an amiable manner. Most auditors keep their clients up to date on findings as they go, in order to avoid the "shock" at a closing meeting.

If this is not succesful, then I agree you should ask to speak with a Supervisor at the registrar. Something does not sound quite right.

Hello there!! :)
What I've been noticing from the questions in this thread is that there seems to be a definite lack of understanding on how certification audits should be conducted. If there is little concensus among auditors as to the interpretation of ISO standards, then at least there should be some consensus as to how audits should be conducted. ISO 19011 gives guidlines for internal and external audits which if followed would eliminate most problems arising during certification audits.

For example: if there are disagreements (and why shouldn't there be??) during the certification audit they should be registered as such. The client should understand what NCs have been registered and should agree to them BEFORE the audit ends.
Auditors should demonstrate the characteristics and qualities clearly described in 19011. My firm conviction is that there is either very little VALUE adding during audits,or in the attempt to "value add" auditors become dictative and inflexible.

Open to disagreement and or comments.
Regards,
Colleen;)

You have to have a serious talk with your registrar's contract manager, and get some of those NCRs removed, if possible.

First of all, what evidence did she cite to prove that "continual improvement is not evident..."?

Initially, our process map. We made three, the first was the overall map based on PDCA. This is similar to the figure that can be found in the ISO 9k2k standard where the "continual improvement" phrase is written on the topmost part. Then we made another one to show the main business processes within the PDCA. Then, another one on the subprocess, like looking into the micro level, so we did not include the "continual improvement" phrase. When we explained that the succeeding maps where like magnified portions of the previous ones, she cited other observations and maintained that she cannot see that we have continual improvement efforts.

Secondly, there is no requirement in ISO9K2K for the internal audit (IQA?) process to have opening and closing meetings...."

That too, but it came to a point where we know that if we speak up and explain, the matter will just get worse and another NC will be raised. *sigh*

Thirdly, there is no minimum amount of training required for "implementing preventive action programs"

True. It was brought up during the management review and included in the minutes. We wanted to equip our personnel with skills to eanble them to conduct preventive action activities. IMHO, that could even be used as proof that we wanted to make improvements in our QMS.


In all fairness, I do not know from your post what you DO have to support those general areas. Perhaps they weren't described sufficiently by your organization or worded properly by the auditor. Maybe you could explain here the following (and we can go from there): How do you track improvement activities? How do you communicate internal audit results? What is included in your training program?

We have made changes in our quality policy and set quality objectives based on the results of our strategic planning. Our IQA results are discussed during management reviews. Our proposed training program includes a) training our personnel on the identification of appropriate statistical tools in addition to the production process charts that are generating and, b) strengthening our preventive action efforts.

Let's just say that by the time the audit neared its end, the group felt like they were treading on eggshells and were careful not to rock the boat. :(

Sigh and double sigh.

It does sound a little inappropriate from your decription. However, for each nc the auditor issued, they had to give you a Corrective Action which identifies the particular requirement and clause you failed to meet. Review these, and ask the auditor to explain any you do not agree with, or do not understand. If the auditor's request is not supported by the requirement, then there should be some discussion to rescind or revise the particular nc. Generally, this can be done in an amiable manner. Most auditors keep their clients up to date on findings as they go, in order to avoid the "shock" at a closing meeting.

If this is not succesful, then I agree you should ask to speak with a Supervisor at the registrar. Something does not sound quite right.

Thank you. We are looking into that direction. :)

Hello there!! :)
Auditors should demonstrate the characteristics and qualities clearly described in 19011. My firm conviction is that there is either very little VALUE adding during audits,or in the attempt to "value add" auditors become dictative and inflexible.

Open to disagreement and or comments.
Regards,
Colleen;)

Amen to that. There were some instances when it was very tempting to ask "are we talking about OUR system or yours?" :frust:

it came to a point where we know that if we speak up and explain, the matter will just get worse and another NC will be raised. *sigh*

Let's just say that by the time the audit neared its end, the group felt like they were treading on eggshells and were careful not to rock the boat. :(

That sounds terrible, and clearly like a good time to start rocking the boat. I know that is easier said than done, but really, there are limits: The auditor is supposed to help the organisation improve, not bully it around and dictate how it should be run. You already have a boss for that purpouse, for crying out loud.

/Claes

The whole thing sounds very unfortunate. There is a better way. A good auditor should attempt to never leave an audit with this kind of atmosphere. I may not be perfect, but we always try to maintain a friendly and open discussion during our audits. Doesn't always work, but almost always. And, I truly do try to be value added, but definitley not in a dictatorial manner. Auditees should not feel they have to walk on eggshells. It is just not how audits should go.

Hello.

Just wanted to say thank you for letting me vent out my frustrations. We're working on the NCs for the follow up this November. It's not something we look forward to but...that's how it goes. :)

Just wanted to say thank you for letting me vent out my frustrations. No problem. We all need to do that every now and then.

/Claes

All objectives must be reviewed during Management Review process. For every objective you should have a metric and the goal. I.E. objective : to have high quality parts, metric: PPM and goal PPM =0. What is your current PPM -this becomes one of your KPI. Other objectives: delivery, customer satisfaction data, customer complaints, continual improvement etc.