Analysis of Data (Rookie Auditor Needs Help) - AS 9100 8.4

[JSRDE]

I work with Brian and helped him with this audit the second time around. We asked the quality manager (also management representative) and the asst. quality manager...

What data relating to conformity to product requirements is analyzed to demonstrate the suitability and effectiveness of the quality management system?
-Respsonse: We analyze internal nonconformances.

When reviewing this data what tells you that our system is effective? At what point do you decide "No, it is not working?", or "Yes, it is working?"
-Response: I look at the internal nonconformances on a weekly basis and if I see anything that repeats I look into it to see if it is an indication that our system is not working. If needed I will write a corrective action.

I tried to explain to the managers that does not demonstrate that the system is effective. If they would set some measurable objectives then it would be obvious when you were analyzing the data that if we did not met the objective then our system is not effective.
I suggested objectives such as:
Rework labor must be < 2.0% of all labor.
No customer returns.
These things to me will demonstrate more accurately that our system is effective rather than subjective analysis on a weekly basis.

Neither of them feel that what they are currently doing is a nonconformance of the standard. So, now I have to present this nonconforming finding to all of upper management to get their input on whether or not this truley is a nonconforming finding.

 

[howste]

Here is a progression of linked requirements that I would walk top management (leadership) through to show the intent of the requirements in 8.4:

The organization shall
a) identify the processes...
c) determine criteria and methods needed to ensure that both the operation and control of these processes are effective,
e) monitor, measure and analyse these processes...

Translation: After we establish QMS processes, we must determine criteria for effectiveness of each process and monitor/measure each process.

Top management shall ensure that quality objectives... are established at relevant functions and levels within the organization. The quality objectives shall be measurable and consistent with the quality policy.

Translation: We must establish measurable objectives that support the needs of the organization. The "functions and levels" may correspond with our QMS processes from 4.1, and the objectives correspond with our metrics from 4.1.

Top management shall ensure that
a) the planning of the quality management system is carried out in order to meet the requirements given in 4.1, as well as the quality objectives...

Translation: The purpose of the QMS is to help us to maintain and support our processes and to achieve our established objectives.

Top management shall review the organization's quality management system, at planned intervals, to ensure its continuing suitability, adequacy and effectiveness....
The input to management review shall include information on... c) process performance...

review
activity undertaken to determine the suitability, adequacy and effectiveness of the subject matter to achieve established objectives

Translation: Management reviews must review process performance metrics. This will allow us to determine if the system is effective in achieving our objectives.

The organization shall apply suitable methods for monitoring and, where applicable, measurement of the quality management system processes. These methods shall demonstrate the ability of the processes to achieve planned results. When planned results are not achieved, correction and corrective action shall be taken, as appropriate, to ensure conformity of the product.

Translation: The process metrics must demonstrate that the processes are achieving results. When they don't, we must take action. Note: the crossed-out text is removed from FDIS ISO 9001:2008 to refocus on process results, not just product.

The organization shall determine, collect and analyse appropriate data to demonstrate the suitability and effectiveness of the quality management system and to evaluate where continual improvement of the effectiveness of the quality management system can be made....
The analysis of data shall provide information relating to... c) characteristics and trends of processes

Translation: Process metrics must demonstrate that our system has been effective (or not) in meeting our goals. This will lead to identifying needed actions for improvement of our processes and system. Analysis of metrics must show the trends of our processes. This will also demonstrate if we are actually continually improving or not (see 8.5.1).

I got a little long in my response, but I feel that if we only look at (or point out to others) individual clauses we sometimes miss the big picture of how they relate to the intent of the whole system. I hope this helps.

 

[JSRDE]

This is great information, howste, I think management will definetely respond well to scripture and verse. Brian and I had both said the same thing to the Quality Manager (Management Representative) that we need to set these objectives not just to meet 8.4 but to also meet 5.4.1, 5.4.2 and 5.6. Now I will just have to spell all of that out for management and include 8.5.1 and 8.2.3. I will try to let you know how it goes. Thanks again.

 

[AndyN]

It reads ('sounds') to me like you're auditing out of the standard requirements, rather than auditing the business's processes, using a process approach! All the way through your posts I read "ISO Speak", so no wonder there's a debate over whether it's a nonconformity......

To my way of thinking, it's a simple case of understanding the process, its objectives, the measurements of performance and comparing the results obtained to the objectives. It shouldn't be beyond the comprehension of the process's management as to whether the process is effective and what they are going/doing about it if it's not performing to plan....

I believe you might need to adjust your audit technique to get the results you're expecting......

 

[howste]

Great point. I was trying to paint a picture to show the overall intent of ISO 9001. The fact of the matter is it should be done because it makes good business sense, not just because of an ISO requirement.

 

[JSRDE]

Your right, and I know it needs to change. This is my first year of experience with auditing, well actually the entire company's first year of experience.
I was trained to do the process approach in an external training course, and changed to audit by standard requirements when our registration auditor wrote us an observation. I was new at it and just changed to please him. I know now that I should have stuck with it. I know that observations are sometimes no more than an auditor's opinion.
I see after this first year how ridiculous it is to try to audit to the standard. I am going back for some additional training along with all of the internal auditors and hopefully get back on track with process approach. To bad the Management Representative is not going.
I think this is a great example of how process approach will help improve our auditing process as well as the company's understanding of what is required.

 

[AndyN]

Your right, and I know it needs to change. This is my first year of experience with auditing, well actually the entire company's first year of experience.
I was trained to do the process approach in an external training course, and changed to audit by standard requirements when our registration auditor wrote us an observation. I was new at it and just changed to please him. I know now that I should have stuck with it. I know that observations are sometimes no more than an auditor's opinion.
I see after this first year how ridiculous it is to try to audit to the standard. I am going back for some additional training along with all of the internal auditors and hopefully get back on track with process approach. To bad the Management Representative is not going.
I think this is a great example of how process approach will help improve our auditing process as well as the company's understanding of what is required.

Awesome response! Sometimes, y'know, we just have to try 'the dark side' to know it's not that great! I'm so pleased that you see why it needs to change - go out there and make a difference!

 

[AndyN]

To address the 'concept' of 'how do we know our system is effective'?, at management review is to be able to look at certain sources of information and ask:-

What did we plan to do? (policies, objectives, customer satisfaction etc.)

What results did we get compared to the plans? (what yields did we get, customer feedback etc.)

Did we follow the (defined) process? (what do our audit results tell us?)

From that, management should be able to decide - "What can I improve or what needs fixing?"

So, if we are getting good results from our process and audits tell us we're following the process = opportunity to improve

If we get good results and our audit results say we weren't following the (defined) process = corrective action.

Now, if the processes are not getting good results, to plan, then whether the people follow the (defined) process is also important in taking action.

That, I believe is Management Review in a nutshell.