Measuring Process Effectiveness - Measures of Effectiveness (MOE's)

[GoKats78]

We never considered document control a process...

My opinion...too many organizations treat every activity as a "process" when many don't really fall into that category...another "process" I have seen like this is IT...

 

[ProDad]

Interesting! How do you get around treating it like a process? You have to handle document control to be in compliance with 4.2. I'd love nothing more than to call it something else and get away from measuring things that don't get parts out the door. I'm happy to audit these "procedures" and make sure things are getting done right, but it's a huge burden to put measures on administrative tasks.

 

[insect warfare]

We had a finding during our Stage 2 AS9100C audit for not having good Measures of Effectiveness (MOE's) for our processes. I predominantly used the results of internal audits to give an MOE for each process. For instance, the Document and Record control process: No findings during internal audit means the process is effective.

Our auditor did not like this.

What you quoted above is the main focal point of this discussion. You essentially confessed to the auditor that you "measure" each process via internal audit, and 2) you specifically cited Document and Record control as a viable example of how the measurement was conducted.

The auditor came to the conclusion that this is not "measurement", in which he/she is correct, because your output of measurement by internal audit does not result in a quantifiable value. What you did here was open yourself up to a finding, by your own admission as a representative of that process.

Now, to address the responses you have just replied to, you are still claiming to "measure" by internal audit. In reality you are "monitoring" by internal audit. Internal audit is a validation that you are, in fact, measuring or monitoring a process, and simply reporting on the state of its effectiveness, not "measuring" it. If you "measure" something, better put an actual numeric value behind it and justify your reason for it or you will repeat history once again.

Don't forget the standard's requirements also (paragraph 8.2.3): "The organization shall apply suitable methods for monitoring and, where applicable, measurement of the quality management system processes. These methods shall demonstrate the ability of the processes to achieve planned results."

In short, you must determine which processes require monitoring, which require measurement, and which require both, but don't claim that you "measure" where you really "monitor" and vice versa.

And don't focus too hard on the examples I gave you (they are just that - examples). These were thrown out there to show that it is possible for those processes (and yes, document control is a process whether we like it to be or not) which do not affect the bottom line to not be excluded from measurement, if that is what the organization desires. These processes can still negatively affect the system's functionality if not monitored (or measured) appropriately...not everybody has to "measure" them per se, but some do...for others, monitoring is suitable enough.

And I don't think your reply was argumentative. It just provides more insight into the situation at hand, and I hope I've clarified things a little better here.

Hope I've Helped...
Brian

 

[ProDad]

Ah. The light bulb is flickering on now.

Like you said at first: The word "measure" is key. So if I'm understanding you correctly, my problem may be in stating that we will "measure" all processes, whereas some processes only need to be monitored.

So that begs the question: What is "monitoring"? How would that be different for a Document Control process than measuring.

I absolutely see the value in spot checking various document control tasks to be sure people are doing what they are supposed to. I think constant follow-up is vital. It was the tracking of some contrived metric that was bugging me because it doesn't add value. Would evidence of monthly spot checks prove monitoring is taking place?

And then for internal audit of that process, that would be more about making sure the documents we control meet the requirements of all clauses of the standard and any customer requirements, right? Am I right in saying the IA is about conformance to the standard, and 8.2.3 is more about us doing what we say we are doing?

Wayne

 

[Bev D]

Am I right in saying the IA is about conformance to the standard, and 8.2.3 is more about us doing what we say we are doing?
Wayne

not really. Internal auditing is about both compliance to the standard and compliance to your internal procedures.

effectiveness is about whether or not the process is giving you good results. you may be completely compliant and producing ****.

start to think about this in very simple terms: what is the intent of a document and record control process?

 

[insect warfare]

Sorry for my late reply (out to lunch)...

Remember that 8.2.3 also contains a note for guidance:
"When determining suitable methods, it is advisable that the organization consider the type and extent of monitoring or measurement appropriate to each of its processes in relation to their impact on the conformity to product requirements and on the effectiveness of the quality management system."

Would evidence of monthly spot checks prove monitoring is taking place?

Yes, as long as you have kept some type of record proving this happened.

So that begs the question: What is "monitoring"? How would that be different for a Document Control process than measuring.

While I don't see any definitions for these two words in ISO 9000, it is not really that hard to distinguish the two...just a simple query in dictionary.com yielded these results:

Monitor: "to watch closely for purposes of control, surveillance, etc.; keep track of; check continually.

Measure: to ascertain the extent; dimensions, quantity, capacity, etc., of, especially by comparison to a standard.

So let's apply this to document control:

Monitoring example: For this month, you have checked all the documents on the floor and on the server and determine that the versions all match up against the master list (here you have found an aspect of document control to be effective, because the document controller is doing a good job, at least for now - no action needed).

Measuring example: You decide to do the same thing as above, but now you want to display it as a percentage. So 100/100 of the documents on the server are of the correct version, or 100% effective, but only 95/100 of distributed hard copies are of the correct version, or 95% effective (here you have determined a measurement of effectiveness from which to act upon - we need to investigate the root cause of wrong versions on the floor so we can raise this %)

Depending on the type and extent of control you want to apply, as stated above, either way can be utilized. On the subject of document control, some people may not bother with quantities or values (and that's fine) while others may want to know this stuff because of their statistical obsessions and whatnot.

Numbers are hard to argue with, though. But common sense usually prevails and most people don't have to overkill things like document control.

Does this make sense?

Hope I've Helped...
Brian

 

[ProDad]

Yes, it makes sense.

I believe this answers my questions. Thanks for all your responses Brian!

 

[insect warfare]

No problem. I'm glad I was able to assist...

Brian

 

[GoKats78]

It was simple...We had a procedure (which IS required...but we were TS..don't know about AS) that covered Document Control..there is nothing requiring that it be a process...

 

[dsanabria]

the process effectiveness are not limited to the control of documents and records and the calibration of measurement device or instrument, and not say these items are not important, these items are severice for production and the end of customer only; the more important is find out a method suitable your company operation, so i think you are mor pay attention to the complaince the actually operation status and ISO standard

Check this document out...

Keep it simple