Auditing for effectiveness

I notice a reluctance on behalf of Registrars to write up anything which is not clearly against a specific requirement of ISO 9000. In 4.17, the standard requires internal auditors to "determine the effectiveness of the quality system". Note that first we verify whether activities comply with the quality system - presumably the answer to that can be 'they do'. Then a jugement is asked as to whether this correctly implemented system is 'effective'.
This is obviously not limited to being objective. It becomes a judgement call. Yet it is of vital importance. I've heard a registrar say that a quality system may meet the requirements of the standard and yet produce junk. But we can't reconcile this requirement with anything but excellence and continuous improvement, which, I would think, was the purpose behind putting it in there.
Soon to be presenting myself as a consultant, I have to have a clear approach to this but it is probably the most difficult area to deal with. My conviction is that I should take the thing at face value and judge subjectively based on my own judgement of what is good and what is bad. But it looks like a minefield.
I'd appreciate the opinion and some feedback from the relevant experience of a cross section of correspondents; What do the consultants do? What do the in-house people and their colleagues expect? What do the registrars say?
thanks and rgds,
John C

[Don Winton]

John,

You bring up an interesting subject, as always.

I notice a reluctance on behalf of Registrars to write up anything which is not clearly against a specific requirement of ISO 9000.

While some may dispute this statement, I am not surprised by it. The primary purpose of the registration assessment is (should be) to determine compliance, not noncompliance, to the standard. Therefore, logic would dictate that in order to 'write up anything,' there must be a area where compliance is not demonstrated.

Now, the 'gray' area.

In 4.17, the standard requires internal auditors to "determine the effectiveness of the quality system".

Under the assumption that an organization is ISO 900x compliant for the right reason, then the internal audit program is the key to effective implementation. The Registrar cannot (should not) determine effective implementation, just compliance. The internal audit program, then, is where the effective implementation must be determined. The internal auditors are presumably well versed on an organization's operations. This being the case, they would be the ones to determine effective implementation. One could assume, then, this is the reason the clause in 4.17 is there.

I've heard a registrar say that a quality system may meet the requirements of the standard and yet produce junk.

This is true. I have seen, and still see, it in action. I remember a discussion a while back that a company could produce concrete life jackets and still be ISO 900x compliant. After a study of the various responses and dialog, I had to agree. I find nothing in the standard that could prevent me from operating a concrete life jacket factory and still be ISO 900x compliant. But, this would be an organization that is ISO compliant for the wrong reason.

My conviction is that I should take the thing at face value and judge subjectively based on my own judgement of what is good and what is bad. But it looks like a minefield.

That would depend upon your role and the scope of your responsibilities. As a consultant, the contractor is paying you for a service. Now, which service is he paying for?

They want an ISO compliant system: In this case, judging subjectively would probably be the mine field you described. They may want to be compliant just to satisfy some customer's requirement. The right thing for the wrong reason. Avoid subjectivity and establish a compliant system.

They want to establish, implement and maintain an effective Quality Management System. In this case you are free to subjectively point out areas where their effective implementation is 'short' and make suggestions on improvement.

As strictly an assessor, you should not be subjective AT ALL, just objective, at least during the assessment. After the closing meeting, you could 'offer' areas where the effectiveness is short, but they would be under no obligation to take the advice and you cannot audit against it during future assessments. I STRONGLY advise against offering any suggestions if your role is as assessor.

Am I completely off base?

Excellent subject, BTW.

Regards,
Don


------------------
Just the ramblings of an Old Wizard Warrior.

Check Out dWizard's Lair:
www.ficom.net/members/donwinton/home.html

Don,
Not off base but right on target, I'd say. You blasted my concerns past the floodlights and they're still going. I see now how the minefield only happens if you go outside your brief - the customer is always right so make sure you really know what their expectations are. It reminds me of what a colleague who left here told me just a couple of weeks ago; She now works in a chemical plant on quality systems and said she has to listen for a long time before suggesting anything; If you want to change anything, they come over suspicious and worried. "What's wrong with it?", What happened?", almost to the point of saying; "What are you suggesting?" Not the same thing, exactly, and I'm not saying that all such plants are like that, but it seems like a place where to go outside objective observations could land you in the minefield. My preferred clients will be chemical plants - I'm planning to combine 9000 and 14000 in my product/service range, so it's good that I now can categorise this attitude as somewhere in between the 'right reasons' and the 'wrong reasons'. I guess it's a case of "the right reasons, but remember there's more at stake here than a quality certification", situation.
So my mind is put at rest on the 'effectiveness' issue and I can make my plans to deal with it. Thanks again.
rgds, John C

[Marc]

I notice a reluctance on behalf of Registrars to write up anything which is not clearly against a specific requirement of ISO 9000.

Start worrying when they cite you for something that's not in ISO9001.

[Don Winton]

... the customer is always right...

This is something I preach in my classes. WRONG! WRONG! WRONG!

The customer is not always ‘right;’ in fact they are seldom right. BUT, they are always the customer. It is our job to be able to explain the difference.

... know what their expectations are.

This is the key to what I was trying to expound upon above. Stay within the scope of the customer’s expectations.

So my mind is put at rest on the 'effectiveness' issue and I can make my plans to deal with it. Thanks again

Do not let the mind rest too easily. You have many a hard road ahead. Best of luck.

Start worrying when they cite you for something that's not in ISO9001.

Agreed.

Regards,
Don

------------------
Just the ramblings of an Old Wizard Warrior.

Check Out dWizard's Lair:
www.ficom.net/members/donwinton/home.html

[Don Winton]

Start worrying when they cite you for something that's not in ISO9001.

Actual 'observation' from a recent Document Review:

"Although reference is made on individual pages in the Quality Manual to ISO standard’s paragraphs, there is no statement in the body of the policy or quality system manual or procedures of the intent of compliance to ISO 9002."

This was sited against 4.1.1. Thought I would share.

Regards,
Don

------------------
Just the ramblings of an Old Wizard Warrior.

Check Out dWizard's Lair:
www.ficom.net/members/donwinton/home.html