Should Systems Effectiveness Be Audited?

[Marc]

From an ongoing discussion in the ISO Standards Discussion listserve. Food for thought.

--------------------

From: ISO Standards Discussion
Date: Tue, 16 May 2000 18:31:26 -0500
Subject: Re: ISO 2000 impact /Briggs/Hankwitz/Vianna

From: "Vianna, Sidney"

John, as usual, very well said.

To expect Registrars to be the measure of product performance and system effectiveness, is not realistic. How many days, in average, does a third-party auditor spend in certified companies/ year? 2, 3, 4???

If users of third-party certificates wanted Registrars to be accountable for such issues, we ( Registrars) would need to spend way more time that we presently spend at certified companies.

Compliance with Management International Standards does not guarantee either product performance (which is clearly stipulated/commented in the ISO 9001 Standard) nor system effectiveness. If compliance with ISO 9000, guaranteed system effectiveness, our UK colleagues would dominate the World Economy, because, relatively speaking, the number of certified organizations is dramatically higher in the UK, compared to any other region in the World. We know this not to be the case.

Concerning the question:
> Has a registrar ever been sued for misrepresenting a company as
> having an effective quality management system?

I do not believe so. By design, Registrars verify COMPLIANCE of the system to a set of requirements. NOT effectiveness.

I would welcome a discussion about making Registrars more accountable for the effectiveness of the certified company's quality system. However, we would need a whole set of new rules for that game. The current accreditation's rules does not provide for this expectation.

What can a third-party auditor do if an organization chooses to have a MRB process with 15 required signatures to disposition non-conforming product, rather than 1 or 2? What can a third party auditor do, if design reviews, although conducted, are totally meaningless because the few organizational interfaces identified did not have qualifications or competence to provide meaningful feedback at that stage of the design?

Opportunities for improvement. That is all third-party auditors can offer. Few organizations pay attention to that.

Instead of complaining about millions of dollars that might have been wasted in certifications, I would be concerned with the BILLIONS of dollars that have been wasted in IMPLEMENTING ISO 9000 compliant systems that are totally inadequate. Unfortunately, there are so many organizations out there that do not realize the importance of aligning their quality system with their business practices. Until organizations understand that quality is a mind set, rather than a department, we will have an uphill battle.

PS In addition to the information you provided about the suspension, by the RvA, of a major Registrar, there is another large Registrar, operating in the US that has their Scope of Accreditation partially suspended, also by the RvA. This information is available in the RvA website, but one must dig, before one finds it.

Best regards,

Sidney Vianna

 

[Marc]

From: ISO Standards Discussion
Date: Wed, 17 May 2000 10:34:53 -0500
Subject: Re: ISO 2000 impact /../Hankwitz/Vianna/Scalies

From: Charley Scalies

> Instead of complaining about millions of dollars that might have
> been wasted in certifications, I would be concerned with the
> BILLIONS of dollars that have been wasted in IMPLEMENTING ISO
> 9000 compliant systems that are totally inadequate. Unfortunately,
> there are so many organizations out there that do not realize the
> importance of aligning their quality system with their business
> practices. Until organizations understand that quality is a mind
> set, rather than a department, we will have an uphill battle.

In my opinion, the latter organizations are not the ones contributing to the large waste of time, effort and money. It is the ones of that ilk who also decide they want to get a Certificate, in spite of their quality culture (or lack of one). ISO9000 is like a handgun. In the hands of a knowledgeable, responsible individual who knows how and why it is to be used, it is a useful tool. But in the hands of a "cowboy" it becomes exceedingly dangerous. No doubt ISO9000 registration, or more precisely the marketing pressure that organizations experience or perceive, has been the impetus many have needed to get their quality houses in order. They may not have done so without that pressure. On the other hand, the existence of those registration schemes has also been directly responsible for much of the focus shift away from quality and toward "wall art".

The idea of 3rd party registration is not bad. It's the application that, IMHO, does not always pass the smell test. Why? Because as it is presently structured, ISO9000 registration - whether done by the creme de la creme of registrars or by the ones RvA declares, "GESHORST" - has absolutely nothing to do with quality. It has all to do with marketing.

Under these conditions, having registrars audit for effectiveness is akin to putting a party dress on a pig.

And before anyone asks, no, I don't have the solution. I know I would like an ISO9000 certificate to have the same weight as a financial statement certified by one of the Big 8 CPA firms.

Until then, I will continue to counsel my customers not to place too much stock in any one supplier's certificate. It may not even be worth the paper it's printed on. If it's a major supplier of critical supplies or services, do what we did in the military. Descend upon them with your own QA people and kick all the tires yourself. With everyone else, you can probably do just as well with referrals and trial orders until they prove their worth.

ISO9000 Good.
Registration - an opportunity for (vast) improvement


Charley Scalies

 

[Marc]

From: ISO Standards Discussion
Date: Mon, 22 May 2000 09:54:56 -0500
Subject: Re: ISO 2000 impact /./Vianna/Scalies/Vianna

Two things:

1) Credibility of third-party registration.

Like I have been saying for the last 7 years in this discussion list. Just like in any other aspect of life, this third-party certification business is no different; there are reputable, professional, serious Registrars and auditors, and, then, there are those just trying to ride the ISO wave and make a buck out of this. It will always be a buyer beware situation.

Can the Accreditation process guarantee credibility of the process? Only to a certain extent.

In my opinion, the accredited third-party system for certification of management systems needs to be refined. At this point in time, it is not a closed loop system. The Accreditation Agencies (ANSI-RAB, RvA, UKAS, etc. . .) do not have to answer to anybody, other than themselves during their peer reviews. In my view point, these agencies need to be accountable to Industry at large who are the "end consumers" of third-party certificates. Otherwise, who polices the police? What if there were knowledgeable, serious entities, such as IAOB, AAQG, Semitech, etc. while representing their Industry sectors, make sure that the Accreditation process guaranteed the integrity and competence of Registrars? Then we would close the loop. Industry could rely on the third-party certificates that they need. Industry would be able to define/refine/augment requirements for the Accreditation process.

Some recent examples of that, starting to take place: through the IAOB TS 16949 and the AAQG/SAE AIR 5359 AS9000 Accreditation processes. Obviously the QS-9000/AIAG process has been addressing this issue, as well, for a number of years, now.

In my opinion, until the Accreditation Agencies and Industry develop processes by which the credibility of third-party certification is maximized, there will always be a lingering shadow over the validity of third-party certificates.

2) The concept of what Charley calls "descend upon them with your own QA people and kick all the tires yourself". Charley, what do you do if you are a small business who can not afford supplier QA people? What if most of your critical suppliers are 12 time zones away? How cost effective is for you to send your QA people to a supplier across the Globe? How effective will be your QA people dealing with suppliers that do not speak your language or do not understand your culture?

I am not suggesting that second-party assessments should be abolished. But they have a purpose that many times overlap with third-party assessments. Second-party assessments should be optimized by focusing specifically on issues that are key to your organization as a buying entity. Second party assessments that rehash over and over again independent evaluations of quality management systems have their value diluted. Let's not forget also that, unfortunately, many second-party auditors are not qualified for the task. A few of them even come with the mind set of micro managing the supplier's business, what can be extremely detrimental to the supplier's operations.

By the way, Charley, I might be reading too much in between the lines of your message, but it feels like we should operate under the mode of not trusting our suppliers, thus the need to police them. Please note that one of the quality management principles supported by the ISO 9000:2000 series is Mutually beneficial supplier relationships: the ability of the organization and its suppliers to create value is enhanced by mutually beneficial relationships.

For quality to prosper, organizations will have to rethink the way they do business with their suppliers. Instead of trying to squeeze as much as possible from our suppliers we will need to partner with them and find ways of making the business relationship rewarding for both parties.

Best regards,

Sidney

From: Charley Scalies

<snip>The idea of 3rd party registration is not bad. It's the application that, IMHO, does not always pass the smell test. .. snip

Until then, I will continue to counsel my customers not to place too much stock in any one supplier's certificate. It may not even be worth the paper it's printed on. If it's a major supplier of critical supplies or services, do what we did in the military. Descend upon them with your own QA people and kick all the tires yourself. With everyone else, you can probably do just as well with referrals and trial orders until they prove their worth. <snip>

 

[Marc]

From: ISO Standards Discussion
Date: Mon, 22 May 2000 15:58:56 -0500
Subject: Re: ISO 2000 impact /../Scalies/Vianna/Hartman

From: dhartman

Sidney Vianna stated in part,
"For quality to prosper, organizations will have to rethink the way they do business with their suppliers. Instead of trying to squeeze as much as possible from our suppliers we will need to partner with them and find ways of making the business relationship rewarding for both parties."

I have dealt with suppliers, in fact we are currently procuring from several, that do not have ISO certified systems in place, and are probably the most dedicated and quality conscious suppliers that we have (verifiable by the measured quality of the product received from them).

We use many small, local suppliers that may consist of less that 10 employees total. Some of these have ISO certified systems in place, but couldn't provide consistent quality product if they had to; others (as I've noted above) don't have a defined quality system (let alone a certified system) and provide a consistent quality level higher than 90% of our vendor base.

Some of the contributors to this phenomena include such factors as: The vendor's willingness to work with our Design, Manufacturing, and Quality engineers; their willingness to fully understand our needs (partially by taking an interest in understanding our final product and customer needs); and their willingness to address issues attributable to their process, as well as notifying us of problems that are attributable to engineering errors, purchase document errors, etc.

We also have suppliers that have certified quality programs that are the largest contributors to vendor-related quality issues. Many times attributable to their management's attitude that we can take what we're given, or go elsewhere (most of the time, we go elsewhere).

Supplier quality is no different than those systems implemented by any other corporation, it is directly reliant on the cultural attitude of upper management. If management drives the organization with the attitude that we will manufacture a product, or provide a service, and they will come - they may come, but they probably won't stay. If management's drive is to understand and satisfy a need of the potential customer base - then not only will the customer be attracted, but long term relationships can be developed (with or without formal, certified quality systems)

David Hartman

 

[Marc]

From: ISO Standards Discussion
Date: Thu, 25 May 2000 10:38:01 -0500
Subject: Re: ISO 2000 impact /./Scalies/Vianna/Scalies

From: "Charley Scalies"

Sidney, I'm with you about increasing the credibility of the third party audit process. I just don't have any ideas about how to do it. Sounds like you have some good ones though.

This much I do know. If, like the current system, the new one requires auditors to spend "x" hours per audit based on the size of the auditee, then it, too, will demonstrate that it hasn't got a clue.

> 2) The concept of what Charley calls "descend upon them with your own
> QA people and kick all the tires yourself". Charley, what do you do if
> you are a small business who can not afford supplier QA people? What
> if most of your critical suppliers are 12 time zones away? How cost
> effective is for you to send your QA people to a supplier across the
> Globe? How effective will be your QA people dealing with suppliers
> that do not speak your language or do not understand your culture?

What did they do before ISO9000? How about getting references and checking/calling other customers of theirs. How about trial orders? How about "Little Guy's" Boss calling "Big Guy's" boss and getting personal assurances of performance to requirements.

I have been fortunate to have many leads come to me through my web site. But I have not booked a single piece of major business from a new customer who has not spoken to my previous customers and/or invited me in for a face-to-face.

Smallness does not imply impotence, weakness or dumbness. If a small firm makes it a habit of relying upon nothing more than ISO9000 certificates to place critical orders with critical suppliers then that small firm is bound to get even smaller. Naiveté does have its own unique set of rewards, especially in business.

> Let's not forget also that, unfortunately, many second-party
> auditors are not qualified for the task. A few of them even come with
> the mind set of micro managing the supplier's business, what can be
> extremely detrimental to the supplier's operations.

How many customers have significantly reduced their QA audits of suppliers (second party) just because suppliers are ISO9000 certified and get audited by third party registrars - who happen to have no allegiance to the customer? (I know, that's what was supposed to happen, so the story went.) If they did them before, I'll bet they still do them, and for the same reasons.

As for some/many(?) second party auditors dropping their baggage all over the place, "that ain't no lie". They do, and recipients of those "opinions" pretty quickly learn how to deal with them. On the other hand, one of my clients just had their registration audit conducted by a 2 man audit team - one of whom - thankfully not the audit leader who had his head screwed on straight - is a QS9000 consultant. He was not at all shy about sticking his nose deep inside my client's tent and telling them how they "should be doing it".

> By the way, Charley, I might be reading too much in between the lines
> of your message, but it feels like we should operate under the mode of
> not trusting our suppliers, thus the need to police them. Please note
> that one of the quality management principles supported by the ISO
> 9000:2000 series is Mutually beneficial supplier relationships: the
> ability of the organization and its suppliers to create value is
> enhanced by mutually beneficial relationships.

I trust suppliers in they same way as I trust my kids. When they demonstrate they are capable and worthy of that trust, they get all of it they can handle. It takes more than "Trust me Dad." And it takes more than a certificate.

It takes real performance. As for policing, I believe in monitoring and measuring supplier performance against my requirements. We don't need 4.6 to tell us to do that. It just makes sense.

> For quality to prosper, organizations will have to rethink the way
> they do business with their suppliers. Instead of trying to squeeze as
> much as possible from our suppliers we will need to partner with them
> and find ways of making the business relationship rewarding for both
> parties.

>From your lips to God's ears. I want to see the same things you do, Sidney, and despite my often pragmatic approach to such matters, I have very high ideals and hopes. Nevertheless, I think it is odd that many people, probably not including you, want to apply different rules to customer/supplier "partnerships". Before I take on a business partner, I want to know everything about him there is to know. I want to see for myself that my partner can hold up his end, because my success, my future, my fate is partly in is hands. Successful business partnerships and successful customer/supplier partnerships are far too critical to trust to blind date rules. I need more than someone's testimony that "She has a nice personality."

Charley

 

[Marc]

From: ISO Standards Discussion
Date: Tue, 30 May 2000 09:53:49 -0500
Subject: Re: ISO 2000 impact /../Naish

From: PNaish

In looking at both sides of this thread, I can see merit in both. But I for one believe that Sidney has the best model of what is appropriate.

Going across the world to see a company may or may not tell you something about them. If they have a different culture what you see may not fit your liking but may provide the product you want. As for calling their customers I find that to be of the least value but have used it as a way to satisfy auditors in the past. What makes you think they will give you the name of any unhappy customer? When you are asked for references do you give them the names of customers you know were unhappy with you? No!! So why do you think anyone else would. All companies that are smart pick the customers they have the best relationships with and give you those names.

As for site audits, some people can do a good job of covering their problems despite the best auditors. If you know how to audit well you know how to cover your problems when you are being audited. I have seen this happen a number of times with auditors who usually do well. And yes registrar auditors make mistakes but sometimes it is because of the skill at hiding things not just poor auditors of which there are some as well.

The proof is always in the pudding. Do some trial orders and see how they do. Slowly increase the depth and breadth of your purchases. Work with developing a clear understanding and agreement on what you want and what they can do. I frequently found in previous positions that design engineers would design beyond the capability of the supplier and wonder why we kept rejecting the material.

A partnership should include training for you as the customer of the capabilities the supplier can provide. So why not ask at the onset what services the customer provides in training material or capabilities materials you can give to your engineers or even to your customers if you pass through items like a distributor. Will they send someone over to provide a class or do they have design capability books.

In the United States we have come to believe the customer has to go to the supplier. In other cultures, the supplier comes to the customer and helps develop the relationship on an on going basis. The later seems to be much more effective than the former.

As for auditors telling companies how to run their business I have seen it way too many times and with only a brief look at a company they really do a poor job because what works for one does not always work for all. I have seen consultants who want to do the same thing. Suggestions softly given are great. Observations given with the implication a non conformance is eminent is unacceptable.

Phyllis