ISO 9001 - Measuring Effectiveness of the QMS System

ISO 9001 - Measuring Effectiveness

I saw this on the Iso ListServe. I want to remind all of you who visit here it is an excellent listserve. I hope you are on their list. For you who are not, this is a question I see as most relevant because of the two ISO 9001:2000 audits I have witnessed. I'll post any 'good' replies, but how about you? What's YOUR opinion?

***********************

From: the ISO ListServe
Date: Tue, 31 Jul 2001 10:40:30 -0500
Subject: Q: Measurable Results from ISO in our World /Naish

From: PNaish

In our change from ISO 9001:1994 to ISO 9001:2000 we added a focus on measures of our effectiveness. And a number of us have gone out and implemented a number of measures on our systems and processes and customer satisfaction (that is our customer's satisfaction). We have been doing some kind of measure of our supplier's quality and probably delivery.

My question may sound like Bah Humbug but I have a question regarding the measurement of ISO effectiveness. And yes I can always change suppliers (maybe if there are some). But what about the overall bottom line measures to our businesses. Now some of us who are in the profession of consulting or auditing have surely seen bottom line improvements but what about other companies?

I was just on the ISO web site and was astounded to find that in 2000, we have generated 46,998 pages in new documents. I think it would take me half a year just to read all of those pages much less understand them all. And total pages for 2000 they indicate to be 391,582 pages. I am sure by the time I get to the end of them I will have to start over with the changes to them.

So how do we measure our effectiveness and process improvement in this realm when it seems we are spiraling out of control? And since ISO does not have a competitor I guess I can't go to another supplier.

I have had no less than 5 calls in the last month from clients with problems (and 2 personal experiences) with suppliers that are ISO. Mostly in the contract review part. And they run from service industry (international trade carrier, to calibration (metrology providers)), to high tech manufacturers, to automotive.

Who if anyone is really measuring the effectiveness of ISO as it grows ever bigger? Has anyone seen any statistics or kept statistics on the overall improvement in either quality or delivery in any particular industry that can be directly related to the advance of ISO standards?

We have had more (or at least more noticeable and deadly) automobile and airline problems. Appliances in the USA have become a throw away item. Of the five computers I have had, only one has lasted more than a year or two without failures. And repair has become a nightmare to get from anyone reliably.

So again what have we as a world gained from ISO?

Sorry for the bah hum bug but as I saw the ISO web site I could not see a value for all the money we are all spending. It is made worse by the mission statement of the ISO site, the ASQ site, and the A2LA site who say it is for our good and yet I have seen a number of problems just trying to weave through their sites to find what I was looking for.

P Naish

I see no one seems to want to comment, so here are a couple of follow-ups from the ISO ListServe:

*********************

Date: Wed, 1 Aug 2001 15:47:52 -0500
Subject: Re: Measurable Results from ISO in our World /Naish/Meron

From: Emanuel

Phyllis

My estimate of the impact of ISO 9000 on management practices and/or business competitiveness is very simple: I look in Sloan Management Review (SMR) and Harvard Business Review (HBR). These are two of the most highly esteemed management journals, published by MIT and Harvard Business School. Worth reading. I believe that if some novel or supposedly useful management idea is not mentioned in one of them it probably means it's not worth researching. So after reading your post I searched the archives of those two publications. Here is what I found for three keywords: Quality, TQM, ISO (number of references):

SMR Quality = 92 TQM = 6 ISO = 0

HBR Quality = 300 TQM = 58 ISO = 3 (two from 1996, one from
2000)

So what do you think? Maybe ISO is not so hot after all. On the other hand, could it be that those academic types never heard about ISO?

Emanuel

********

Date: Wed, 1 Aug 2001 15:51:54 -0500
Subject: Re: Measurable Results from ISO in our World /Naish/VanDorp

From: Darryl

Replies embedded

From: PN

-Snip-

I have had no less than 5 calls in the last month from clients with problems (and 2 personal experiences) with suppliers that are ISO. Mostly in the contract review part. And they run from service industry (international trade carrier, to calibration (metrology providers)), to high tech manufacturers, to automotive.

-Reply-

And I have had almost 5 calls from my customers even though we have ISO 9002. We still make "mistakes" but now we have a documented process to resolve them and hopefully still providing good customer service.

-End Reply-

Has anyone seen any statistics or kept statistics on the overall improvement in either quality or delivery in any particular industry that can be directly related to the advance of ISO standards?

-Reply-

I may be a nonconformist here but how can you attribute the improvement of any particular industry to one item. There are hundreds (if not thousands) of variables of which ISO would be just one.

-End reply-

We have had more (or at least more noticeable and deadly) automobile and airline problems. Appliances in the USA have become a throw away item. Of the five computers I have had, only one has lasted more than a year or two without failures. And repair has become a nightmare to get from anyone reliably.

-Reply-

Are there statistically more vehicular failures causing serious injury or death? Have there been statistically more fatal or serious plane crashes? Are they just more thrown into the media spotlight in today's news hungry society?

Of the five computers you bought what models were they and did that company have ISO 900x? Did you buy the el-cheapo models from the local quickie mart? I have 10 computers here bought from a reputable supplier who uses quality components of which i've had virtually no problems and excellent service if there's an occasional glitch (and no they don't have ISO)

As for Appliances... it's a consumeristic society and the appliance companies business model is to make disposable appliances to there by sell more. There is a market here. Just like if you want your washing machine to last you buy Maytag there are brands of appliances which you purchase if you want them to last.

-End reply-

So again what have we as a world gained from ISO?

-Reply-

It's a market driven society and business environment. You get what you pay for and what customers demand for the most part they get. ISO is in the big scheme of things a small player in these market dynamics IMHO, it's a useful and valuable tool but not an ace in the hole.

Besides, without ISO what would all those registrars and consultants be doing?

-Darryl

******************************

Date: Thu, 2 Aug 2001 11:03:40 -0500
Subject: Re: Measurable Results from ISO in our World /Naish/VanDorp/Naish

From: PN

Darryl,

If we did not have ISO I would look for another job which would probably contribute more to society. But I thought a couple of responses you sent were very good and will respond to a couple of the questions you asked.

You said: "Are there statistically more vehicular failures causing serious injury or death?" At some not so recent Senate hearing regarding the tire and car issue, the transportation safety board along with some other people presented data for five years. And the answer to your question is yes there were significantly more fatal and non fatal accidents especially related to tire issues. As we know, the companies in question are ISO or QS.

In addition, the data did some comparisons to other car and tire manufacturers. Interestingly enough they were up as well but not significantly as was the specific tire and car companies. So my perception is that the data collect by and presented by the NTSB would indicate more accidents and fatalities. But again as you mentioned there are numerous things that play into this data.

You said: Of the five computers you bought what models were they and did that company have ISO 900x? Did you buy the el-cheapo models from the local quickie mart?

I bought the computers from three different manufacturers. 2 were direct, the other three were through two separate distributors. The direct manufacturer may be what you call el cheapo as they were a local company. All three manufacturers have or have had ISO including Mr. El cheapo. The one that has lasted the longest is the Mr. El cheapo. The one to run the second longest is Mr. El cheapo. So price does not necessarily relate directly to quality nor service.

As for calls from customers. We all make mistakes and we all get calls regarding them just like you said. The problems is in each of these calls from clients they could not obtain a corrective action response from their suppliers even after repeated efforts to do so. I do not want to go down the path as to why or the requirements of ISO versus the suppliers procedures but I have found several examples where they say they don't do corrective actions in their procedures and have their certificates. So what have we gained from ISO except a useless piece of paper when it comes to my getting what I agreed upon and getting a reasonable response when there is a problem.

I am personally working with one company where the management refuses to take or return my calls and after 2 weeks has failed to respond to a formal request for corrective action and resolution of a dispute. Even their registrar has been notified and while I was told a week and a half ago I would get a response from them I still have heard nothing more back from them.

I have received a couple of off line responses as well as those posted. I find so far that I have not seen any positive measure being made by anyone including ASQ or ISO or A2LA as to the effectiveness of these systems nor what the customers are getting from them.

My question is if ISO went away tomorrow with all the consultants and audits and registrars and ASQ and A2LA and their world wide equivalents, would the world be any worse? Or are these groups simply a new kind of business that makes money off of the consumer?

I know some would say they keep us from having major quality problems but that is obviously not true. Some would say the suppliers would not care and not provide goods that meet customer requirements. Some don't today and some will always work that way. In today's economy, I would say more companies provide what customer require in order to stay in business than to meet some international standard. And I know first hand that any number do not follow their procedures except for when they are being audited.

Phyllis

*******************

Date: Thu, 2 Aug 2001 11:04:01 -0500
Subject: Re: Measurable Results from ISO in our World /Naish/Meron/Holtz

From: John

> From: Emanuel
>
>> From: PN
-snip-
>>
>> Who if anyone is really measuring the effectiveness of ISO as
>> it grows ever bigger? Has anyone seen any statistics or kept
>> statistics on the overall improvement in either quality or delivery
>> in any particular industry that can be directly related to
>> the advance of ISO standards?
>>
>
> My estimate of the impact of ISO 9000 on management practices and/or
> business competitiveness is very simple: I look in Sloan Management Review
> (SMR) and Harvard Business Review (HBR). These are two of the most highly
> esteemed management journals, published by MIT and Harvard Business School.
< Worth reading. I believe that if some novel or supposedly useful management
> idea is not mentioned in one of them it probably means it's not worth
> researching. So after reading your post I searched the archives of those two
> publications. Here is what I found for three keywords: Quality, TQM, ISO
> (number of references):
> So what do you think? Maybe ISO is not so hot after all. On the other hand,
> could it be that those academic types never heard about ISO?

Oooo, Emanuel...ouch! Those Sloan and HBR stats hurt.... Boy, Phyllis, thanks for getting us here! [Just kidding....]

Some 400,000 registrations worldwide, perhaps 40,000 in the US, and only 3 citations from two of the leading [US] business journals. It sounds to me as if these two august journals are LONG overdue to conduct a study of ISO-registered firms to find out what, if any, benefits have accrued from their efforts, and why and why not. I mean, correct me if I'm wrong, but like it or not, ISO 9000 is one of the significant business developments of the 1990s. It's worthy of a story.

While they're at it, make sure they include CO(P)Q in their analysis. Can you imagine the fodder for our discussion group if either journal made the move! We should probably be prepared for some bleak news amidst the sporadic raves.

Are there any volunteers that would like to contact MIT and Harvard to put a bug in their ear?

Cheers...and have a great day!
John

[This message has been edited by Marc Smith (edited 03 August 2001).]

I'm not as technical as those who have posted or been quoted in this thread. However, I love the question about what would happen if ISO went away. I have always been of the opinion that it's just another club for the Customer to beat you over the head with if you don't meet their pricing needs. My family is quite large and many are sucessful in their careers working for successful companies. Hardly any of them know what ISO is. It's not part of their life, or their company, and they manage quite nicely. None are working for a "certified" company as we know it. When I explain what it is, they wrinkle up their noses and say Huh? Right now, for those of us that have to dabble in it, ISO is the name of the game. That's all it is. A game with lots of people making lots of money at your, or your comany's expense, with no measurable results.

later all,

energy

[This message has been edited by energy (edited 03 August 2001).]

Maybe a way to make it easy:

Customer Satisfaction measurements, and data related to new business aquired because of the sole reason of registration.

A good management review process should already be taking these things into consideration.

ASD...

Just thought I'd throw in my 2 cents worth.

Companies buy ISO for 2 reasons. Either they are forced into it by their customers as a requirement of doing business, or they embrace it as a method of improvement. Obviously those who embrace it are going to feel it benefits them more than the ones who are forced into it. But there may be another point.

Companies purchase machinery, computer systems, buildings, etc, to A) improve their capacity/capability, and/or B) to satisfy some need of their customers.

If my customer says to me he has 5 or 10 million dollars in business that I can get if I have a certain $250,000 piece of equipment, you can bet I'll try like hell to get it. It helps me and it helps my customer. Plus I can likely get business for this same machine from other customers.

If my biggest customer says, "We have ZXZXZXZ software to send orders, drawings, etc and also for invoicing, and we want you to get this software, I'm going to get the software if at all possible. It satisfies my customer, and improves my capabilities.

ISO is really no different. If becoming certified means you will get more business then it's worth doing. If my customers do not care, and you see no benefit then don't do it.


James

------------------
Low tech is better than no tech.

I try to visualise it this way ...

The Deming/Shewhart PDCA continual improvement cycle is a wheel.

The wheel is on a journey to the legendary place of "world class quality". This trip is all UPHILL.

ISO 9001 is a wedge that you place behind the wheel. It prevents you from rolling backwards.

There are many mileposts along the road. To get the ISO 9001 wedge you must first have passed number of them, such as "Top Management Commitment". There are other mileposts still ahead, such as MBNQA or the Deming Prize.

No matter how far you have come, there is always a little bit further to go.

Graeme

[This message has been edited by Graeme (edited 21 August 2001).]

Well,

I've been hearing a lot of doubt about ISO 9000 standards recently. Much of it is really true, much - not...
However, I think the only reason for all this doubt is based on overestimation of the Standard's meaning. Such overestimation results in all the problems of understanding and applying the ISO 9000.
First of all, I would agree: it is true that ISO 9000 is not a magic remedy - it is only one of the factors providing for quality and success.
Secondly, ISO 9000 is only a standard and there may be a certificate for this standard. - Nothing more! What gives you the value-added is a quality system designed for quality, not for certificate.
It is bitter to say that, but the ISO 9000 certificate is not a 100% evidence of a company's quality, as we would like it to be. It is so in any country and by any registrar. It probably even could not ever become (would be too naive to believe it). We anyway have to trust experience more than a paper.

It is especially true for companies in countries like Russia, where I am working:
Here it does not mean much what ISO 9000 certificate you have. It only means much what quality you can provide.
However, ISO 9000 is a perfect reason and a brilliant technique to build-up regular management including its quality-related functions. And this is the most important thing about ISO 9000 in Russia.
A formal certificate is a secondary question. You can apply for it if you need it for raising a qood contract, or if you feel truly ready for it after months and years of re-engineering and optimizing your quality function. These are two way equally possible and useful, but aimed at different goals.

So, a resume: I never mind if the ISO 9000 shall lose its popularity and value for companies. The main thing it is extremely useful today and not only as a standard, but also as a technique to provide better quality. If it goes away, there is going to be something new and better as a standard and a benchmark.

Anton

[This message has been edited by Anton Ovsianko (edited 22 August 2001).]

Anton,
Nicely said. The very exercise of trying to structure your business using the standard as a guideline is beneficial. However, all the Quality Systems I've been exposed to, and in thirty+ years there have been several, did the same thing. The company complied with MIL-Q-9858, Mil-I 45208 and associated standards which addressed most of the requirements in the new standard. There are new twists added, all designed to improve Quality. The need to be certified puts additional pressure on companies to do it now! I would like to take a few years working towards registration. Like you say, it's an excellent technique to improve Quality and if it goes away, it will be replaced with something else. I don't necessarily agree that it will be better. Personally, I preferred the Standards used in the 70's. The customer was constantly monitoring you for compliance. The customer would not entertain purchasing your product unless you met the requirements. You had Gov't Agencies, in the case of Defense contracts auditing you continuously. Not twice a year. Talk about Customer Satisfaction. One did not need an independent organization to "approve" your processes.

Thanks for your comments Energy,

It is very nice to notice the similarity between people of the sameee age and experience in very different countries with very different economies (at least economic histories). People of your experience in managing Russian enterprises - especially technologically advanced military ones - would talk the same way.
Yeah the military and other industry-specific standards have always been more efficient in getting manufacturers comply to certain requirments. And it is very important in the spheres, where the compliance is tighly bound with security, life danger etc.
My opinion is that these specific standards have to continue being used - they are just necessary together with their strict auditing and monitoring procedures.
However ISO 9000 series has a bit different aim. It is meant for those industries, where customers would like to enjoy the economy of not auditing their supplier's processes, but trusting them.
Any rule supposes a wish to break it. That is why no-one can be 100% sure of evenn ISO9000-certified quality, however, the risk of non-conformity and dissatisfaction of a cutomer in this case has to be significantly lower.
Military standards are very effective, which means they can provide 100% perfect goods and services, but they are very expensive. ISO 9000 could be called their 'light' version, for universal use.

Anton

Although many bash quality systems to some extent, don't most of us make a living because of them?

ASD...