Trusting ISO 13485 Certification of a Supplier... A Sad Story

[Sidney Vianna]

First point: Is there anyone who argues that certification/registration (regardless of the registrar identity) is anything more than someone attesting the auditee has met the MINIMUM standards for meeting a Standard?

Second point: Almost every 3rd party auditor says up front they monitor "processes," NOT "products," so customers who require suppliers to get 3rd party audits STILL have no idea whether the products will meet customer requirements even if the supplier is certified/registered.

While the thrust of this thread is around ISO 13485, which (rightfully so) places much more focus on product regulatory compliance than ISO 9001, we should remain aware of what should be reasonable expectations of properly accredited QMS certification and the IAF Expected Outcomes for Accredited Certification to ISO 9001 document does a good job at that, in my opinion.

 

[Wes Bucey]

While the thrust of this thread is around ISO 13485, which (rightfully so) places much more focus on product regulatory compliance than ISO 9001, we should remain aware of what should be reasonable expectations of properly accredited QMS certification and the IAF Expected Outcomes for Accredited Certification to ISO 9001 document does a good job at that, in my opinion.

I don't see anything in the 6 page document that argues AGAINST my use of the phrase
"auditee has met the MINIMUM standards for meeting a Standard"

No auditor for a 3rd party registrar has EVER done anything more than check "records" as to whether a product meets requirements (via reading complaint files, inspection records, etc.) - all kept by the auditee. There is no cross check with customers to confirm the validity or sufficiency of those records. At least the CPA firm that audits my bank sends out a letter detailing some tidbit of MY record as shown by the bank and asks if I concur. I can't recall ever getting such a request from a registrar in my experience with suppliers registering to ISO Standards since 1987.

Certainly, no 3rd party auditor undertakes to do independent product testing as part of an ISO audit to determine if it meets all pertinent requirements.

The whole thrust of this thread has been that the "expectation" of good, consistent quality is "sometimes" NOT met. No one argues that all or most registrant companies fall short, only that the registration itself only says that "at the moment of the snapshot" the registrant's operations seemed to confirm its records. There are never any guarantees by an auditor that ALL the operations matched ALL the records, only those which were directly examined or observed by the auditor. (Isn't that similar to sampling a batch of product and saying "the sample was good - we infer the rest is equally good, but do not guarantee it." OR, in the language of the brochure cited by Sidney, "It does not necessarily ensure that the organization will always achieve 100% product conformity, though this should of course be a permanent goal.")

 

[Ronen E]

Yep. Second point DOES undermine the concept because "some idea" is NOT assurance. In the ten years and more that I've been reading and contributing to the Cove, we continually see threads pop up where customers and even employees of certified/registered suppliers decry the fact they have a certificate on the wall and yet they ship dreck out the door or, worse, have high internal reject rates.

I recall being a crank here in the Cove about a stainless steel teakettle I bought which looked super slick enough for me to plunk down money to own it only to find out the manufacturer had used a plain steel (1214 - I had it chemically analyzed) bolt to fasten the heat proof knob to the lid. Within 2 weeks in the steam and heat, the bolt had corroded to the point where oxide flakes were dropping into the pot.

Investigation revealed the company manufacturing the tea pot did, indeed, have ISO 9001 registration from a reputable registrar. I, of course, did not pre-condition my purchase on the registration status of the manufacturer, but I presumed the department store chain where I purchased it had done a due diligence of its suppliers. Some chains flex their economic muscle on manufacturers to meet "price points," forcing the manufacturers to seek ways to lower their costs. In the case of the kettle, they may have supplemented their profit by saving a penny or two on the bolt. The department store chain purchasing departments rarely have the level of expertise to ferret out some item like that and, even so, the original PPAP samples may have had stainless steel bolts. Specialists like the kind I hired would never be employed to monitor a commodity item like a teapot and so the scam continued. Most consumers would not have the resources or even think to look for them because of a rusty bolt - they'd just start over. For me, if I buy another stainless steel commodity like the tea pot, I'll examine the components MUCH more closely before purchase and either forego purchase or make preemptive replacements before use, depending on a cost-benefit analysis.

What's the bottom line solution?
Customers must be vigilant. A former President of the USA had a wonderful sound bite:
"Trust, but verify!"

Hi,

When I wrote "some idea", it was an understatement. Regardless, QMS auditing is a layer in the overall scheme. Alone, it doesn't provide 100% assurance of anything.

A functioning QMS is better than no QMS.
An audited QMS is better than an open-loop one.
A registered QMS is sometimes better than a self-certified one.
But as I said, even most reputably-registered QMS won't alone guarantee 100% compliance with customer requirements. They are only more likely to come near enough.

So, in your opinion, where/how exactly this kettle manufacturer's ISO 9001 registered QMS failed to ensure that the kettle complied with the "stainless" requirement?

Cheers,
Ronen.

 

[John Broomfield]

As I wrote (see highlighted word):

"Third point: There are "some" (very few) non-registrar auditing firms that purport to monitor product quality as well as process quality, but (to my knowledge) no surveys of the clients of such firms exist to assure product quality has improved (at a net savings of cost of audit versus cost of poor quality) to justify hiring such non-registrar auditing firms.
​

In regard to Kitemark, is it substantially different from the Good Housekeeping Seal of Approval wherein advertisers had their goods essentially "insured" by the magazine where they placed paid advertisements because the magazine "guaranteed" satisfaction or money back? It still doesn't answer the issue of whether the quality is actually better from a cost standpoint or just a good marketing ploy to allow charging higher prices for the "guaranteed" product? (the higher price essentially an insurance premium for assurance of replacement or money back, but no real guarantee the product will outlast or outperform a competitor at a lower price.)

[color me cynical, to say the least :]

Wes,

And this article explains how the Kitemark stands apart from the Good Housekeeping Seal of Approval. It includes a Kitemarked kettle.

Unfortunately the EU's CE mark does not enjoy the same high reputation for quality and safety.

John

 

[Ronen E]

I don't see anything in the 6 page document that argues AGAINST my use of the phrase
"auditee has met the MINIMUM standards for meeting a Standard"

No auditor for a 3rd party registrar has EVER done anything more than check "records" as to whether a product meets requirements (via reading complaint files, inspection records, etc.) - all kept by the auditee. There is no cross check with customers to confirm the validity or sufficiency of those records. At least the CPA firm that audits my bank sends out a letter detailing some tidbit of MY record as shown by the bank and asks if I concur. I can't recall ever getting such a request from a registrar in my experience with suppliers registering to ISO Standards since 1987.

Certainly, no 3rd party auditor undertakes to do independent product testing as part of an ISO audit to determine if it meets all pertinent requirements.

The whole thrust of this thread has been that the "expectation" of good, consistent quality is "sometimes" NOT met. No one argues that all or most registrant companies fall short, only that the registration itself only says that "at the moment of the snapshot" the registrant's operations seemed to confirm its records. There are never any guarantees by an auditor that ALL the operations matched ALL the records, only those which were directly examined or observed by the auditor. (Isn't that similar to sampling a batch of product and saying "the sample was good - we infer the rest is equally good, but do not guarantee it." OR, in the language of the brochure cited by Sidney, "It does not necessarily ensure that the organization will always achieve 100% product conformity, though this should of course be a permanent goal.")

I'd say it's very much down to the specific auditor's competence.

a) A good auditor would be able to sample wisely so that they actually cover the weaker spots; and infer correctly from what they sample and see to the entire system. Organizations sometimes arrange and keep few exemplary files which they try to push forward during audit, to make a good impression, but a good auditor should be able to identify and avoid them.

b) If all an auditor does during audit is "verify that the operations meet the records" then they are only doing half the job. The first task should be to identify that the system is Standard-aligned AND ALSO geared for the specific type of product/service. I'm not an expert on ISO 9001, and definitely not on all the nuances of accreditation, so perhaps what I say here is not a formal requirement; however IMO it's a must for an effective audit and for achieving the goal of an effective QMS. In the medical devices field, it's almost impossible to have a truly compliant QMS that DOESN'T look at product compliance through appropriate verification.

In all my comments above I have of course excluded fraud. If an organization is fraudulent, it may take more than a competent auditor to uncover.

 

[Ronen E]

Wes,

And this article explains how the Kitemark stands apart from the Good Housekeeping Seal of Approval. It includes a Kitemarked kettle.

Unfortunately the EU's CE mark does not enjoy the same high reputation for quality and safety.

John

I think the CE mark is not intended for the same purpose. The CE mark does not indicate meeting ANY published standards. It only suggests that relevant published standards may have been applied in the process. Application of Harmonized Standards is voluntary by nature.

The CE mark is a declaration that legal EC instruments that have a provision for CE marking (not all of them do) have been complied with.

 

[Wes Bucey]

I don't think registrars are bad or evil. The problem, as I see it, is that too many folks have a higher expectation of what a certificate of registration means in terms of a supplier's ability to consistently deliver goods and services per requirements than is justified.

I wasn't joking when I said "Trust, but verify!" Too often, customers get complacent [lazy?] when they have a supplier in their supply chain with a certificate of registration to an International Standard (ISO, TS, etc.) Some suppliers, when desperate from economic pressures (or just plain venality), will note that complacency and make moves to take advantage of slack vigilance by customers. I needn't bore readers with a litany of real life examples, but, if pressed, can provide them. In most cases, no disaster occurs, no end users are injured or killed, and those few cases where customers DO COMPLAIN are dealt with in the same way GM dealt with auto issues BEFORE Ralph Nader came out with "Unsafe at Any Speed" - they are treated as a minor annoyance and "cost of doing business.

Sometimes, though, public ire is aroused and the result is a MASSIVE financial penalty such as Toyota was just assessed. I recall our various threads when the Toyota story first hit the news four years ago. I had a comment then that Toyota would probably regret not being more open and transparent when the problems first arose.`Even I never imagined the price of a coverup would be so high!

When I first entered the investment banking business nearly 45 years ago, I heard tales of how easy it was to make a quick buck (like the characters in "Wolf of Wall Street.") One old boy I considered a Mentor back then gave me a little aphorism which burned into my brain and has remained bold and clear ever since:
"He who takes what isn't his'n must give it back or go to prison."

Relating that to the topic of this thread:
Suppliers who cheat are cheaters plain and simple. Registrars who fail to detect the cheating aren't bad or incompetent - they aren't paid to perform a forensic audit. Customers who don't take steps to verify the supply chain are doing the equivalent of leaving an unlocked auto in a bad neighborhood with the motor running with the keys to house and office on the ring with the address of both places clearly marked on the ring. Thus, the burden is on customers to recognize registrars for what they are - organizations which verify honest organizations are doing what they say; that registrars are not paid to detect outright criminality, only "lapses" or pure incompetence. Certainly, if Toyota had been undergoing 3rd party audits, the entire "acceleration" issue would never have popped up, since the paperwork said Toyota was "dealing with" complaints. Certainly the sufficiency of that dealing would not have been a focus of an audit.

How does a customer "verify?"
There are myriad ways, running from pure statistical data gathering to sending out a qualified inspectors and investigators to look at supplier operations for hints or clues that some topic should be forensically investigated. It all depends on how crucial an individual supplier is to the customer's operation.

 

[MIREGMGR]

Registrars who fail to detect the cheating aren't bad or incompetent - they aren't paid to perform a forensic audit. (...) the burden is on customers to recognize registrars for what they are - organizations which verify honest organizations are doing what they say; that registrars are not paid to detect outright criminality, only "lapses" or pure incompetence.

In other threads, I think the P.I.P. silicone implant scandal has been discussed. European courts have held that P.I.P.'s ex NB was and is civilly liable to patients that received P.I.P. implants. So far the ex NB has been ordered to pay EU3000 to each of 1700 patients as an interim health-maintenance cost, irrespective of the outcome of further appeals. The ex NB's full liability could be as great as EU16,000 to each of 400,000 patients, or upwards of EU6 billion, if the courts allocate all of the asked-for penalty to the ex NB since P.I.P. is bankrupt.

The upshot of this is: at least in Europe, courts do regard NBs as legally obligated to detect cheating and outright criminality, whether they're nominally paid for it or not.

 

[Reg Morrison]

In fairness, and in relation to PIP case, so far ONE (French) court ruled to keep the NB liable to the deceipt. Other (German) courts have ruled in favor of the (German) NB. The case will progress in the Europen Judicial system and it could be earth shattering in terms of liability jurisprudence for NB's.

 

[Wes Bucey]

In other threads, I think the P.I.P. silicone implant scandal has been discussed. European courts have held that P.I.P.'s ex NB was and is civilly liable to patients that received P.I.P. implants. So far the ex NB has been ordered to pay EU3000 to each of 1700 patients as an interim health-maintenance cost, irrespective of the outcome of further appeals. The ex NB's full liability could be as great as EU16,000 to each of 400,000 patients, or upwards of EU6 billion, if the courts allocate all of the asked-for penalty to the ex NB since P.I.P. is bankrupt.

The upshot of this is: at least in Europe, courts do regard NBs as legally obligated to detect cheating and outright criminality, whether they're nominally paid for it or not.

In fairness, and in relation to PIP case, so far ONE (French) court ruled to keep the NB liable to the deceipt. Other (German) courts have ruled in favor of the (German) NB. The case will progress in the Europen Judicial system and it could be earth shattering in terms of liability jurisprudence for NB's.

Thanks for this - I'll follow it. I should point out that the bare bones facts of the case would not have gotten past the preliminary hearing stage in a USA court in recent years, but there has been a rising tide throughout the world that
"ANYBODY WHO IN SOME WAY PROFITED BECAUSE OF A FRAUD OR PARTICIPATED IN A POST-FRAUD COVERUP CAN BE HELD LIABLE FOR DAMAGES TO THE INJURED PARTY."

If, however, the parties alleged to be part of the conspiracy or cover up can demonstrate that they participated in good faith due diligence and were themselves cheated and lied to by the perpetrators, they have a good chance of being dismissed from the case.

I suspect the Napoleonic Criminal Code of France (Guilty unless proven innocent) may have a lot more to do with the outcome in France than the pure merits of the case.

In the P.I.P. case, the defendant "auditor," TUV, did much more than merely audit the operation, despite being lied to by the manufacturer, they went on to certify the PRODUCT safe for sale. Even in the USA, sticking one's neck out for a manufacturer in that manner requires much more due diligence than the French court believed was done by TUV. In MY mind, I would compare this to a certification by Underwriter's Laboratories, who actually test the products on their own premises before granting their aegis. I don't know enough about the case to determine whether TUV actually did off-premises testing and whether the test results (done by whoever) were done on the implants with commercial grade silicone rather than with hyped up samples of pharmaceutical grade substituted by the actual fraudsters.

In the criminal trials against the fraudsters, even the French courts found the fraud was also committed against TUV, not only the women who received implants. Civil trials often hinge on different evidence than criminal trials.

Civil cases drag on a lot longer than criminal cases. It may be years before all the dust settles to learn the final outcome.