When do we tell customers and regulators we have a problem?

Wes Bucey

Prophet of Profit
I noticed two small items by AP news bureau late Friday regarding a problem with the giant "atom smasher" recently put online in Switzerland. Coincidentally, my daily newspaper had a big editorial extolling the wonders and glories of the same device.

I'm providing links to the items and printing an excerpt from each ("fair use" for review, comment, or teaching purposes.)

My primary question is:
When do officials have a duty to disclose problems with a highly public project or process, especially when there is insufficient information whether the problem is minor or symptomatic of something much bigger and potentially more risky for people and environment?

The end isn't near! ((broken link removed))

We're talking about Planet Earth now. Not just Wall Street.

They've turned the Large Hadron Collider on. It's running. So far, the universe still seems to be in its appointed space, give or take.

Told you so.

For those who avoid bulletins from the woolly world of physics, a little background: The collider, built near Geneva, Switzerland, is a big, expensive (about $8 billion) machine that flings protons around a 17-mile race track, buried 330 feet underground, and watches what happens when they collide at nearly the speed of light. (Interesting note: The LHC really only accelerates the particles a tiny bit faster than Fermilab's now-overshadowed collider, relatively speaking, but that makes a big difference, we are told.)



It is supposed to re-create conditions last seen a trillionth of a second after the Big Bang. Or, some critics warned, when it gets fully revved up several months from now, it could end the universe as we know it. But that's only if the collider misfires and disgorges something called a "strangelet" that could transform this planet into a giant lifeless lump.
<SNIP>

(https://hosted.ap.org/dynamic/stori...R_QA?SITE=NYSAR&SECTION=HOME&TEMPLATE=DEFAULT) <SNIP>The European Organization for Nuclear Research says its new particle collider has been damaged worse than previously thought and will be out of commission for at least two months. On Thursday, the organization said the collider — the world's largest — malfunctioned within hours of its launch to great fanfare, but its operator didn't report the problem for a week. Spokesman James Gillies says experts have gone into the Large Hadron Collider to examine the damage. Gillies said Saturday the part that was damaged will have to be warmed up well above absolute zero so that repairs can be made. He said that will require having to shut off the new particle collider do to the repairs. <SNIP>
Q&A about problems with Large Hadron Collider
9/20/2008, 8:58 a.m. CDT (https://hosted.ap.org/dynamic/stori...LIDER?SITE=WHIZ&SECTION=HOME&TEMPLATE=DEFAULT)

The Associated Press
GENEVA (AP) — The damage to the Large Hadron Collider on the Swiss-French border will delay for at least two months the quest for scientists to learn more about the nature of the universe and the origins of all matter.
Following are questions and answers about the LHC, what happened to it and what it might reveal after it is repaired.<SNIP>
Q: What went wrong?
A: The temperature in one of the eight sections started rising slightly. Experts determined that a large electrical transformer which handles part of the cooling in that sector had failed and replaced it last weekend. Then they recooled the machine.
Q: Why do they think now the problem is worse?
A: There was still something wrong after the recooling, so experts went down into the tunnel to inspect it. There appears to have been a faulty electrical connection between two magnets that led to a leak of supercooled liquid helium.
Q: Are such failures unusual?
A: No, they can happen in any particle accelerator or collider, and they are expected in the LHC, said to be the most complex machine ever. However, failures can be more difficult to repair in the LHC because it uses such low operating temperatures. That can require a slow, controlled warmup taking several weeks before the repairs can be made.<SNIP>
 
M

MIREGMGR

When do officials have a duty to disclose problems with a highly public project or process, especially when there is insufficient information whether the problem is minor or symptomatic of something much bigger and potentially more risky for people and environment?

The first news story you quote included a mention of the long-standing allegation that the supercollider will generate a growing black hole and destroy the Earth. That story is popular with the press, even though it originated with the cranks and it's hard to find sensible scientists that believe it. If that's the "problems" that might be disclosable, well, it's been a subject of discussion for years already. It was a subject of discussion before they broke ground, and for a long time before that. So it's been "disclosed".

If OTOH you're referring to their bad transformer, wiring problems and liquid helium leaks, as far as I know those were announced to the press within a day of discovery, or maybe sooner.

Probably every industry has one or two well-known situations annually that would fit your general discussion-construct, though. I'm worried about the estrogen-mimicking of bisphenol A, used in food can linings and water bottles (very common but not relevant to my employer) and in all sorts of rigid transparent injection-molded medical devices and carbon-fiber-epoxy composites (both very relevant to my employer). I also worry about phthalate plasticizers used in flexible vinyl plastics. It's the classic "unquantifiable-but-small-risk-of-very-great-harm" risk management problem.
 
M

MIREGMGR

Let me try to spin your proposed problem slightly differently:

Do businesses have an ethical or civic duty to "disclose" information that official, or mainstream scientific, authorities have concluded does not represent a significant risk, but that fringe scientists or technically unknowledgable members of the press or the public do regard as a risk?

Suppose the dispute is between scientific stances of the official authorities, who have determined that there is no significant risk, and a significant body of mainstream scientists who hold an opposing view that a major risk does exist. Does that change the ethical or civic duty?

Suppose the governmental body that regulates you has determined that there's no significant risk, but an opposing view is held by a major scientific agency within the same government, plus other nations' governments' top scientific agencies.

Suppose you sell both domestically and internationally. A risk is recognized in other countries, but not your own. You have to comply for your export product, of course. Do you "disclose" any information to your domestic customers about the alleged risk and that your stances domestically and internationally are different?

None of the above is hypothetical. That's the real status of the bisphenol A problem. The FDA says it presents no significant risk. The Japanese government's evaluation agrees. The US government's National Institute of Health National Toxicology Program concluded that a risk exists...not of a very large magnitude in absolute terms, but nonetheless greater than levels that for other chemicals in the past have resulted in bans, recalls and consumer lawsuits. The Canadian government agrees, and is moving in the direction of imposing a ban for some uses.
 

harry

Trusted Information Resource
This may be a bit :topic: but in the same vein.

Years back, when I was the treasurer of our local NGO club, I brought in a new accounts person to help with the accounts / accounting system. In the course of the year, some problems were discovered - classification, treatment and other technicalities and it was promptly reported to the club. These can be expected because we do not have a full time or professional guy to handle it.

After the audit for that year, I felt obliged to discuss with the auditor on this issue and my accounts person felt that the auditor should be informed because they should had discovered it after so many years of auditing.

When the report comes back and to our amazement, there was a several page supplementary report regarding the issue we informed them. It was written in a way as if they had discovered it instead and we were the party in the wrong.

Talk about borrowing your watch to tell you the time and this is a local branch of one of the top five. I think I understand why some people start to cover-up and ended up with more cover-ups.
 
M

MIREGMGR

When the report comes back and to our amazement, there was a several page supplementary report regarding the issue we informed them. It was written in a way as if they had discovered it instead and we were the party in the wrong.

Perhaps the lesson from that situation would be, if you decide that you're obligated to disclose, make sure your disclosure goes simultaneously to enough different recipients that it's impossible for there to be any funny business regarding credit, blame and the situational facts.
 

harry

Trusted Information Resource
Perhaps the lesson from that situation would be, if you decide that you're obligated to disclose, make sure your disclosure goes simultaneously to enough different recipients that it's impossible for there to be any funny business regarding credit, blame and the situational facts.

You are right there. Fortunately, I had obtained approval from the club for changes to be made and so every member were aware of what's happening.
 

Wes Bucey

Prophet of Profit
The first news story you quote included a mention of the long-standing allegation that the supercollider will generate a growing black hole and destroy the Earth. That story is popular with the press, even though it originated with the cranks and it's hard to find sensible scientists that believe it. If that's the "problems" that might be disclosable, well, it's been a subject of discussion for years already. It was a subject of discussion before they broke ground, and for a long time before that. So it's been "disclosed".

If OTOH you're referring to their bad transformer, wiring problems and liquid helium leaks, as far as I know those were announced to the press within a day of discovery, or maybe sooner.

Probably every industry has one or two well-known situations annually that would fit your general discussion-construct, though. I'm worried about the estrogen-mimicking of bisphenol A, used in food can linings and water bottles (very common but not relevant to my employer) and in all sorts of rigid transparent injection-molded medical devices and carbon-fiber-epoxy composites (both very relevant to my employer). I also worry about phthalate plasticizers used in flexible vinyl plastics. It's the classic "unquantifiable-but-small-risk-of-very-great-harm" risk management problem.
Actually, the part of the story which caught my attention was:
On Thursday, the organization said the collider — the world's largest — malfunctioned within hours of its launch to great fanfare, but its operator didn't report the problem for a week.
As I was reading more and more about the latest "food adulteration" (China's milk/melamine today, but it could have been anything, since even one of Japan's most prestigious milk producers had a similar scandal EIGHT YEARS AGO - see below), it occurs to me the worst part about these scandals is always how the culprits and even the government regulators create even more fear and panic by trying to limit the news and almost always, the first reports include some phrase to the effect the "problem" poses no danger to the public. Yet, the problem continues to get more and more horrific as news leaks out about the actual damage, and how widespread the problem really is, even to the point where fear mongers spread the panic by starting false rumors for their own nefarious benefit.

I'm savvy enough to know "some" folks are actively selling shares short on dozens of companies having anything to do with China's dairy industry while simultaneously buying up shares in companies which have zero links to China's foodstuff industry. That alone doesn't make them bad, but there are also those few individuals who are out there actively repeating any damaging stories and adding "enhancements" to make the problems seem even more horrific and widespread.

If the culprits and the regulators would be less secretive, the rumor mongers would be less able to unroll "conspiracy" theories about how the culprits and the regulators are in cahoots.

These "coverups" never work in the long term, because the story ultimately DOES come out, sometimes inflicting even worse damage on the companies which first sought to hide the problem and, certainly, individuals and executive teams within the culprit companies often have permanent damage to their careers as a result of being involved.

We in the Quality industry have a responsibility to the truth. Often, that responsibility is overshadowed by our own fears of reprisal if we speak a truth the bosses don't want heard. Some time back, we covered this topic at great length in the thread Ethics - Moral law vs. Criminal law .


So . . . I can understand [but not condone] an individual working for the company who bites his tongue and doesn't speak up, but I absolutely have no sympathy or understanding for the government regulators in ANY country who go along with the idea of "controlling" the extent of the disclosure.

Here in America, we just finished a news cycle about recalling dangerous baby cribs and the recurring theme as the story grew and grew was that government regulators meekly went along with the culprit companies in trying to keep a lid on the extent of the disclosure and recall.

Worse, some news outlets willingly went along with a theme the culprits tried to peddle that anyone who feared for their child's life or well-being was a "crank" talking about a "remote possibility" even in the face of the truth that the story emerged because children had actually died.

What do we Quality professionals need to do?
As I see it, our first responsibility is to educate EVERYBODY, bosses, politicians, and, especially, consumers, about the importance of PREVENTIVE measures to ensure life, health, safety of end users of any product. That taught, we need to help those same people learn how to investigate to ensure preventive measures are in place to protect them. That means we DO NOT lump in honest mistakes with criminal acts. An honest mistake needs to be disclosed as soon and as widely as possible to maintain confidence the system "works."

Criminal, purposeful acts can be limited by a system of checks and balances, reasoning the cost of the checks and balances saves money in the long run - I imagine the lifetime care of the thousands of children affected by this latest food adulteration in China will far outweigh the criminal profit the bad folks [who introduced melamine into the foodstream] have made. I am equally certain the cost for the kind of routine checks for quality which may have detected food adulteration would have been MUCH less than what the OEM companies will pay in reparations to afflicted victims of the melamine.

My current understanding of the reasoning for introducing melamine into the food chain is that it gives a false positive for protein content of the food, allowing the criminal to dilute the product with water and thus sell greater volume of product, essentially selling water at milk prices.

To my way of thinking, the folks who SOLD the melamine are equally culpable with the ones who bought it and put it in the food chain. Perhaps part of the preventive measure might be to determine what products can give false positive readings for desired qualities and, first, establish a method to test for adulteration and second, put producers of the substance on notice (much as American producers of the substance pseudoephedrine (https://en.wikipedia.org/wiki/Pseudoephedrine) are put on notice to limit who may purchase and how much quantity at a time may be purchased.

One requirement which would help a lot is true traceability and "chain-of-custody" data on every product, a process which is becoming economically feasible because of computers, advances in printing micro bar and 2-D codes of products. Hard data from such traceability would help limit recalls to actual suspected nonconforming products instead of wide recalls of EVERYTHING.

The best part, in my estimation, is the actual savings from using such a program can be demonstrated, especially when you throw in the psychological VALUE to lure customers to a supply chain which displays such attention to ensuring customer safety.

I don't advocate punishing mistakes, but I sure do advocate punishing purposeful fraud and harmful activity.



First death reported in Japan's milk scandal

Last Updated: Friday, November 10, 2000| 11:57 PM ET
(broken link removed)
Japan's contaminated milk crisis is getting worse and may now be responsible for one death. The problems started two weeks ago when infected milk was sold by Japan's leading producer. But now the whole dairy industry is under suspicion. And the Japanese public is questioning the safety of all its food.
An 84-year-old woman who became sick after drinking tainted milk more than two weeks ago has died.
Police say she may be the first fatality in a contaminated milk crisis that has made about 14,000 people ill. They're now considering charges against Japan's leading milk producer of negligence, resulting in death.
The company, Snowbrand, has closed all 21 of its plants in Japan and they'll be inspected next week by government health officials.
Now other milk companies are also coming under suspicion after media reports that hygiene standards throughout the industry are lax.
Media reports suggest that the Japanese public, which used to think that its food industry had among the highest safety standards in the world, is very concerned.
The latest batch of media reports include a dairy company in the north of Japan which recalled packages of cheese tainted with pvc (polyvinyl chloride) — and a noodle maker in southwest Japan who recalled 150,000 packs of mouldy spaghetti.
 
M

MIREGMGR

Actually, the part of the story which caught my attention was:
Quote:
On Thursday, the organization said the collider — the world's largest — malfunctioned within hours of its launch to great fanfare, but its operator didn't report the problem for a week.

It's funny how one can read something, and come away with an understanding that differs from the words they just supposedly read, based on what they think they saw earlier. I completely missed the above sentence in your post, Wes, which obviously differs from the report I had seen elsewhere to the effect that the problems had been disclosed promptly, albeit not as widely as later became the case.

In any case, the problems with the LHC don't really have a public safety aspect, unless you believe the esoteric black hole supposition.

Chinese contaminated milk certainly has a public health aspect. My understanding in that case is that, at least in the current instance, the usual practice was to informally source the melamine (technically: reacted urea-formaldehyde resin) from junkyards in the form of scrap Formica-type plastic sheet, and grind it into powder using similarly informal approaches. There is no commercial use for already-reacted, powdered urea-formaldehyde resin...it's a thermoset and can't be melted or reacted again...so anyone wanting such powdered material would have to have it custom-ground, or custom-grind it themselves.
 

harry

Trusted Information Resource
These "coverups" never work in the long term, because the story ultimately DOES come out, sometimes inflicting even worse damage on the companies which first sought to hide the problem and, certainly, individuals and executive teams within the culprit companies often have permanent damage to their careers as a result of being involved.

Snowbrand had been a favorite name for those teaching CSR (corporate social responsibility). If you are case-study nut like me, there's a paper here.

And if you want to know what happened to those who fail to learn their lessons, read this: Japan’s battered Snow Brand Food to shut up shop by April
 

Wes Bucey

Prophet of Profit
It's funny how one can read something, and come away with an understanding that differs from the words they just supposedly read, based on what they think they saw earlier. I completely missed the above sentence in your post, Wes, which obviously differs from the report I had seen elsewhere to the effect that the problems had been disclosed promptly, albeit not as widely as later became the case.

In any case, the problems with the LHC don't really have a public safety aspect, unless you believe the esoteric black hole supposition.

Chinese contaminated milk certainly has a public health aspect. My understanding in that case is that, at least in the current instance, the usual practice was to informally source the melamine (technically: reacted urea-formaldehyde resin) from junkyards in the form of scrap Formica-type plastic sheet, and grind it into powder using similarly informal approaches. There is no commercial use for already-reacted, powdered urea-formaldehyde resin...it's a thermoset and can't be melted or reacted again...so anyone wanting such powdered material would have to have it custom-ground, or custom-grind it themselves.
For the sake of debate, IGNORE any "black hole" aspects of the atom smasher.

There is a lot of public and private money involved in putting a multi-billion dollar facility like the Large Hadron Collider (LHC) up and running. Immediate announcement of a glitch (together with an explanation that small glitches are part of the "tweaking" process and help illuminate events to allow operators to create "mistake proofing" procedures to reduce the impact of any future event) would have made the incident a small story that it should have been.

We are Quality professionals here. We all realize the FMEA (Failure Mode & Effects Analysis) process is more guesswork than science because it depends so much on imagining "what if" scenarios and trying to attach an "importance value" to each one. I have continually and consistently written and said, "Knowledge is power!" A corollary of that motto is, "Knowledge drives out fear of the unknown."

Hundreds of years ago, Shakespeare wrote (in Hamlet),
"And makes us rather bear those ills we have
Than fly to others that we know not of?"

demonstrating even then audiences would understand the natural proclivity of humans to face a danger they understand and know about rather than one their imaginations create (black holes?).

After weeks of publicly flogging the value and importance of the LHC, the operator was most likely embarrassed to report the little glitch, proving that even a supposedly minor decision to avoid temporary embarrassment creates an opportunity for even greater embarrassment down the road when it is discovered the responsible people were too cowardly to face public scrutiny.

I readily forgive mistakes. I rarely forgive hiding a mistake, especially when hiding the mistake creates more problems down the road.

Imagine a more prosaic example of hiding a mistake which can create more trouble down the road. This is an actual example of one which started small and ballooned to a $100,000 loss of income.

(This event happened at a competitor)
A CNC machine operator made a mistake in setting up a machine to run a relatively simple part for an important customer. The customer had requested an elaborate SPC analysis of the part as part of a projected "dock to workbench" JIT program.

The operator was confident in his setup, but too "busy" to pull samples after the first piece, measure them, and complete SPC charting. Accordingly, he "fudged" the SPC charts to make them look the way he thought they should look.

After eight hours, he "graciously" pulled a few pieces from the finished parts bin to measure them, only to discover, to his horror, that one of the settings on the machine had "drifted" due to his error in setup and the random samples were undersize on a critical dimension. As he frantically went through the bin of over 1,000 parts he found some that were oversize and some undersize. No supervisors were in sight, so he rolled his bin to the scrap hopper and dumped them all, pulled more raw stock and began to cut more, this time checking twice as frequently as the original SPC plan called for. He put in four hours of overtime and cut about half as many as he had in the eight hours preceding.

When he came to work the next day, he went to pull more stock, only to discover the stock bin was empty. After a frantic search, he discovered the bosses had only purchased enough stock for his order and another small order which had run during the night shift.

Now the operator couldn't finish the job because he had wasted half his raw stock allotment. He finally 'fessed up. The owner of the shop told the truth to the customer, explaining the balance of the order would take weeks because the special alloy was unavailable from his regular distributor.

The customer was gracious, but firm. In the customer's eyes, this was a systemic failure from top to bottom and the customer was not willing to trust a "dock to workbench" JIT supply chain in the hands of that machine shop and so canceled ALL its pending orders with that shop.

With the loss of over $100,000 in business, the owner was forced to cut back. Guess who got cut?

In his parting shot, the customer told the owner there would have been no problem if they had learned of the spoiled parts as soon as the operator knew, but the two day delay signaled deeper problems to the customer.

The customer came to us (telling the tale from his perspective), but the total business was not profitable enough for us, so we referred him to another machine shop which was able to meet his needs.

I, however, went to the machine shop owner who had lost the business and was impressed with his equipment and facilities, so I offered to teach him how to be "customer-centric" and not fear telling the truth so I could feel confident in feeding him overflow business.

Essentially, the only thing I had to help him learn was Deming's System of Profound Knowledge, where every member of his organization was aware of the big picture and his place in it. With that knowledge, they realized "nuisance work" like keeping SPC charts was actually not a nuisance, but a benefit. Within one year, we were feeding him $50,000 in business every month and happy with the work and the responsiveness of his team.

Oh yes, he grew to realize that the machinist who hid the damaged goods did so out of fear of reprisal for making a mistake. With that realization, he made it clear to all his employees he viewed mistakes as opportunities for learning and imporovement and that the faster they were reported, the faster they could work on a solution, eliminating all the wasted time in hiding or finger pointing..
 
Top Bottom