Did somebody drop the ball at Boeing?

[Wes Bucey]

Having dealt with Boeing as a customer on occasion in the past for parts to go into the 777, I recognize that the execution of their quality plan is often at odds with the content and intent of the plan.

Even so, I was more than mildly surprised when the FAA grounded ALL 787 Dreamliners this week.

One of the trade magazines to which I subscribe, Design News, carried an article concerning this event
([FONT=Arial, sans-serif]https://www.designnews.com/document.asp?doc_id=257519&print=yes[/FONT] )

Combining the information in the article with my own experience and knowledge about new product introduction, I intuit that FMEA (Failure Mode & Effects Analysis) was not rigorously performed. Here are a few meaningful excerpts from the article available in full at the link above

A succession of problems has plagued Boeing's 787 Dreamliner, but investigators are now most concerned about incidents involving overheating of lithium-ion batteries.
Federal Aviation Administration (FAA) officials grounded Boeing's high-tech Dreamliner after battery electrolytes reportedly leaked from a lithium-ion battery onboard an All Nippon Airways flight on Wednesday. The liquid reportedly traveled through an electrical room floor to the outside of the aircraft, leaving burn marks around damaged areas.
The latest incident followed on the heels of two battery-related problems encountered on Japan Airlines flights and another on a United flight earlier this month. Those incidents happened in parallel with multiple other episodes, including two fuel leaks. Since July, the 787 has also encountered a damaged cockpit window, an oil leak, and two cracked engines, according to multiple news reports.

Boeing appeared to be dissembling (avoiding the issue) just prior to the FAA decision when it proclaimed

On its website, Boeing emphasized the safety of its new aircraft, releasing a statement saying, "The airplane has logged 50,000 hours of flight and there are more than 150 flights occurring daily. Its service is on par with the industry's best-ever introduction into service -- the Boeing 777. Like the 777, at 15 months of service, we are seeing the 787's fleet wide dispatch reliability well above 90 percent."

Most quality professional folks (including me) are of the opinion there is no single direct root cause for the various "failures" which have cropped up, but I am of the opinion that management's decision to outsource the 787 components around the world without sufficient supplier evaluation and monitoring may be the underlying "root cause" of ALL the problems and failures noted to date.

As a Demingite, I can be expected to lay most of the responsibility at the door of top management, but even I am stunned by the apparent disconnect and callousness of the Boeing management in dealing with this, especially in this passage from their press release

For that reason, today we jointly announced with the U.S. Federal Aviation Administration (FAA) the start of a review of the 787's recent issues and critical systems. We welcome the opportunity to conduct this joint review. Our standard practice calls on us to apply rigorous and ongoing validation of our tools, processes and systems so that we can always be ensured that our products bring the highest levels of safety and reliability to our customers.

I agree the Boeing quality documents for standard practice absolutely call for "rigorous and ongoing validation," but the nagging question in MY mind is
"Did the employees who work directly for Boeing and members of their supply chain actually perform what the quality documents call for or was the performance riddled with lapses and shortcuts?"

The lithium-ion battery:
A great deal of attention has been focused on the overheating and leakage of fluids of the lithium-ion batteries used in the 787 (be sure to take a good look at the photo of the battery from the JAL 787)

Aviation experts said the energetic quality of lithium-ion can be a concern onboard aircraft. "One of the issues with lithium batteries is they get very hot," Freiwald said. "When they ignite, they can burn so hot that Halon 1301 won't extinguish a fire."
[snip]
Even with cooling, however, lithium-ion automotive batteries have been known to have problems on rare occasions. In 2011, a fire started in a Chevy Volt weeks after government crash testing, causing a ripple of concern. "The chemistry is edgy," Donald Sadoway of MIT wrote in an email to Design News after the incident. "The electrolyte is an organic fluid that is flammable, highly volatile at even moderately elevated temperature and in the presence of metallic lithium, which can form on the negative electrode at high charging rates."
Although it's not known whether the Dreamliner employs battery cooling systems, its batteries are smaller than those of plug-in hybrid cars. A National Transportation Board (NTSB) examination of an auxiliary power battery unit from the JAL Boeing 787 that caught fire in Boston's Logan Airport on January 7 showed that it measures 19 inches x 13 inches x 10 inches and weighs just 63 pounds. In contrast, electric vehicle batteries can weigh more than 400 pounds.

Just remember, as you consider this, that US TSA agents get hinked if a passenger attempts to bring a butane cigarette lighter on board an aircraft.

Added in edit: Boeing CEO statement (press release)

CHICAGO, Jan. 11, 2013 /PRNewswire/ -- Boeing (NYSE: BA) Chairman, President and CEO Jim McNerney issued the following statement today after U.S. Transportation Secretary Ray LaHood and FAA Administrator Michael P. Huerta announced that the FAA and Boeing will start a review of the 787's recent issues and critical systems:
"Boeing shares the same commitment to air travel safety that Transportation Secretary LaHood and FAA Administrator Huerta spoke of this morning in Washington, D.C. We also stand 100 percent behind the integrity of the 787 and the rigorous process that led to its successful certification and entry into service. We look forward to participating in the joint review with the FAA, and we believe it will underscore our confidence, and the confidence of our customers and the traveling public, in the reliability, safety and performance of the innovative, new 787 Dreamliner."
Contact:
John Dern
Boeing Corporate Offices
312-544-2002


SOURCE Boeing

 

[Atom_Ant]

Quick note regarding the batteries: there are different types of "lithium-ion" batteries with very different chemistries. The type used on the Boeing 787 is a cobalt oxide type which is lighter with higher energy density.

Electric cars, as far as I know, do not use cobalt oxide so they are heavier with lower energy output.

https://www.technologyreview.com/ne...eamliners-use-batteries-prone-to-overheating/

 

[MIREGMGR]

It's being reported today that Japanese investigators think the batteries are receiving a significantly higher-than-designed voltage during aircraft operation. They say they see evidence of this (particular overheating patterns) in two out of two aircraft checked so far.

It's hard for me to understand how that can be occurring at this late stage in development, in a production system as sophisticated and mature as for major aircraft. Engineering mistakes do occur, but surely (?) a mismatch between the very complicated, many-component electrical system operating voltage and the battery pack design voltage would have been noticed by someone...?

 

[Wes Bucey]

It's being reported today that Japanese investigators think the batteries are receiving a significantly higher-than-designed voltage during aircraft operation. They say they see evidence of this (particular overheating patterns) in two out of two aircraft checked so far.

It's hard for me to understand how that can be occurring at this late stage in development, in a production system as sophisticated and mature as for major aircraft. Engineering mistakes do occur, but surely (?) a mismatch between the very complicated, many-component electrical system operating voltage and the battery pack design voltage would have been noticed by someone...?

This really goes back to the concept of Configuration Management as I have been preaching for twenty-five years. When any document or design of a component of a product is changed, a competent person or team MUST check all associated documents and components for the effect of the change to assure there is no conflict between or among them.

This Configuration Management used to be a prime consideration of Boeing Quality requirements when I was selling components to them in the 1990s.

So, from your report, we might infer the system for charging the battery may not be compatible with the battery (begs the question, "Which was changed, if at all, from the original design and testing, battery or charging system?")

 

[andygr]

"Did the employees who work directly for Boeing and members of their supply chain actually perform what the quality documents call for or was the performance riddled with lapses and shortcuts?"
A bit of an inflammatory question in my view. A bit early looking to blame somebody even though we all like to blame the supplier or someone on the other shift.

We are a ?right now society? and do not want to wait for answers. Most suppliers know the most frustrating thing is to receive a C/A with a short response time requirement. It just takes time to get to the bottom of things and some time it takes longer than others. Remember many of these parts and systems were actually produced years ago driven by the delay in AC certification. It?s not like you can pull the records from this week?s testing . The records have to be pulled from storage.

What is needed is a solid plan with a time line reflective of the risk involved. With the grounding of the fleet the risk is mitigated but the pain to Boeing and the airlines remain and I am sure it is providing a pressure of its own that I would prefer not impact the actual investigation process but only help provide the resources needed in a timely manor.

Since in my personal view the key item that drove the FAA to ground the fleet is the Batteries.
I would like to see the investigation address why the issues with the Lithium battery was not seen till just this point in time. No reports during the years of pre type certificate testing or during the 50,000 flight hrs till this point in time.
Why now? What is different in the conditions to this point and the test parameters checked to this point in item and system qualification that is causing the batteries to fail?
Over charging by ground crews? Defective battery lot produced by the supplier (if so why did ATP test not find the issue)? Throw you own guess in with lots of questions that I am sure everyone possible is chasing to the ground with 50 180 mil dollar aircraft sitting on the ground.

As in the fuel used to run the engines these batteries store energy at high densities. When any high energy density system storage container or distribution system fails to manage the energy storage or transfer properly bad things will happen.

Bottom line I like Boeing and the FAA to be cautious and follow the proper analysis that we as QA professionals like to harp about and not jump to conclusions or preconceived paths. Keep everything on the table as a possibility and properly check each item out. Till then act on the side of caution and ground the fleet since any ignition source in an aircraft including bic lighters is something to avoid.

Anyone who travels a lot who thinks that the number of issues with the 787 aircraft is out of the ordinary just think back on the number of flights you were scheduled on have been delayed or canceled to a mechanical issue (yea I know we think that the airlines do this to avoid flying a less than full flight). Not a month goes by that I am not subject to one.
Just do what it takes to keep me safe since I am on these flying tubes way to much of the time.

 

[Atom_Ant]

So, from your report, we might infer the system for charging the battery may not be compatible with the battery (begs the question, "Which was changed, if at all, from the original design and testing, battery or charging system?")

I believe the reasons Boeing would choose batteries like this (extremely high power:weight ratio and the ability to be produced in a variety of shapes to fit specified areas) are pretty unique to the type of battery used. I don't think it would squeak by as a minor redesign to replace them with another type.

 

[Randy]

New aircraft, krap happens, it'll all get worked out.

If that seems cold to ya, get real...Stuff happens when you have 1,000,000 things glued, welded, fastened and pressed into one big thing. Aircraft are complicated, systems get stressed and stuff breaks, it's as simple as that. We were finding design and engineering problems and flaws in Bell 204's 40 years after the 1st flight test (1956)...For y'all that don't know what a Bell 204 is think Huey helicopter. The bathrooms on the 787 are more complicated than a Huey, and they'll probably break as well!

After over 5,000 crew hours in rotary wing aircraft, most of which was in Bell 204/205 Hueys and close to 3,500 hours of maintenance test flight crew-time I can tell you that no matter how well designed, engineered, built, maintained or fussed over, aircraft break (In addition to Hueys I have seat time in AH1 Cobra's, AH64 Apachee's, UH60 Blackhawk, CH47 Chinook's, OH58 Kiowa's, OH6 Cayuse, CH54 Tarhe, OV10 Bronco, U21 Ute Queen Air, C12 Huron King Air and a couple other airframes)....They were designed well, flew lots of hours and they still broke.

The ugly downside isn't the plane's breaking, they're new and it's early in their history, it's that people have the plane strapped to their butts, that's the real downer.

 

[silentrunning]

I still feel WAY,WAY safer flying commercial than I do driving on I-75.

 

[andygr]

Put me in the air over the south side of Chicago any time. I feel more at risk heading into Miday on the roads that the flights- allthough the landing is like comming down on an aircraft carrier.

The south side drivers are just a bunch of kamikazi drivers looking for an accident.

I would say the express ways into down town are accidents waiting to happen but the trafic never moves on these roads fast enough to cause any damage other than death by old age.

 

[gpainter]

If I remember correctly. Were they not way behind on their delivery schedule?