ECO (Engineering Change Order) Risk Management Criteria and QA

[dwend]

Hello to all:

First post on the forum so here goes....

This post is an intersection of aviation, document change, configuration control, FMEA, design, and possibly others but I am posting here.

I am working for a small aviation/aerospace job shop as a QE. Anyway, there is a large number of ECO's that are generated as a % of units sold/jobs/revenue etc and is a source of internal failure costs as well as turnbacks, time lost etc and I am in the process of reviewing the whole ECO process. In addition to tuning up the whole "document change" and configuration control processes, I am looking to incorporate some more objective criteria for ECO's to facilitate approvals and minimize risk.
The firm technically does not have "design authority" on many jobs but does own a number of drawings on lower level assemblies and has some lattitude on "changes." Most of these changes are triggered by obsolete parts from suppliers, some for cost savings, and other minor reasons. Of course we are starting to track the cost of quality for some of these changes but I am interested in "front loading" the change process with some more specific justification/criteria for analyzing changes...sort of a mini FMEA approach embedded within the ECO process. I am not interested in handcuffing Engineering so much as I would like to be a little more thorough on evaluating the risks and justifications and alternatives. Some of the criteria would include:

Where else used
Mil or other specifications
Supplier test data,


Most of the "questions" raised during these changes would go away if initiators would be a little more thorough on the front end. I will avoid internal political discussions but I simply want to take some of the guess work out of the approval process and be more thorough.

In the age of global cost competiveness, cheap imports, etc...the risks of quality lapses are very real despite ISO and other certifications. I do not want layers of paperwork but want to be more clear on what is allowed in the ECO process. Any suggestions on what the perfect ECO checklist for risk management would look like?

Again, I have a very good form already that calls out reasons, descriptions, class of change and so on. But what I see lacking is more detail on risk mitigation - objective evidence for justification on that gray area between really minor and major changes that would normally trigger a more thorough review. I am simply trolling for additional thoughts and ideas on the subject.

Thx,
DW

 

[George Weiss]

The [Search] button feature on the top of the page finds similar Elsmar Cove discussion threads.
There is an exact match to your question/interest.
Please review the link provided, with comments and attachments:
https://elsmar.com/Forums/showthread.php?t=38095

 

[dwend]

Thx for the link to this thread. I did actually do the search before I posted my question and am familiar with this thread and yes it does cover some of my question. I was really trying to focus on what others may be using to deal with specific questions regarding risk management of the change order process and how much evidence is required before approvals. For instance if a process engineer or materials engineer says that the BOM or drawing will have to be updated because a supplier has gone out of business or a part has been obsoleted, how much "legwork" should be in evidence before Quality approves a change. Or if a suppliers specification has changed since the last order point etc. I realize the circumstances may be highly variaable but was wondering if others have "brainstormed" this requirement and come up with a specific formulated criteria. I certainly plan on doing this in our own organization and will have no difficulty with others in the approval process in formulating a reasonable standard that matches our needs but again just trolling for any recent ideas on the specifics of this subject. I think there is more to the ECO process than just an elegant form or flow chart. The right questions need to be asked and answered here.

DW

 

[RosieA]

Does your process include inputs and sign-off from multiple disciplines? If so, you might do a bit of research and see what happens within each discipline to implement the ECO. Those disciplines could probably help define what is missing when the ECO gets to them. That becomes your checklist.

One thing I don't see mentioned in your potential issues is material disposition for the older material. this can be a real cost issue with some products.

 

[dwend]

Thx RosieA for the response,

Yes there are multiple signatories including all the usual suspects but because of "lean" conditions it usually comes down to Quality following up on Process Engineering changes and asking a lot of questions that in theory should already be addressed in my estimation in the reason for change (or maybe a another block that includes supporting evidence/justification). I am really looking to add in the ECO (with concurrence from other parties) a requirement that the change has been adequately vetted. I am new to this organization and there seems to be some animosity that was in place between certain parties and I will leave it at that if you catch my drift. I am looking to codify some of the requirements so as to minimize the questions.

Yes the form is quite complete including production stage, disposition, class of change, and so on. Yes you are right about questioning the process involvement of others. I am used to semiconductors where the Process Engineering manager, device manager and production had supreme authority and QA was relegated to stamping incoming and outgoing materials. In Aviation and Automotive, the QA function seems to carry more weight and has to snoop around more for evidence that things are done properly. I am looking to formalize some of the gray areas but do not want to become regimental either. Thx again....

 

[Wes Bucey]

George has been helpful, but there are many more threads more relevant to your query. Back in 2003, I wrote

There is an SAE Standard
Revision of Engineering Drawings and Associated Documents Y14.35M-1997
available from the Society of Automotive Engineers, which discusses the issue of changes to documents and how to number them. The treatment of the terms version versus revision is pretty much ignored.

When a document (part drawing, text, Procedure, etc.) is revised to a point where it is completely different from the original and "Configuration Management" (search this term in the Cove for my other discussions on CM) determines the document deserves a new name or number rather than an additional "revision" designation, most organizations do not bother to keep any reference to the original in the naming or numbering.

There is discussion on this process in the Standard.

An interesting point is that the Standard suggests Revisions be given an alphabet letter rather than a number (A, B, . . . AA, AB, . . . XX) The Standard is quite clear an organization may adapt the Standard to its own purposes!

In 2004, I wrote this

One of the things most folks know about me is that I try to take advantage of the expertise of others who have gone before me. Therefore, I rarely try to reinvent the wheel.

When it comes to creating, revising, and controlling documents, I am pleased to say there are Standards compiled by teams of experts which we can follow to reduce the possibility we have overlooked an important factor in the process.

For experienced practicioners, they may seem "obvious," but without them as a template, I am sure I might not be as effective as I am with them.

Current Standards (in America) are created and maintained by (among others) American National Standards Institute (ANSI), American Society of Mechanical Engineers (ASME), American Society for Testing & Materials (ASTM), Electronic Industries Alliance (EIA), Institute of Electrical & Electronics Engineers (IEEE), Society of Automotive Engineers (SAE), and Institute for Interconnecting and Packaging Electronic Circuits (IPC).

Three important Standards which apply to creating, modifying, and maintaining/controlling documents are:

• ASME Y14.100-2000 Engineering Drawing Practices
• ASME Y14.35M-1997 Revision of Engineering Drawings and Associated Documents (there may be a more recent edition - I'll check later and revise this list if necessary)
• ASME Y14.34M-1996 Associated Lists

All reputable software programs for controlling documents adhere to these Standards. U.S. Government contractors are compelled to adhere to these Standards when creating and maintaining documents for the government. Commercial customers "appreciate" suppliers who follow these Standards.

All that said, Standards are written by committee and are sometimes indecipherable by us ordinary folk (viz. the thousands of words written to help decipher ISO9k2k.)

It is important to keep in mind that folks created and maintained documents long before computers. Software merely replicates the simple tasks which were performed by real people.

The primary quality which permeates ALL the tasks is "consistency" - that the task is done the same way EVERY time. The value of computers is that they will ALWAYS perform a task the same way unless a human intervenes.

In the case of modifying/changing/revising documents, the key is that ONLY authorized folks may change a document and they MUST follow a protocol of getting approval before releasing the document to be acted upon. Simultaneously with the release of a changed document, all involved parties must be put on notice of the change, and (ideally) all obsolete versions are retrieved and destroyed (or at least stamped "OBSOLETE")

The question here is not that anyone is giving "wrong" information, but that this Forum format does not facilitate "complete" information. What our experts write here are really just "clues" to help in directing research on a topic. __________________

Later, in 2007, this aspect arose

I have just began working for my new employer and I am preparing an assessment of their document control system. The first item that stood out to me was the numerous amounts of engineering change orders within days of a drawing's initial release. This is waste and not typical in my experience. Any ideas or methods to reduce this wasted activity? Appreciate it!

Yeah. This happens in a lot of organizations where someone gets an inkling that changes to a document need to be formal and then they go overboard in trying to affix a change letter each time the original author or one of the approvers of the document gets a new bright idea for an amendment or deletion to a document.

Here's the way many efficient organizations handle the system:

1. somebody has an idea for an original document
2. he or someone else is selected to author the document
3. the first draft of the document, identified as "DRAFT #1" is circulated among one or more other folks for either approvals as is or suggestions for changes
4. if there are no suggestions or changes, the document is approved and "released for use" with ANY naming or numbering system which makes sense to the organization.
5. if there are suggestions for changes, they may be discussed among the approval group or with experts outside the initial approval group until a consensus is reached.
6. once consensus is reached, document goes back to original author for the next "DRAFT #2" and then cycles through the approval process again and again until ALL approvals are in place, at which time, the naming or numbering changes from "DRAFT" to the nomenclature used in approved documents.

The key is not to give a formal number to a document UNTIL it has gone through the "DRAFT" and "APPROVAL" stages.

Sometimes, after all approvals and formal release of the document for use, but BEFORE it is implemented, somebody (whether in the original approval group or not) discovers an error or suggests a change. Many organizations may make a "temporary" approval of the change, often called a "redline change" to allow use to go forward because the time lag of going through the formal approval process might cause a cascading effect on other processes. A "Redline change" should never be made as a matter of regular practice and then only when sufficient information is available to assure the "formal approval process" will accept the redline change as a new revision to the document.

Similarly, "redline changes" may take place AFTER implementation of the document for the same reason outlined above.

Most organizations which resort to "redline" also have an expedited approval process to rapidly go through the formal approval process to issue a new revision to the document.

My experience with "redline" is primarily in "aftermarket aerospace" where an engineer overseeing installation on an individual aircraft will note some characteristic of the aircraft not previously known when the installation documents were prepared. A flurry of phone calls, emails with photos, conferences with aircraft owners, etc. ensues and a decision is made to "redline" the documents to complete the installation and not delay an aircraft on the ground any longer than absolutely necessary. The part responsible for creating the original document then prepares a formal revision which catches up to the aircraft sometime later and the redline copy is withdrawn.The FAA condones this practice by allowing "minor" changes to such documents to be reported to the FAA AFTER the fact, whereas "major" changes may not be made without FAA pre-approval. So a big part of the decision to redline or not to redline is whether the change is "major" or minor."

An example of such a minor change might be relocating a borehole through a bulkhead because the original designated location has a rack of some sort attached from some previous aftermarket product. The relocated borehole may only be inches away from the original design.

In 2008, we took up the topic again

I have encountered more resistance to the quality system, currently the procedures call for all drawings to be released as a formal drawing with title block, rev status, and approvals, etc. We recently had an issue where the engineer did not take into account a change to the drawing and put in an incorrect depth for drilling and taping holes which broke through material. All issues taken care, we came up with a fix and now it's time to come up with a way to make sure the fix worked. Any way a test piece was needed, and was actually done up on solidworks, but the engineer decided to say to heck with procedure and tried to get the piece made without it going through the release procedure because this test piece would basically be a throw away piece. He argued the point with the Pres. and here I am trying to see if anyone has ever been posed with this type of thing. It's really just a a way to say oh we need this now let's throw quality to the wind and do it. Personally I think it's crap. But I was told to see if there was a good way to do this.

Any thoughts anyone? Sorry so Long winded

If all that was needed was a single piece for test purposes, why do you feel that a formal, fully-vetted print was needed? I presume that if the test piece proved out, the necessary changes would have been formally incorporated, no? It sounds to me like maybe your written procedure calls for something that might not be necessary, unless I'm missing something.

Actually, you describe a relatively common situation. In "document terms" here is what is really taking place:

1. Despite FMEA (Failure Mode & Effects Analysis), the drawing was allowed to proceed to the next stage, PPAP (Production Part Approval Process)
2. The design flaw was discovered/detected in the PPAP stage, at which point, production was suspended while a root cause analysis and a tentative Corrective Action were proposed.
3. The designer made a "temporary" change to the initial design for a prototype "research" piece to be manufactured and evaluated for suitability to function according to the design.
4. If the prototype research piece passes function evaluation, designer will "redline" the drawing as a new revision, getting approvals of the redline changes from the entire list of approvers normally used for ANY revision.
   (redline is ONLY used in lieu of formal revision when time to prepare complete new drawings and go through formal approval process would create economic hardship for supplier or customer or both.)
5. Production may continue with redline drawing while designer works up complete new revision drawing which goes through formal revision/approval process.
6. When new revision is ready, redline drawing is superseded with new revision, but there is no change in production which continues.

With modern computer drafting and combined manufacturing direct from computer assisted drawing, the concept of redline (once using actual red ink on the engineering drawings) may not be necessary because the computer can generate complete drawings incorporating design changes faster than a human can add red ink changes to a paper drawing. The same technology allows geographically scattered approvers to view changes on a computer screen via internet and add electronic approvals so formal revision could actually happen faster than formerly using redline approvals on paper drawings.

The important factor is a prototype research piece may be manufactured WITHOUT a formal drawing or approval process. I am reminded of the days when design models of cars used to be made in clay with various folks tinkering with the design by adding or removing clay and then the final drawings "reverse-engineered" from the final clay model to build the sheet metal tooling for the production design. In concept, tinkering with the depth of tapping holes in a machined piece is the same process.

Later, in 2008, we had this

Hello,

At my company, we just formed a team to identify a best document managment system for us.

One of our approaches is to determine what organizations out there do an excellent job at EDCM. We'd like to learn from them to understand why its so good. Hopefully, what we learn is transferrable to us. We've started on this approach rather that trying to find an ideal product because there are so many and I don't know that we could be sure of what we're buying before we bought it unless we can also evaluate it implemented somewhere.

Here is our problem statement:
The current organization and storage of engineering documentation makes retrieval of information difficult, allows duplicate documentation of the same revision, and uncontrolled documentation. This results in lost productivity associated with retrieval and identification of appropriate configuration, additional costs associated with rework/scrap of material fabricated to an incorrect configuration, and premature failure of incorrect components. It also limits our ability to competitively bid fabrication of components due to the risk associated with an incomplete or incorrect specification.

Although we are a manufacturer, we do not vary our products. The type of documentation we need to manage better are: engineering development files, project documents, published engineered drawings (both CAD & scanned) as well as paper media that is currently not scanned, process changes/history, maintenance or operating manuals. With this, we need to develop and incorporate change control protocols and figure out how to address legacy documentation from poor practice.

Our team plans to read a book by Frank B. Watts on the subject

With that in mind, can you recommend any industrial organizations that we could learn from?

Along this same line, if you can recommend a good consultant that would help us figure out what we need, please do.

Thanks so much for any response,
Dean

Hello Dean,

Welcome to The Cove Forums!

Great post! I have added a link to the book that you've mentioned.
That way, our Fellow Covers can take a peek at it and may look inside.

I hope you'll get responses soon.

Stijloor.

I'm an expert on electronic document management. I've looked over the table of contents in the book (if Stijloor linked the correct one - if it is a different book, let us know so we can enter the correct link) and there is nothing in the table of contents or the index which addresses electronic document management. The table of contents mirrors a lot (to my memory - I won't bother to cite line-by-line examples) of the stuff contained in Military Standard Drawing Practices (superseded by ASME Y14.100 - Engineering Drawing Practices and Associated Documents - ASME Y14.35M - Revision of Engineering Drawings and Associated Documents - PLUS ASME Y14.34M - Associated Lists)

If you hire an consultant to help your organization decide what it needs and whose software to buy, you should start with a four-year-old summary of the features you might want to consider which I posted here. If the consultant is not conversant with all the features in the list, he is probably NOT the one you should hire!

For years, I've been addressing this topic here in the Cove and over in the ASQ Forums. We have a recent thread What is the best Electronic Document Control Management System? where this topic has also been raised.

An important fact to know is that if you buy the correct software, you don't need to convert every document to pdf (very time-consuming) because modern software can allow "authorized" users to view any document regardless of its original format (CAD, photo, doc, pdf, etc) and, further, allow the authorized user to make "redline changes" in overlays, without affecting the original document. The software can automatically prevent unauthorized viewers from seeing or accessing obsolete documents.

 

[Stijloor]

I'm an expert on electronic document management. I've looked over the table of contents in the book (if Stijloor linked the correct one - if it is a different book, let us know so we can enter the correct link) and there is nothing in the table of contents or the index which addresses electronic document management.

I checked the link on Amazon, and it's still available. And yes, I hope it's the correct book.

Stijloor.

 

[dwend]

Thx Wes for your thorough response....I have certainly enjoyed reading many of your posts over the past several weeks as I come up to speed on some subjects.

Let me first say that I totally agree with this statement:

"One of the things most folks know about me is that I try to take advantage of the expertise of others who have gone before me. Therefore, I rarely try to reinvent the wheel.


And this is one of the reasons why I posted. Always looking to Benchmark when I can....

I have read Watts book and am familiar with the standards for the most part but your reply certainly includes many valid considerations. But let me say that it is not the ECO process itself that I am trying to address so much as the specific RISK Management portions of the ECO process and how much evidence needs to be presented before a certain type of change is allowed. Its one thing to have wonderful flow diagrams, clean specifications, etc but it is garbage in, garbage out. If I am considering a change that might encroach on fit/form/function based on the "quality" of the supplier or maybe a change in the supplier, or maybe the supplier changed one of their materials etc, it is not sufficient evidence to send an ECO through and say, "....out of date" without some evidence that we are not going backwards or that the supplier did his homework etc. Again its not the actual ECO process but more about the thresholds required to meet certain criteria, particularly within the gray areas between Class 1 or 2 or what Watts described as a Class 1.5 change or something similar.

As I have perused this subject I see some who have rightfully pointed out extremes such as a company that had a "draft" change process that wound up taking as long as the change process itself so changes took 2x. Then there is the old rubber stamp approach which is that the initiator obviously knows what they are doing so lets approve it so we can make our shipments this month. I can go on but I want to reiterate that the main question here is how much risk assessment is done in processing changes through the ECO process.

I plan on paretoing all of our ECO's for the past 12 months, tabulate redo's and see if I can connect to internal failure costs and then determine what type of evidence would have been required to prevent the failure. I realize things are a little easier in the rear view mirror but I will collect my own data. When I speak to an FMEA approach I was curious as to what type of statements others put in their ECO documentation to prompt/require evidence, risk assessment etc as part of the ECO process. OK, thx to all for taking the time to read this far....


DW :truce:

 

[Hodgy Hotsauce]

dwend,

The discussion below may already be very obvious to you and already taken into consideration. The bottom line is that for any effort at risk mitigation to be successful it must be well disciplined with full traceability and must employ objective and measureable evidence of effectivity.

This forum post has brought forth some excellent suggestions and ideas as to how to possibly structure your configuration management system. However, I would like to address the issue of risk management. I am a recently retired aerospace QA engineer and have experience with many various risk mitigation methods (fads that came and went over the years). The most effective methodology involved increasing the change qualification requirements as the change complexity and impact potential increased. This method was developed within the Navy FBM programs within the framework of the contract. It is a melding of FMEA, Taguchi and various attributes of the ANSI, MIL, ASTM and ASME specs that were mentioned by Wes. At the bottom of the scale would be drawing changes that correct errors that did not affect form, fit or function. Essentially, changes that update the drawing to the as-built condition. This level of change would require no qualification effort. In this example both the change complexity and impact values would be at "zero". At the other end of the scale would be, say, a change in adhesive (or adhesive formulation) that would have a major impact upon strength or structural integrity that would have a high degree of impact in case of bond line failure. This example would have a high degree of change qualification effort (lotsa testing, proofing effort, first articles, etc.). This type of change would get a value of 5 assigned to both complexity and impact potential. Criticality is another good term to assign. Given these numbers a matrix can be built to assess the risk.

I wish you the best in your efforts!!

This methodology must also take into account the materials, people, process, procedures (i.e.; 'fishbone' elements)

 

[dwend]

Thx Hodgy for your response...

What you describe sounds like the most thorough approach to the matter. I definitely like the matrix approach with a "severity" or risk rating similar to FMEA but am mindful of KISS principles as well. Our config control is in good shape overall but I want to use the 80/20 rule in reducing some of the more obvious problems/risk by beefing up the language in the actual ECO specification and maybe tweaking the form to include a prompt for "objective evidence/justification" in addition to a "description." Quality is looking for evidence that the change not only is required but that we went through an orderly process of verification and that we are not stepping into something. I really am targeting that "gray area" that exists between a routine tweak of a tolerance dimension, label or similar change to a full blown redesign/new design. Vendor part and material changes are definitely a concern for reasons previouly described and obvious to most. Somewhere there is a balance for each organization and I will find it for ours as I go forward.


DW