How can we Measure the Effectiveness of an FMEA

T

tymer5

#11
Jim,
It seems to me that there are three things that need to be measured when looking at the FMEA work. The three are, did the teams follow a disciplined approach (follow a standard like the AIAG standard), Does the product have proper FMEA coverage (was the approach systematic? Were FMEAs completed wherever there was something new, changed or there was a new application or location?), And how effective were the FMEAs (Measured in terms of NPI unreliability spike and MTTF. Here is how I go about it. First recognize that the measurement of RPN alone is a poor measure of FMEA effectiveness. RPN treats Severity, Occurrence and Detection as equal and they are not.

Consider the case where S=10 (Hazardous without Warning), O=10 (Occurs 10% of the time or more) and D=1 (almost certain to detect) ==> RPN = 100

Now consider the case where S=1 (No discernible effect), O=10 (Occurs 10% of the time or more) and D=10 (Can not detect or is not checked) ==> RPN = 100.

These two cases surely are not equal. One is a safety hazard and the other no one cares about. However RPN says they are of equal risk. NO WAY! :jawdrop:

Risk is defined as the possibility of something bad happening. So if you are to quantify risk in an FMEA then you need to quantify two things...how bad is bad and just how possible it is (probability). Therefore Risk is really defined as Severity x Occurrence.

Severity and Occurrence can only be changed through a design or process change. (There are other debates raging on this but this is where I stand.)

So lets look at risk on a chart. See the attachment. (sorry I can't add the attachment for some reason, maybe someone else has a copy they can post. It is a 2D matrix that maps Severity vs. Occurrence and shades in cells as red, yellow and green to show where action must be taken) This is called the Criticality Matrix, and it defines when the design or process team needs to make changes to the design or process because the risk is too high. Now that we have defined risk and when to make design or process improvements it is easier to use the following SAE survey questions to understand if the FMEA is considered good or not. Here is a list of those questions...

For Design:

1. The FMEA drives Design Improvements as primary objective.
2. The FMEA addresses all high risk Failure Modes, as identified by the FMEA team, with executable Action Plans. All other failure modes are considered. (Enter none where no action is required)
3. The Analysis/Development/Validation (A/D/V) and/or Design Verification Plan and Report (DVP&R) considers the failure modes from the Design FMEA.
4. The FMEA scope includes integration and interface failure modes in both block diagram and analysis.
5.The FMEA considers all major "lessons learned" (such as high warranty, campaigns, etc.) as input to failure mode identification.
6. The FMEA identifies appropriate Special Characteristic candidates, as input to the Special Characteristics selection process.
7. The FMEA is completed during the "window of opportunity" where it could most efficiently impact the product design.
8. The right people participate as part of the FMEA team throughout the analysis, and are adequately trained in the procedure. As appropriate, a facilitator should be utilized.
9. The FMEA document is completely filled out "by the book," including "Action Taken" and new RPN values.
10. Time spent by the FMEA team, as early as possible, is an effective and efficient use of time, with a value -added result. This assumes Recommended Actions are identified as required and the actions are implemented.

This means you will have to audit some of your FMEAs to understand if the teams are following this common disciplined approach.

For coverage you will have to understand what was new or changed in the product or process and then evaluate if those new or changed conditions were systematically covered with FMEA work. This assumes a previous FMEA or FMEAs were done on the previous product or process.

Last of all you will have to do a postmortem once the product and/or process has been released. That means you will have to gather your quality data from production launch through the product life. You will have to enter those failures into the FMEA through your CPI projects (or whatever your corrective action process is). This is what is meant by keeping the FMEAs as living documents. Metrics should show your progress over time as your teams become more disciplined and effective with FMEAs.

That's my :2cents:

I hope it helps.
 
Elsmar Forum Sponsor

Wes Bucey

Quite Involved in Discussions
#12
Jim,
It seems to me that there are three things that need to be measured when looking at the FMEA work. The three are, did the teams follow a disciplined approach (follow a standard like the AIAG standard), Does the product have proper FMEA coverage (was the approach systematic? Were FMEAs completed wherever there was something new, changed or there was a new application or location?), And how effective were the FMEAs (Measured in terms of NPI unreliability spike and MTTF. Here is how I go about it. First recognize that the measurement of RPN alone is a poor measure of FMEA effectiveness. RPN treats Severity, Occurrence and Detection as equal and they are not.

Consider the case where S=10 (Hazardous without Warning), O=10 (Occurs 10% of the time or more) and D=1 (almost certain to detect) ==> RPN = 100

Now consider the case where S=1 (No discernible effect), O=10 (Occurs 10% of the time or more) and D=10 (Can not detect or is not checked) ==> RPN = 100.

These two cases surely are not equal. One is a safety hazard and the other no one cares about. However RPN says they are of equal risk. NO WAY! :jawdrop:

Risk is defined as the possibility of something bad happening. So if you are to quantify risk in an FMEA then you need to quantify two things...how bad is bad and just how possible it is (probability). Therefore Risk is really defined as Severity x Occurrence.

Severity and Occurrence can only be changed through a design or process change. (There are other debates raging on this but this is where I stand.)

So lets look at risk on a chart. See the attachment. (sorry I can't add the attachment for some reason, maybe someone else has a copy they can post. It is a 2D matrix that maps Severity vs. Occurrence and shades in cells as red, yellow and green to show where action must be taken) This is called the Criticality Matrix, and it defines when the design or process team needs to make changes to the design or process because the risk is too high. Now that we have defined risk and when to make design or process improvements it is easier to use the following SAE survey questions to understand if the FMEA is considered good or not. Here is a list of those questions...

For Design:

1. The FMEA drives Design Improvements as primary objective.
2. The FMEA addresses all high risk Failure Modes, as identified by the FMEA team, with executable Action Plans. All other failure modes are considered. (Enter none where no action is required)
3. The Analysis/Development/Validation (A/D/V) and/or Design Verification Plan and Report (DVP&R) considers the failure modes from the Design FMEA.
4. The FMEA scope includes integration and interface failure modes in both block diagram and analysis.
5.The FMEA considers all major "lessons learned" (such as high warranty, campaigns, etc.) as input to failure mode identification.
6. The FMEA identifies appropriate Special Characteristic candidates, as input to the Special Characteristics selection process.
7. The FMEA is completed during the "window of opportunity" where it could most efficiently impact the product design.
8. The right people participate as part of the FMEA team throughout the analysis, and are adequately trained in the procedure. As appropriate, a facilitator should be utilized.
9. The FMEA document is completely filled out "by the book," including "Action Taken" and new RPN values.
10. Time spent by the FMEA team, as early as possible, is an effective and efficient use of time, with a value -added result. This assumes Recommended Actions are identified as required and the actions are implemented.

This means you will have to audit some of your FMEAs to understand if the teams are following this common disciplined approach.

For coverage you will have to understand what was new or changed in the product or process and then evaluate if those new or changed conditions were systematically covered with FMEA work. This assumes a previous FMEA or FMEAs were done on the previous product or process.

Last of all you will have to do a postmortem once the product and/or process has been released. That means you will have to gather your quality data from production launch through the product life. You will have to enter those failures into the FMEA through your CPI projects (or whatever your corrective action process is). This is what is meant by keeping the FMEAs as living documents. Metrics should show your progress over time as your teams become more disciplined and effective with FMEAs.

That's my :2cents:

I hope it helps.
I think this is a very well reasoned response!
 
J

Jim123

#13
Where can I read this paper?
"Integration of FMEA and the Kano model..."

Jim
 
Last edited by a moderator:

Mikael

Quite Involved in Discussions
#16
As stated I yet don´t know enough about FMEA to evaluate the paper, it might be good, it might not. But he do cover the problem that RPN indexes are all equally treated, as one of you mentioned above. For that purpose he use the Kano-model, which in relation to FMEA cover Must-be quality elemnets. (BTW: Is there a opposite to FMEA, handling the positive effects?).

The Kano-model I know much more about, and Shahin (2004) use Tan & Shen's (2000) general function "S =cp(k)," and then tranforms it into a new RPN formula. Practically it migth work fine, but in academic terms, it is problematic to argue for a general formula concerning the Kano's quality elemnets.
 

Caster

An Early Cover
Trusted Information Resource
#17
1. The FMEA drives Design Improvements as primary objective.
on the previous product or process.

<edit>

Last of all you will have to do a postmortem once the product and/or process has been released. That means you will have to gather your quality data from production launch through the product life. You will have to enter those failures into the FMEA through your CPI projects (or whatever your corrective action process is). This is what is meant by keeping the FMEAs as living documents. Metrics should show your progress over time as your teams become more disciplined and effective with FMEAs.That's my I hope it helps.


Hi Tymer5

Nicely done.

Is this a theoretical construct or is this actually happening where you consult or work?

Are you describing "as it should be" or "as it is"

Please give me hope, I'd like to hear someone is actually doing FMEA for all the right reasons, not just to fill up a tab in a PPAP binder.
 
J

Jim123

#19
Hi Helmut,

how can I measure good value?
Is a good FMEA only a good document with detailed description?
Or is a good FMEA a good high scientifical discussion, but sometimes
lean documentation? Or is a good FMEA when the design need less samples or tests or sometimes tests that not planed up to now? Or is a good FMEA when I have no trouble to SOP?

How can I measure a good job with FMEA?
:cool:
Jim
 

Helmut Jilling

Auditor / Consultant
#20
Hi Helmut,

how can I measure good value?
Is a good FMEA only a good document with detailed description?
Or is a good FMEA a good high scientifical discussion, but sometimes
lean documentation? Or is a good FMEA when the design need less samples or tests or sometimes tests that not planed up to now? Or is a good FMEA when I have no trouble to SOP?

How can I measure a good job with FMEA?
:cool:
Jim

I don't know. I think you are guessing.

But, if you really want to measure a FMEA, you have to define what good means first. What is a good FMEA? Then you decide how to measure that. If a good FMEA means low numbers, then that is what you measure. If a good FMEA is a launch that went very well, then measure that instead. You can't measure whether something is "good" until you define "good." You are asking us to tell you what we think is a good FMEA, and I think that is something you or your customer must define.

I am not sure I would want to "measure" the FMEA, at all. I would want to evaluate the effectiveness of the whole program vehicle launch. The FMEA is part of that.
 
Thread starter Similar threads Forum Replies Date
qualprod ISO 9001 5.1.1 - How to measure System effectiveness ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 4
qualprod Best criteria to measure Corrective Action effectiveness - Poor Maintenance ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 10
M How to measure effectiveness and efficiency of the established QMS as Auditors ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 14
J How to measure effectiveness of the Metrology Process General Measurement Device and Calibration Topics 5
Chennaiite How to Measure Effectiveness of the Design Validation Process? Design and Development of Products and Processes 7
V How to Measure Effectiveness of Risk Assessment Processes FMEA and Control Plans 7
M How To Measure Training Effectiveness Quality Tools, Improvement and Analysis 8
Q How to measure Effectiveness in Tube Repairments? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
C How do you measure the Effectiveness of a QMS (Quality Management System) Quality Manager and Management Related Issues 34
D How to Measure QSR Training Effectiveness? Training - Internal, External, Online and Distance Learning 6
E How to Measure Internal Audit Effectiveness General Auditing Discussions 12
J Effectiveness of training? How to Measure Training Effectiveness Training - Internal, External, Online and Distance Learning 51
T How to measure the Effectiveness of a Quality Management System? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 7
T Need to design a test to measure inspection effectiveness Design and Development of Products and Processes 9
L Product Preservation - Measure of effectiveness IATF 16949 - Automotive Quality Systems Standard 12
Raffy How do I measure Measuring Training Effectiveness? Performance evaluation? Other ISO and International Standards and European Regulations 26
S Measuring Training Effectiveness - How do you measure Effectiveness? Training - Internal, External, Online and Distance Learning 50
C First Article Inspections - 'how to measure' General Measurement Device and Calibration Topics 11
lanley liao Question regarding the calibration of monitoring and measure equipment. Oil and Gas Industry Standards and Regulations 0
J Should a Class 1 medical device with an option to measure body weight be considered Class 1m? EU Medical Device Regulations 0
M Gage R&R and right way to measure Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 16
P How to measure DPPM in Chemical batch manufacturing Quality Tools, Improvement and Analysis 1
silentmonkey How to measure severity if my product is designed for emergency use and failure would result in death? ISO 14971 - Medical Device Risk Management 9
A Looking for a third party to calibrate our measure equipment - South Carolina General Measurement Device and Calibration Topics 6
J Validity / outcomes measure for custom made medical device ISO 13485:2016 - Medical Device Quality Management Systems 2
D Performance specification as a Risk Control Measure, EN 14971 ISO 14971 - Medical Device Risk Management 7
J KPIs or Metrics to Measure a New Complaint Handling Process 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 9
D Torque fractional unit of measure; cN-m or N-cm? General Measurement Device and Calibration Topics 2
J How to measure a Critical Characteristic that is a Basic Dimension Misc. Quality Assurance and Business Systems Related Topics 6
I Can we use pin gauges to measure an accurate Cpk ? Capability, Accuracy and Stability - Processes, Machines, etc. 9
1 How to measure straightness on a HEX Long Bar Manufacturing and Related Processes 7
Ron Rompen Measurement Method Challenge - Measure feature #91 Inspection, Prints (Drawings), Testing, Sampling and Related Topics 9
G Measure 3 times, for calibration report? Customer Owned Reference Standard General Measurement Device and Calibration Topics 4
M Informational The ANSM decides, as a precautionary measure, to withdraw from the market macrotextured breast implants and breast implants with polyurethane-coated s Medical Device and FDA Regulations and Standards News 0
M Informational Health Canada advises Allergan of its intent to suspend its licences for Biocell breast implants as a precautionary measure Medical Device and FDA Regulations and Standards News 0
U SI 13 - Not Every Process Requires an Efficiency Measure IATF 16949 - Automotive Quality Systems Standard 1
P NPR Metric - What if Customer has relaxed measure? TL 9000 Telecommunications Standard and QuEST 1
T Tape Measure, the right device for 2 or 3 decimals accuracy on prints General Measurement Device and Calibration Topics 8
M How do we document and measure a service we do not provide? IATF 16949 - Automotive Quality Systems Standard 1
M For a Type 1 Gage R&R I want to use 1 unit and measure it 30 times Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 5
M How to Measure Paper considering Variation and Humidity General Measurement Device and Calibration Topics 10
rob73 Advice/recommendations for equipment to measure a soft pvc tube - Optical Measuring? General Measurement Device and Calibration Topics 4
R Height gauge to measure short hard to access heights General Measurement Device and Calibration Topics 3
D Proper Tool to Measure Wall Thickness (ISO 3611 - Micrometers) Manufacturing and Related Processes 7
S Quality Measure for Engineering Services Misc. Quality Assurance and Business Systems Related Topics 1
A What are methods and tools to measure Workload analysis ? Human Factors and Ergonomics in Engineering 1
M How do I measure variation in my process ? Capability, Accuracy and Stability - Processes, Machines, etc. 3
J How to measure process quality? Is DPPM the correct metric? Capability, Accuracy and Stability - Processes, Machines, etc. 5
M Micrometer Anvil Placement to measure the diameter of Steel Wires Measurement Uncertainty (MU) 6
S Where to Get SPC Training to Measure our Processes? Training - Internal, External, Online and Distance Learning 12

Similar threads

Top Bottom