How to Measure Effectiveness of the Design Validation Process?

Chennaiite

Never-say-die
Trusted Information Resource
#1
My query is with regard to Component and Vehicle testing carried out during Design & Development.

Thanks in advance.
 
Elsmar Forum Sponsor

somashekar

Staff member
Super Moderator
#2
By effective pre and post warranty repair / replacement data analysis on the vehicle population.
In a way your components / vehicles are being continuously validated on and off the streets.
OR
By extreme condition testing like MRF
"The tyres we race are the tyres you buy" ....so goes the MRF slogan
 

Chennaiite

Never-say-die
Trusted Information Resource
#3
I don't deny that the product performance in the field is the measure of effectiveness of Design validation 'in a way'. But since any product failing in the market is preceded by a lot of process steps, and Validation during Design and Development is only one of those, the analysis of field data does not categorically conclude the effectiveness of one particular step, I believe.

I have personally retro analyzed so many warranty failures that are attributed to Design & Development phase and in some cases I can say that the test relevant to the failure was conducted and cleared without any failure. But only with this, I am unable to scientifically judge my testing process. I further get into and compare the test conditions against field conditions, test standard used, equipment used, etc but that is getting me nowhere and moreover it becomes case to case analysis of systemic root cause. Instead of that I am looking for a metrics that measures the ability of a particular test to detect the failure mode and hence gives me an indication of effectiveness of the testing process in short term i.e. without having to wait for warranty failures. I am intereted in knowing if any such practice exists.

I am not sure if I really managed to translate my thoughts into words here.
 

Ronen E

Problem Solver
Staff member
Moderator
#4
Hi,

Generally speaking, "effective" means "achieving its defined objective(s)".

What are design validation's objectives? IMO, to establish through sound objective evidence that users' needs, as defined for the given device, are met.

Have you defined and documented what the users' needs are, for the given device? If your design validation provided good (or at least reasonable) confidence that these needs were met, then it was effective by the definition I began with. Risk management is an interacting and completing activity for enhancing your confidence that there will be as little as practically possible consequent warranty failures.

If you're looking for 100% confidence, then I don't think any single QA tool can provide that. I'm not even sure any combination of tools available will guarantee a 100% failure free device (at least not in real-life world). That's why post-marketing monitoring and continuous improvement tools are so important.

Cheers,
Ronen.
 

Chennaiite

Never-say-die
Trusted Information Resource
#5
<snip>If your design validation provided good (or at least reasonable) confidence that these needs were met <snip>
This is something that I want to zoom in. My results as such can give me confidence of the product meeting user needs, but what is the probability of the results not showing false alarm or miss rate. This is something in the lines of calibration 'uncertainity'. I am not good at this subject of calibration, but I understand 'uncertainity' is the indicator of how uncertain the results can be. Similarly, how uncertain can any testing result be, given the variables involved in the process, and how to measure it.

Let me take an example, a suspension system is being tested for durability under certain load conditions. The test results conclude that the system, under the given conditions, will crack at 50000 cycles. Now, the results could be influenced by variation in 4M, which brings the question of uncertainity to the fore. Considering a miss rate, the product is prone to failures in the field and if it is a false alarm, the product ends getting overdesigned.
 
Last edited:

somashekar

Staff member
Super Moderator
#6
You kept me thinking if everything that we do, can get into statistics and the probability, confidence level, uncertainty and such stuff can be found and improved.....
 

Ronen E

Problem Solver
Staff member
Moderator
#7
Similarly, how uncertain can any testing result be, given the variables involved in the process, and how to measure it.
The way to address it is through sound statistical techniques and proper validation planning. When I talked about "good confidence" I didn't refer to a subjective feeling.

Cheers,
Ronen.
 
M

markspend01

#8
Hey Guys well i think that Metrics are developed to evaluate the effectiveness of a procedure. Analytic's may be situated at any given step of a procedure and should make sure that client specifications are being met.Thanks!!
 
Thread starter Similar threads Forum Replies Date
T Need to design a test to measure inspection effectiveness Design and Development of Products and Processes 9
qualprod ISO 9001 5.1.1 - How to measure System effectiveness ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 4
qualprod Best criteria to measure Corrective Action effectiveness - Poor Maintenance ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 10
M How to measure effectiveness and efficiency of the established QMS as Auditors ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 14
J How to measure effectiveness of the Metrology Process General Measurement Device and Calibration Topics 5
V How to Measure Effectiveness of Risk Assessment Processes FMEA and Control Plans 7
M How To Measure Training Effectiveness Quality Tools, Improvement and Analysis 8
Q How to measure Effectiveness in Tube Repairments? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
C How do you measure the Effectiveness of a QMS (Quality Management System) Quality Manager and Management Related Issues 34
D How to Measure QSR Training Effectiveness? Training - Internal, External, Online and Distance Learning 6
E How to Measure Internal Audit Effectiveness General Auditing Discussions 12
J Effectiveness of training? How to Measure Training Effectiveness Training - Internal, External, Online and Distance Learning 51
J How can we Measure the Effectiveness of an FMEA FMEA and Control Plans 24
T How to measure the Effectiveness of a Quality Management System? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 7
L Product Preservation - Measure of effectiveness IATF 16949 - Automotive Quality Systems Standard 12
Raffy How do I measure Measuring Training Effectiveness? Performance evaluation? Other ISO and International Standards and European Regulations 26
S Measuring Training Effectiveness - How do you measure Effectiveness? Training - Internal, External, Online and Distance Learning 50
C First Article Inspections - 'how to measure' General Measurement Device and Calibration Topics 11
lanley liao Question regarding the calibration of monitoring and measure equipment. Oil and Gas Industry Standards and Regulations 0
J Should a Class 1 medical device with an option to measure body weight be considered Class 1m? EU Medical Device Regulations 0
M Gage R&R and right way to measure Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 16
P How to measure DPPM in Chemical batch manufacturing Quality Tools, Improvement and Analysis 1
silentmonkey How to measure severity if my product is designed for emergency use and failure would result in death? ISO 14971 - Medical Device Risk Management 9
A Looking for a third party to calibrate our measure equipment - South Carolina General Measurement Device and Calibration Topics 6
J Validity / outcomes measure for custom made medical device ISO 13485:2016 - Medical Device Quality Management Systems 2
D Performance specification as a Risk Control Measure, EN 14971 ISO 14971 - Medical Device Risk Management 7
J KPIs or Metrics to Measure a New Complaint Handling Process 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 9
D Torque fractional unit of measure; cN-m or N-cm? General Measurement Device and Calibration Topics 2
J How to measure a Critical Characteristic that is a Basic Dimension Misc. Quality Assurance and Business Systems Related Topics 6
I Can we use pin gauges to measure an accurate Cpk ? Capability, Accuracy and Stability - Processes, Machines, etc. 9
1 How to measure straightness on a HEX Long Bar Manufacturing and Related Processes 7
Ron Rompen Measurement Method Challenge - Measure feature #91 Inspection, Prints (Drawings), Testing, Sampling and Related Topics 9
G Measure 3 times, for calibration report? Customer Owned Reference Standard General Measurement Device and Calibration Topics 4
M Informational The ANSM decides, as a precautionary measure, to withdraw from the market macrotextured breast implants and breast implants with polyurethane-coated s Medical Device and FDA Regulations and Standards News 0
M Informational Health Canada advises Allergan of its intent to suspend its licences for Biocell breast implants as a precautionary measure Medical Device and FDA Regulations and Standards News 0
U SI 13 - Not Every Process Requires an Efficiency Measure IATF 16949 - Automotive Quality Systems Standard 1
P NPR Metric - What if Customer has relaxed measure? TL 9000 Telecommunications Standard and QuEST 1
T Tape Measure, the right device for 2 or 3 decimals accuracy on prints General Measurement Device and Calibration Topics 8
M How do we document and measure a service we do not provide? IATF 16949 - Automotive Quality Systems Standard 1
M For a Type 1 Gage R&R I want to use 1 unit and measure it 30 times Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 5
M How to Measure Paper considering Variation and Humidity General Measurement Device and Calibration Topics 10
rob73 Advice/recommendations for equipment to measure a soft pvc tube - Optical Measuring? General Measurement Device and Calibration Topics 4
R Height gauge to measure short hard to access heights General Measurement Device and Calibration Topics 3
D Proper Tool to Measure Wall Thickness (ISO 3611 - Micrometers) Manufacturing and Related Processes 7
S Quality Measure for Engineering Services Misc. Quality Assurance and Business Systems Related Topics 1
A What are methods and tools to measure Workload analysis ? Human Factors and Ergonomics in Engineering 1
M How do I measure variation in my process ? Capability, Accuracy and Stability - Processes, Machines, etc. 3
J How to measure process quality? Is DPPM the correct metric? Capability, Accuracy and Stability - Processes, Machines, etc. 5
M Micrometer Anvil Placement to measure the diameter of Steel Wires Measurement Uncertainty (MU) 6
S Where to Get SPC Training to Measure our Processes? Training - Internal, External, Online and Distance Learning 12

Similar threads

Top Bottom