SBS - The Best Value in QMS software

DPPM - Calculating the Defect Rate of a Software Test

D

dmpizzle

#1
I'm trying to calculate the defect rate of a software test, and provide a current "score" for each software build I receive. We've been using DPPM as our score of how often a given event has occurred since the moment we saw the bug. It's an easily calculation for an individual defect, but what I really want is an overall score for the test for all the defects I have seen that have not yet been fixed.

Let's say I have the following scenario:

I've discovered Defect A at cycle 0, and I've run 10,000 cycles since then and that was the only time I've seen it. DPPM = (1/10,000)*1,000,000 = 100. At cycle 5,000, I moved from Build1 to Build2 and continued running the test and did not see Defect A. However, I have no reason to believe it has been fixed yet.

Now let's say during that same run of 10,000 cycles, I also discovered defect B at cycle 6000 (after I had moved from Build1 to Build2). DPPM of Defect B = (1/5,000)*1,000,000 = 200.

What I would *like* to do is calculate an overall DPPM based on where I stand today. I know that I've seen Defect A once over the last 10,000 cycles, and Defect B once over the last 5,000 cycles, spanning 2 SW builds. I want to calculate a current DPPM that would represent the current stability of the system. Can I reasonable calculate an overall DPPM based on these 2 metrics? Or am I looking at all wrong?
 
Elsmar Forum Sponsor

Bev D

Heretical Statistician
Staff member
Super Moderator
#3
what is the difference bewteen build 1 and build 2?

It seems that you are being somewhat arbitrary in determining the area of opportunity for defect A and Defect B.

Defect A occurs once in the first 5000 cycles (build 1) and doesn't occur in the next 5000 cycles (build 2) so you assume that Defect A 'hasn't been fixed' so you divide 1 by 10,000.

defect B does exactly the opposite: it doesn't occur in teh first 5000 but does occur in the second 5000, so you assume it wasn't possible in the first 5000 (because it didn't occur? or because it couldn't occur? ) and you only divide it by 5000.

I think it's important to resolve this conflict before we move on.
 

Miner

Forum Moderator
Staff member
Admin
#4
You might consider using Duane or Crowe-AAMSA plots to track your progress. The method is relatively simple and the only data required is the accumulated number of faults and the accumulated test time (or cycles) associated with each fault.

The approach can handle mixed failure modes and will show improvement (or deterioration) over time by changes in slope.
 
D

dmpizzle

#5
You are correct. In my example, I am making some assumptions that DefectA still exists in Build2, and DefectB did not exist in Build1. This certainly may not be true, but I don't know how to resolve it otherwise. All I have to go on is the data that I observed as we move from build to build and continue testing the same thing. How else *might* I think about this?
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#6
Don't make unfounded assumptions.
Stay pure on the data.
The denominator is 10,0000 for each defect. So the total rate is 2/10,000
 
D

dmpizzle

#7
Don't make unfounded assumptions.
Stay pure on the data.
The denominator is 10,0000 for each defect. So the total rate is 2/10,000
What if the bug was found 10 times, or 100 times, in Build2, but never in Build1? At what point can I say that it was *probably* introduced in Build2, and thus, not include cycle counts from Build1?

Also, is it correct to say that an overall build score for Build2 would be the probability of defect A + the probability of defectB?
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#8
I have attached a spreadsheet showing the math of my explanation.

Let me start with your first assumption about defect A and defect B:

IF the defect rate was 1/5000, there is a 36% chance that a defect is found in 5000 cycles. There is also a 36% chance that no defect is found in 5000 cycles. If the defect rate were 1/10000 cycles then the probability of finding a defect in 5000 cycles is 30% and the probability of finding no defect in 5000 is 60% this holds for both defect A and B. so unless you have a physical reason (you know how defect B was created and the condition for defect B was not present in build 1) you have NO reason other than wishful thinking to assume that B wasn't present in build 1...

You would need to see about 10 defects in 5000 cycles to say with any confidence that the defect wasn't present in a cycle of 5000 with no defects.
 
Thread starter Similar threads Forum Replies Date
P How to measure DPPM in Chemical batch manufacturing Quality Tools, Improvement and Analysis 1
M Metric/Way to show how Internal DPPM and External DPPM relate to each other Nonconformance and Corrective Action 1
R DPPM? Scrap? Both? How to report Returns Misc. Quality Assurance and Business Systems Related Topics 6
J How to measure process quality? Is DPPM the correct metric? Capability, Accuracy and Stability - Processes, Machines, etc. 5
L DPPM (Defective Parts Per Million) Reporting - Large Quantity Approach Benchmarking 8
C What is the appropriate DPPM for to be used In-Process Inspection Other Medical Device Related Standards 2
Ajit Basrur Dashboard Software for representing Quality Metrics like DPPM, On Time Delivery, etc. Quality Tools, Improvement and Analysis 4
V Calculation of Cable Assembly DPPM Customer Complaints 6
C Calculate DPPM/PPM for second yield rework Quality Tools, Improvement and Analysis 1
L Is there any place for sampling plan if customer requirement is 20 DPPM Inspection, Prints (Drawings), Testing, Sampling and Related Topics 4
J Failures-in-time (FITs), DPPM and Probability Calculation Questions Reliability Analysis - Predictions, Testing and Standards 9
B DPPM - 'Standard' minimum quantity to measure DPPM? Statistical Analysis Tools, Techniques and SPC 21
A DPPM vs. PPM - What is its purpose and use? Quality Tools, Improvement and Analysis 1
Ajit Basrur Trending segregated for cosmetic and functional dppm? Automotive requirements Misc. Quality Assurance and Business Systems Related Topics 5
S Packaging Specification template and Reasonable dppm target for printed material? Document Control Systems, Procedures, Forms and Templates 1
Y Definition DPPM - Quality Indicator? Using DPPM instead of DPMO as our OQA quality indicator Definitions, Acronyms, Abbreviations and Interpretations Listed Alphabetically 4
K DPPM vs. AQL - What is the difference between DPPM and AQL? AQL - Acceptable Quality Level 17
R MTBF to DPPM Calculation Based on the Unit's Typical Working Hours Reliability Analysis - Predictions, Testing and Standards 9
G DPPM Industry Standard - Metal fabrication (Job shop) Misc. Quality Assurance and Business Systems Related Topics 3
G Definition DPPM - Defective Parts Per Million Calculation Formula Definitions, Acronyms, Abbreviations and Interpretations Listed Alphabetically 5
M Lean: Conducting Capacity Study, calculating Cycle times on laser cutting machines Lean in Manufacturing and Service Industries 2
A Calculating Carbon Footprint in the Organization Miscellaneous Environmental Standards and EMS Related Discussions 2
C Calculating your KPIs ISO 13485:2016 - Medical Device Quality Management Systems 7
K Calculating Heat Dissipation Manufacturing and Related Processes 1
Q Calculating Defect Rates ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 4
S Formula for Calculating NoGo Major Diameter for UN gages Inspection, Prints (Drawings), Testing, Sampling and Related Topics 4
P Calculating a weight for a machine for the CE label CE Marking (Conformité Européene) / CB Scheme 0
R Remote Support - Calculating Number of Employees IATF 16949 - Automotive Quality Systems Standard 2
G Calculating Tolerance of a Panel Meter with a 2 degrees of an Arc General Measurement Device and Calibration Topics 0
T Calculating LCL, UCL, Cp, and Cpk in an Excel Spreadsheet Statistical Analysis Tools, Techniques and SPC 3
V Product Development - When to start calculating Process Capability Capability, Accuracy and Stability - Processes, Machines, etc. 3
J Calculating Reliability for Subsystems in Series Reliability Analysis - Predictions, Testing and Standards 15
Stijloor Calculating (3rd Party) Audit Days for Company with Seasonal Employees General Auditing Discussions 3
T Formulas for Calculating Coefficients for RTD / PRTD Calibration According to ITS-90 General Measurement Device and Calibration Topics 5
J Calculating part variation from historic data for GRR study Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 1
P Purpose of calculating Ta, Tb & R Square in Linearity Study Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 1
P Purpose of calculating Uncertainty value in calibration study Measurement Uncertainty (MU) 5
M Calculating MTBF for plug-in PCI cards Reliability Analysis - Predictions, Testing and Standards 2
J Calculating impact force, expressed in lbs, of a 5 lb weight dropped 48 inches Inspection, Prints (Drawings), Testing, Sampling and Related Topics 1
M Calculating Cpk when sample size equals to 1 Capability, Accuracy and Stability - Processes, Machines, etc. 12
D Calculating Cpk on Non-Normal Data Distribution Capability, Accuracy and Stability - Processes, Machines, etc. 10
K Calculating Capability of a process when data is skewed in the upper range Capability, Accuracy and Stability - Processes, Machines, etc. 4
A Calculating Accuracy for a Multimeter General Measurement Device and Calibration Topics 1
M Calculating Coefficients for an RTD (Resistance Temperature Detector) Probe General Measurement Device and Calibration Topics 3
T Calculating Plating/Coating Weight for IMDS RoHS, REACH, ELV, IMDS and Restricted Substances 3
D Calculating Lower and Upper 3 Sigma Control Limits Statistical Analysis Tools, Techniques and SPC 2
M Calculating Adequate Receiving Inspection Sample Size Statistical Analysis Tools, Techniques and SPC 2
M Calculating Capability of Delivery Performance Capability, Accuracy and Stability - Processes, Machines, etc. 5
K Advice on Calculating Control Chart Control Limits Statistical Analysis Tools, Techniques and SPC 13
B Calculating Combined DPMO and Sigma Level for Two or More Different Work Areas Six Sigma 3

Similar threads

Top Bottom