DPPM - Calculating the Defect Rate of a Software Test

[dmpizzle]

I'm trying to calculate the defect rate of a software test, and provide a current "score" for each software build I receive. We've been using DPPM as our score of how often a given event has occurred since the moment we saw the bug. It's an easily calculation for an individual defect, but what I really want is an overall score for the test for all the defects I have seen that have not yet been fixed.

Let's say I have the following scenario:

I've discovered Defect A at cycle 0, and I've run 10,000 cycles since then and that was the only time I've seen it. DPPM = (1/10,000)*1,000,000 = 100. At cycle 5,000, I moved from Build1 to Build2 and continued running the test and did not see Defect A. However, I have no reason to believe it has been fixed yet.

Now let's say during that same run of 10,000 cycles, I also discovered defect B at cycle 6000 (after I had moved from Build1 to Build2). DPPM of Defect B = (1/5,000)*1,000,000 = 200.

What I would *like* to do is calculate an overall DPPM based on where I stand today. I know that I've seen Defect A once over the last 10,000 cycles, and Defect B once over the last 5,000 cycles, spanning 2 SW builds. I want to calculate a current DPPM that would represent the current stability of the system. Can I reasonable calculate an overall DPPM based on these 2 metrics? Or am I looking at all wrong?

 

[Stijloor]

A Quick Bump!

Can someone help?

Thank you very much!!

 

[Bev D]

what is the difference bewteen build 1 and build 2?

It seems that you are being somewhat arbitrary in determining the area of opportunity for defect A and Defect B.

Defect A occurs once in the first 5000 cycles (build 1) and doesn't occur in the next 5000 cycles (build 2) so you assume that Defect A 'hasn't been fixed' so you divide 1 by 10,000.

defect B does exactly the opposite: it doesn't occur in teh first 5000 but does occur in the second 5000, so you assume it wasn't possible in the first 5000 (because it didn't occur? or because it couldn't occur? ) and you only divide it by 5000.

I think it's important to resolve this conflict before we move on.

 

[Miner]

You might consider using Duane or Crowe-AAMSA plots to track your progress. The method is relatively simple and the only data required is the accumulated number of faults and the accumulated test time (or cycles) associated with each fault.

The approach can handle mixed failure modes and will show improvement (or deterioration) over time by changes in slope.

 

[dmpizzle]

You are correct. In my example, I am making some assumptions that DefectA still exists in Build2, and DefectB did not exist in Build1. This certainly may not be true, but I don't know how to resolve it otherwise. All I have to go on is the data that I observed as we move from build to build and continue testing the same thing. How else *might* I think about this?

 

[Bev D]

Don't make unfounded assumptions.
Stay pure on the data.
The denominator is 10,0000 for each defect. So the total rate is 2/10,000

 

[dmpizzle]

Don't make unfounded assumptions.
Stay pure on the data.
The denominator is 10,0000 for each defect. So the total rate is 2/10,000

What if the bug was found 10 times, or 100 times, in Build2, but never in Build1? At what point can I say that it was *probably* introduced in Build2, and thus, not include cycle counts from Build1?

Also, is it correct to say that an overall build score for Build2 would be the probability of defect A + the probability of defectB?

 

[Bev D]

I have attached a spreadsheet showing the math of my explanation.

Let me start with your first assumption about defect A and defect B:

IF the defect rate was 1/5000, there is a 36% chance that a defect is found in 5000 cycles. There is also a 36% chance that no defect is found in 5000 cycles. If the defect rate were 1/10000 cycles then the probability of finding a defect in 5000 cycles is 30% and the probability of finding no defect in 5000 is 60% this holds for both defect A and B. so unless you have a physical reason (you know how defect B was created and the condition for defect B was not present in build 1) you have NO reason other than wishful thinking to assume that B wasn't present in build 1...

You would need to see about 10 defects in 5000 cycles to say with any confidence that the defect wasn't present in a cycle of 5000 with no defects.

 

[Miner]

Bev, You forgot to attach the spreadsheet.

 

[Bev D]

ooops. here is the attachment