Design Verification Sample Size vs Repeats

DanMann

Quite Involved in Discussions
I've been trying to determine sample sizes vs repeats for a product, but I don't seem to be able to find an answer (and in fact, think I might not know the correct terminology).
An example is that we make a printer and we want to confirm that the printer can print in under 10 seconds. We can test multiple printers (X) and make each printer print multiple times (Y); we obviously want to test as few printers as possible. How do I determine X vs Y and what statistical test should we use?
Sorry if this is a simple question, but I have been searching for about three hours now.
 

Miner

Forum Moderator
Leader
Admin
This is similar to a measurement systems analysis. You are talking about with-printer capability (repeatability) vs. between printer capability (reproducibility). Which is more important from your stakeholders' perspective? Or is it a combination of both?

How do you plan to assess whether either are acceptable (i.e., 1-sample t-test vs. requirement, capability study)?
 

DanMann

Quite Involved in Discussions
It's a combination of both, we want to be confident both that the printer performance will be consistently within the specification and that each printer will be within the specification. If it were a single printer, we would use a 1-sample t-test, but we're not sure what to do when testing across multiple devices. Previously, we have use 1-sample t and spread the total number of prints across the printers available, but in that case, we could just print all of them on 1 printer and this definitely isn't right.
 

Miner

Forum Moderator
Leader
Admin
A few more questions to better understand your requirements. The 1-sample t-test checks to see whether the mean is different from the requirements, but does not show whether 100% of the results meet specification. This is acceptable for certain types of requirements, such as the average net weight of food products must meet or exceed the advertised weight. However, other requirements expect 100% of the results to meet requirements. Are you just trying to show the average performance of your printers meet or exceed the advertised performance?
 

Bev D

Heretical Statistician
Leader
Super Moderator
This situation is rarely covered by anything published. I have spent a lot of time putting together the few resources that exist and developing guidelines on how my organization does this. (Someday I'll write a book)

One other question: what constitutes a "print" that must complete in 10 seconds?

the sample size required will depend on:
1) the standard deviation of print times
2) the standard deviation between printers - is there a systemic difference in the average or standard deviation of print times?
 

DanMann

Quite Involved in Discussions
A few more questions to better understand your requirements. The 1-sample t-test checks to see whether the mean is different from the requirements, but does not show whether 100% of the results meet specification. This is acceptable for certain types of requirements, such as the average net weight of food products must meet or exceed the advertised weight. However, other requirements expect 100% of the results to meet requirements. Are you just trying to show the average performance of your printers meet or exceed the advertised performance?
Hi Miner, In this case, it's just the average is within the specification, but I think there will be items where the other scenario is true; do you know what to do in either scenario?
 

DanMann

Quite Involved in Discussions
This situation is rarely covered by anything published. I have spent a lot of time putting together the few resources that exist and developing guidelines on how my organization does this. (Someday I'll write a book)

One other question: what constitutes a "print" that must complete in 10 seconds?

the sample size required will depend on:
1) the standard deviation of print times
2) the standard deviation between printers - is there a systemic difference in the average or standard deviation of print times?
Hi Bev, Tell me when you release your book so I can get a copy! A print is timed from pressing go to the label being released from the printer (we plan on setting up a light gate that the label will break). We can determine a standard deviation for print times by doing some initial testing and we only expect there to be common cause variation between print speeds.
 

John Predmore

Trusted Information Resource
Dr Taguchi advocated the idea of inner array of design factors and outer array of noise factors. He also recommended, when you have identified several noise factors, it may be unnecessary to test under every combination of noise factors. If you have prior knowledge, for example, that high temperature and low humidity and thin paper are worse for your product performance response, then create one scenario with all noise factors at the worse level and run your verification test there. Dorian Shainin employed a similar strategy, where he called the worse level of environmental factors WoW (which represented the Worst of the Worst) and Bob (Best of the Best). Since your design verification is testing against a unilateral specification (must run faster than....) it may suffice to only run your verification experiment under the Worse scenario of noise factors. Mr. Shainin used a variant of Tukey's Endcount Test to test for significance, when the focus is on one end of the distribution rather than the mean. If you can show the worst case hardware combination fulfills the 10 second requirement under the worst case environmental conditions, this approach greatly reduces the number of units you have to test to verify your design.
 

Bev D

Heretical Statistician
Leader
Super Moderator
Correction: The BOB and WOW levels were for all factors not just ‘environmental’ or ‘noise’. The original use was for post hoc testing - pick the worst parts and the best parts as measured by the critical characteristic or the Y. But the concept of running under worst case conditions is a universal fact of physics: if you know which factors and conditions will produce the longest print times when set at their Max allowed limits‘ then you can see how much true margin you have to the specification(s). It is much more powerful and uses many fewer samples or print runs in your case than random sampling.
 

Miner

Forum Moderator
Leader
Admin
Hi Miner, In this case, it's just the average is within the specification, but I think there will be items where the other scenario is true; do you know what to do in either scenario?
BevD is correct. If you still need to pursue your original plan the number of different printers is more important than the number of repeat measurements for each printer. For example, if you tested 10 printers, your degrees of freedom is 9 (i.e., 10 - 1 = 9). If you perform 1 repeat measurement per printer, your degrees of freedom is 19 (i.e., 10 x 2 = 20, 20 - 1 = 19). Therefore, focus on the number of printers and perform two measurements per printer. You can analyze the results using ANOVA and calculate the within and between printer variation as well as the combined variation. The within printer variation will be in the Residual or error term of the ANOVA table. The combined variation will be the square root of the sum of the two variances. You could then compare the confidence intervals to the requirement.

If you require all units to meet the requirement, this will require a true capability study preceded by an evaluation of the process stability using a control chart. The recommended sample size for a capability study ranges from 50 - 100 printers depending on your information source.
 
Top Bottom