Calculating Ppk for Design Verification - Variable Sampling

[ga2qa23]

Hi all, I sure could please use some advice on how to perform variable sampling for design verification of a product. Too be honest, I'm not even sure how you'd do it, because you need to calculate your Ppk but design verification should occur BEFORE you start ramping up significant numbers of product to use for test samples.

If I manufactured a batch of 20 products and I had performed a 100% measurement/verification check (of continuous/variable data) so that I don't need to validate the manufacturing process, but I have to put all 20 products through a shipping test and confirm they still work with a variable sampling plan (for design verification), then can I calculate my Product Performance Index (Ppk) using data from the earlier 100% check from my manufacturing process?


To give some more details (if you're interested in my tale)...

My product is a medical device, but this question should still apply to non-medical device products. Basically, my product is super expensive and my company will only be making 20 of them to start with. We're doing a 100% verification check after assembly, so thus the manufacturing process will not be validated. However, one of our product requirements is that the finished product must survive shipping in our package and remain functional. Our product uses a gyroscope sensor to detect angles. So our functional test (after shipping) is to confirm that the product can detect angles within acceptable parameters of accuracy.

The only hang-up is that because this design verification activity is a true "test", not just a simple inspection/demonstration/analysis. Thus, the both the US FDA and European Union will expect us to have a statistical approach for this test (per the standard ISO 13485 - basically required by EU and FDA is currently transitioning to use the standard as well).

Sounds great, right? My test is measuring the angles that my product measures, which is continuous (variable) data. So I'd use a variable sampling plan, which allows for lower sample size, and I already know my specification limit. But I can't calculate a Product Performance Index (Ppk) without previous measurements! I have no historical data from similar products and a small population because the cost is so high. Can I just take the data from my earlier 100% check of the manufacturing process, and use that to calculate the Ppk in this new shipping test?

 

[indubioush]

I have not heard of that. Design verification sample size is based on risk. If your product is low risk, using 20 samples may be sufficient. Typically, design verification testing is done prior to design transfer. The manufacturing process may not be set and certainly is not required to be under any kind of statistical control. The only exception to this is a process that itself affects the design and where the output is unable to be verified.

 

[ga2qa23]

Thanks for responding! Our product is technically Class 2 but on the lower end. In my understanding, design transfer isn't set in stone and can start even before the design verification stage. This is because product can be designed/redesigned for manufacturing. Please let me know if I'm mistaken.

This isn't just a feature demonstration. In this situation, I'm treating the shipping/transit validation as a "Design Verification" activity. I also thought that "Design Verification" performed by testing requires a statistical approach/rationale. I'm working off of clauses in ISO 13485 and the book "Statistical Procedures for the Medical Device Industry" by Dr. Wayne A. Taylor.

For the shipping test, the prototype parts we'll ship in the test are high-fidelity (near-final) parts that are manufactured by the same vendor we chose for final product manufacturing. After shipping, we must ensure that the sensor can measure within X degrees of accuracy, and thus it's a "Design Verification" test that requires a sampling plan.

 

[Jim Wynne]

I don't see any way that Ppk (a dubious statistic to begin with) will help you. Ppk is a process metric, not product. I can't speak to the requirements for medical devices, but I think you're barking up the wrong tree with using a process capability approach.

 

[ga2qa23]

Doesn't Design Verification by test require a statistical approach/rationale, whether it's medical devices or the automotive industry?

I just chose a variable sampling plan because we're measuring how accurate a sensor is. Should I instead just use a t-test?

Both options assume a normal distribution of data. So I assumed that I'd still need to measure the data for all 20 units pre-shipping test, to confirm that the data is normal.

 

[Bev D]

Shipping tests do not typically require a statistical sample as they are usually what are called ‘directed’ tests. The standard approach is to submit the product to worst case temperature and stress conditions. Measure before and after and look for degradation or loss of accuracy.

Will you be submitting the verification data to your regulatory agency for review or will you be just filing the report to be bailable for review upon request?

 

[ga2qa23]

That is very interesting. Is the term "directed test" generally used outside of the semiconductor industry? I'm hesitant to apply that logic to medical devices. I'm not asking you for the answer, I'm just wondering if you've heard that term used in other industries.

I understood the aspect of worst-case / stress conditions testing. I was instead applying that logic to reduce my initial sample size. For example, I started with a sample size of 15 for variable sampling, and then reduced it by a factor of 5 to get a sample size of 3. This is because the shipping testing was determined to be a five-times stress test.

Our report would be filed away, for review upon request. For a medical device, you just tell the FDA in the 510(k) submission that you validated the shipping per a standard but don't give details. However, FDA still is supposed to audit you every 2 years afterward and has full access to your reports, and could potentially write up a "483 Report" on you if they find issue with it. Your failure can then become public knowledge through a FOIA (Freedom of Information Act) Request.

 

[Bev D]

I used to work in the medical device industry (Veterinary but based on FDA requirements). Not sure if directed testing is used in all industries but it is the basis of most of these tests that are required and covered in many many standards. and in many of these the sample size is one.

by the way it isn't valid to reduce your sample size by 5 because the stress is 5X....also you shouldn't use arbitrary stresses. understand what can actually happen in the worst case and use those stresses...that is what you have to justify to the auditors not the sample size.

Sample sizes are used when you don't know what the conditions are or how they vary. So you come up with a statistical sample size (based on confidence and reliability) and then ship those parts to diverse locals under diverse seasons...remember a statistical sample is only valid if it is randomly sampling the entire population of possible conditions.

 

[ga2qa23]

Thank you, I really appreciate your input. My only hangup is that if sample size for ship/transit testing should be one, then what about high-volume / low cost medical devices? Those manufacturers make millions or billions of "units" (medical device products) each year. So then they conduct testing, and they have a "shipping configuration" of say 30 units in a single box, then do they only strictly need to do a shipping test on one box? I find that very interesting.

 

[indubioush]

It all depends on risk. A sample size of one is not typical in my experience.

What are the risks associated with a failed product? If the risk to the user/patient is minimal, you may be comfortable with a lower sample size. You can also do a modified shipper with dummy devices in the low risk locations. Using your example of 30 units, a manufacturer could use multiple shippers that only have real devices in the corners and bottom, with equivalent weight in all other locations. There is no one-size-fits-all approach to this. You need to figure out what your risks are and what makes sense for your organization.