Computer system validation approach for Minitab Statistical software

[C. Tejeda]

I have been tasked with the process of validating Minitab for our facility. The company that I work for is a medical device manufacturer (Class I and II products) that needs to meet USA FDA Quality system requirements and ISO 13485.

As per our risk assessment, our main risk with using minitab is that it could yield wrong results (the software would be used to assist on the analysis fo process/design validation reports, CAPA investigations, and Complaint trends). We do aknowledge that a) Minitab has widespread adoption in the industry and b) it is a off-the-shelf (GAMP Category 3) software that will need black-box testing in order to check functionality.

My question is: Is it acceptable, for software validation test scripts, to test a reduced number of functions (lets say a sample) of all the functions minitab has available?

The acceptance criteria four our validation protocol would be something like test 10 functions, if they pass our test script, then the whole system is validated.

 

[ECHO]

Generally yes, you can using automated test scripts to validate 3rd party tools.

That said, I would first elaborate on the risk assessment you did above. As you stated, Minitab would be used in assisting other processes that will need to be manually reviewed. I would most likely consider this a low risk tool. Depending on how your internal tool validation SOP is written, you might be able to get away with not testing the tool.
If you do end up testing your tool, I would clearly state what functions are being tested and why other functions are not being tested.

 

[Steve Prevette]

I suggest that a lot depends upon if the calculations you perform in Minitab are life critical or safety critical. If so, I would target specific examples related to the modules used in Minitab for those functions rather than a global review of Minitab. For example, if I am reading dosimetry and computing a dose, I may want to have a zero or low dose example, an average or normal example, and an overdose example all calculated by independent means This is also a good opportunity to test the procedure for doing the calculation as you run the three cases through the process.

 

[Miner]

I have been tasked with the process of validating Minitab for our facility. The company that I work for is a medical device manufacturer (Class I and II products) that needs to meet USA FDA Quality system requirements and ISO 13485.

As per our risk assessment, our main risk with using minitab is that it could yield wrong results (the software would be used to assist on the analysis fo process/design validation reports, CAPA investigations, and Complaint trends). We do aknowledge that a) Minitab has widespread adoption in the industry and b) it is a off-the-shelf (GAMP Category 3) software that will need black-box testing in order to check functionality.

My question is: Is it acceptable, for software validation test scripts, to test a reduced number of functions (lets say a sample) of all the functions minitab has available?

The acceptance criteria four our validation protocol would be something like test 10 functions, if they pass our test script, then the whole system is validated.

Does this help? Minitab Products and Services and Compliance with CFR Title 21 – Part 11

A software validation kit is provided to tailor the validation to your specific needs.

 

[C. Tejeda]

Thank you Miner, I've got Minitab's validation kit. My question is if it is common practice, in this type of off-the-shelf software, to do an "abbreviated" approach to testing.

I suggest that a lot depends upon if the calculations you perform in Minitab are life critical or safety critical. If so, I would target specific examples related to the modules used in Minitab for those functions rather than a global review of Minitab. For example, if I am reading dosimetry and computing a dose, I may want to have a zero or low dose example, an average or normal example, and an overdose example all calculated by independent means This is also a good opportunity to test the procedure for doing the calculation as you run the three cases through the process.

My idea is similar to what Steve proposed. Basically using the predefined test cases provided by Minitab's validation kit but focusing only key functions that we use often instead of testing 100% of functions. I'd consider the software as validated based on these results. What do you think?

 

[Tidge]

I have been tasked with the process of validating Minitab for our facility. The company that I work for is a medical device manufacturer (Class I and II products) that needs to meet USA FDA Quality system requirements and ISO 13485.

As per our risk assessment, our main risk with using minitab is that it could yield wrong results (the software would be used to assist on the analysis fo process/design validation reports, CAPA investigations, and Complaint trends).

Unless Minitab is actually making decisions for your company, you may have overestimated the risk of the software. Usually it's a human making the call and they don't defer to a software package (if they are qualified to understand the tool). Write what we must about the "lowest common denominator" but the typical user of minitab is not a complete knucklehead.

We (medical device manufacturers) are actually in the process of updating the validation status of Minitab; we plan to leverage the validation package provided, and we will further enhance our validation by applying a number of typical, foundational techniques for which there already exists "expected results" as extra assurance that the latest version is still behaving as past (validated) versions. We aren't the quality control department for Minitab; we're just folks familiar with the variety of things we're going to use Minitab for... and know how to check our own work (and that of the software package).

 

[Hi_Its_Matt]

As others have mentioned, your company's approach to computer system validation should be risk-based.
What approaches you take, and what evidence you generate/require is more or less up to you, so long as they are based on those risks.

Having said that, the FDA is expected to release a draft guidance this year about Computer Software Assurance, their "new term" for Computer System Validation. (Really its not just a new term, but rather a new paradigm for managing risks associated with software used in production and quality system support).

Compliance Group Inc has a great collection of YouTube videos/discussions that discusses what CSA entails, and several real-life case studies from manufacturers that have changed their approach. Several of them even have the FDA CDRH's Case for Quality program manager (Francisco Vicenty) as a participant.

USDM has also written a whitepaper that talks about the upcoming draft guidance, which I have attached here for everyone's ease.

My impression, based on this material, is that a lot of companies seem to have gone completely overboard with their approach to computer system validation, and have focused more on compliance rather than true quality (can anyone say "non-value add busywork"?).
The FDA is now clarifying that it encourages manufacturers to take a more holistic approach to assuring that computer software is appropriate for its intended use, rather than attempting to independently verify that every function and feature of a particular software tool works.

So @C. Tejeda with that, my question back to you is, based on (1) how your company uses Minitab, (2) it's overall role and purpose within your QMS, (3) the specific risks that are involved with its use, (4) the non-software based controls you have in place external to Minitab to manage any risks should they materialize, and (5) Minitab's standing within the industry as one of, if not the leading statistical analysis tools, do you think that validation test scripts are even required, in order to have a high degree of confidence that the tool will operate as you need it to?

edit: I am not affiliated with either Compliance Group or USDM. I just found these resources particularly useful as I have been working to re-vamp my company's approach this topic.

 

[Tidge]

My impression, based on this material, is that a lot of companies seem to have gone completely overboard with their approach to computer system validation, and have focused more on compliance rather than true quality (can anyone say "non-value add busywork"?).

Your impression is NOT wrong (for the Medical Device industry, other industries may have different experiences). A foundational reason for where we find ourselves in the morass is that for NPS it is has been incredibly rare for folks to intelligently answer "Compliant to what?" This is, of course, something of a trick question.

I personally witness a host of diverse root causes for how the (US, likely elsewhere) medical device industry ended up with the NPS path that the FDA is desperate to course-correct. The most polite reason, assuming competence and good intentions, is that the "waterfall model", "inputs-trace-to-outputs", and "we work in an industry that saves lives" all got tangled up in such a way that it required real backbone to answer "no" to questions like "how about one more screenshot?"

If competence is not guaranteed, and intentions are less-than-pure, the situation only gets more problematic.

 

[patilrahuld]

Is it possible to share a template of CS validation or an example? Apologies if I am asking too much but need some urgent help with CSV and don't have any baseline to work with.

 

[Tidge]

Is it possible to share a template of CS validation or an example? Apologies if I am asking too much but need some urgent help with CSV and don't have any baseline to work with.

You can request the validation kit for MSS directly from Minitab. The templates they provide are shockingly reminiscent to templates used by multiple medical device manufacturers, dating back at least a decade.