Validation of Commercial off-the-shelf software - Spreadsheets

[ISO 13485 - Medical]

In a recent audit we were picked up for the use of spreadsheet in the GxP environment and that although we had some controls around them we ha not validated them and also that the audit trail was poor.

Can anybody help on how we would go about validating the spreadsheets and also to maintain an audit trial?
Many Thanks in Advance

[harry]

Related threads:
IS0 13485:2003, Validation of the application of computer software

Quality System Database Validation - FDA's General Principles of Software Validation

[BradM]

In Reply to Parent Post by ISO 13485 - Medical

In a recent audit we were picked up for the use of spreadsheet in the GxP environment and that although we had some controls around them we ha not validated them and also that the audit trail was poor.

Can anybody help on how we would go about validating the spreadsheets and also to maintain an audit trial?
Many Thanks in Advance

Hello, there! As far as your query... good luck. Many different entities have contacted Microsoft in regards to validation parameters and the like, but to not avail. They state that the programs have been "validated" for their intended use, and not designed for purposes as defined by ER/ES issues and the like.

IMHO, it might be better to move to a validated software application. Many in the business are becoming more validation friendly, with many having IQ/OQ packages for sale.

If you are so inclined, you might want to share your application, and we can discuss different software applications for that purpose. But validating an Excel Spreadsheet (in terms of ER/ES and certain security measures) will be difficult.

Sorry I could not provide you better information on that one. Just for fun, are you certain that you need to validate your Excel application?

[Steve Prevette]

One question - are you really talking about validating Excel itself - or validating that the formulae on your spreadsheet do what they are supposed to do?

[Kevin Mader]

ISO 13485 Medical,

What you experienced is becoming more common place and can lead to a major nonconformance. Generally speaking, the focus is more on database applications such as Microsoft Access, but Word and Excel also fall into this category. There are many bolt on solutions available to organizations that help to support a more compliant environment as Brad mentions, but some are better than others. Do your homework! I’ve purchased off the shelf QMS software along with the validation package that they sell and found that the software was reasonably useful while the validation package hopeless. In the end, the software manufacturer learned a great deal from our validation to improve both their product and their validation package. Still, we are moving to another software solution as the bugs in the program are numerous and the software manufacturer unwilling to improve the product to the extent it can be fully validated. Validations aren’t cheap and you cannot keep up an indefinite practice of continuing validation. An auditor will look for a single problem to declare the software was inadequately validated. FYI: this validation effort we currently are under is in its 8th month with dedicated resources.

Harry has pointed you in the right direction. Read all you can on the subject but expand your reading to include the principles of process validation as well. QMS software is the combination of the product itself as it interfaces with the humans in the business processes. Validation should be expanded to cover both aspects and keep in mind that you are trying to achieve a validated state from two perspectives: design controls and process controls. Also, keep Steve’s comments in mind as well: are you validating the software, a spreadsheet, or both? Create an internal procedure with a decision making algorithm that allows you to determine from a risk-based perspective how much validation is necessary. A simple spreadsheet with limited imbedded formulas might require very little to support its use as intended. However, keep in mind that FDA’s view on hybrid systems tends to place higher scrutiny and emphasis on the software as opposed to the manual QMS processes it may be coupled with.

Good luck on your endeavor!

Kind regards,

Kevin

[vg_vvg]

I would request for further clarification on validation of "COTS" category of software products.

as per the general definition, any configurable application implemented in GXP environment would be categorised as Category 4 [GAMP Definitions of Validations requirements ]

How do we draw the line between "configuration" and "customisation"?

Especially where applications like SAP and LIMS are providing with built-in platforms for coding/altering the application to suit user/business/organisation requirements

[Kevin Mader]

vg_vvg,

Very good question – This could take a “group think” here to dial in on a good answer for you.

I tend to think about configuration in an application such as SAP as the process of establishing the pathway for decision making/routing information using the pre-established pathways/settings provided by the maker. Customization, however, is a more aggressive approach whereby the code within a supplier’s application must be rewritten in order to provide the user with the expected functionality/look. Many times this is done by folks/consultants not affiliated with the maker.

From an FDA perspective, the expectation is that the user performs the right amount of verification/validation at both the structural and functional levels. Many times we hear that we ought to use the software out-of-the-box without customization as anytime you change the structure of the software, you risk creating new ‘bugs’. This is probably good advice, but many times, impractical to the user. Additionally, many organizations do not want to disrupt their established workflows, especially if they have become fairly good at executing them. As a result, customization becomes more inevitable. That being said, the less you customize the software, the less validation (structural and functional) needs to be considered. However…

Many organizations do not do their proper homework. Using a software package out-of-the-box does not negate the need to perform validation (or consider it). Many operate under the assumption that since the ‘world is using this stuff’, validation isn’t necessary (you need to confirm that the supplier validated your scenario). The ‘world’, however, may not be using it in the same way you do. Such is the case with configurable software. Developed for a broader audience, they try to support a variety of functionality. However, not all software suppliers have thoroughly tested their product to the extent they should (as was the case I noted above). This should be an important note for those engaged in this thread should take away. Even with supplier supplied validation documentation and guarantees, the reality was different. This package was in no way customized for our use.

Performing a risk-based assessment helps to establish what and how-much validation is necessary. Taking this approach helps to save time, optimize resources and effort, and ultimately results in a better validation.

Back to the group...

Regards,

Kevin

[Pudge 72]

This is great topic.
In my mind, this is also a in many instances a great waste of time. No offense to the people who are doing it, but - let's think about this.
As this relates to second and third tier manufacturers and suppliers that provide Medical Device Companies with a products such as a machined component or a stamping, how many instances has Excel failed to correctly reproduce data that was originally entered thus causing a failure for a feature on a part?
The majority of what my customers seem to be telling me is that they need justification that a Microsoft application is not, assuming the data is entered correctly, manipulating the data and giving false output by which we are basing acceptance or rejection of a products feature on. To which I say - show me an instance where it did? I have never had any of them able to produce results of that nature. So, what are we doing here? We are now spending a lot of money and tying up resources because we have no instances by which to base true Risk Analysis? That's kind of like forgetting about SPC, FMEA and all of the other diagnostic tools in our toolbox and going on the gut isn't it? You can't show me a failure or impact, but you think the possibility is there, so, now we are going to do it?
It is mind boggling to me that as Quality Professionals we allow industry's to dictate excersises in futility and we just go along with it. Now, I have heard the story of the failure of the machine in the 80's that burned people from the inside out due to some software coding that was incorrectly interpreted in the programming. I do feel bad for the affected victims, but, give me an example of a device or a machine that failed based soley on the fact that data that was entered into Excel came out with a CPK Level of 1.35 instead of 1.28 because the background calculation that Microsoft uses failed thus causing a major issue in the field.
None of my Medical Customers can do it. In fact, none of them can provide me with any kind of a Risk Analysis that shows me how they arrived at deciding that Excel or any other Microsoft application needs validation as it relates to their product. These people should start to practice what they preach before they come and suck the resources out of our facilities.
As you can probably tell, this whole thing leaves a bad taste in my mouth and it certainly is not because I don't understand it - in fact, several of my customers have complimented me on my knowledge of GAMP, JETT, etc. etc. But, I still cannot find a document or instance which explains it without sounding like the tower of babble - which leads me to believe this is one of those CYA excersises in the Medical Device Industry - get out the pocketbooks everyone, but, just because it's here to stay doesn't make it right.