Validation of Commercial off-the-shelf software - Spreadsheets

ISO 13485 - Medical

Involved In Discussions
In a recent audit we were picked up for the use of spreadsheet in the GxP environment and that although we had some controls around them we ha not validated them and also that the audit trail was poor.

Can anybody help on how we would go about validating the spreadsheets and also to maintain an audit trial?
Many Thanks in Advance
 

BradM

Leader
Admin
In a recent audit we were picked up for the use of spreadsheet in the GxP environment and that although we had some controls around them we ha not validated them and also that the audit trail was poor.

Can anybody help on how we would go about validating the spreadsheets and also to maintain an audit trial?
Many Thanks in Advance

Hello, there! As far as your query... good luck:D. Many different entities have contacted Microsoft in regards to validation parameters and the like, but to not avail. They state that the programs have been "validated" for their intended use, and not designed for purposes as defined by ER/ES issues and the like.

IMHO, it might be better to move to a validated software application. Many in the business are becoming more validation friendly, with many having IQ/OQ packages for sale.

If you are so inclined, you might want to share your application, and we can discuss different software applications for that purpose. But validating an Excel Spreadsheet (in terms of ER/ES and certain security measures) will be difficult.

Sorry I could not provide you better information on that one.:) Just for fun, are you certain that you need to validate your Excel application?
 

Steve Prevette

Deming Disciple
Leader
Super Moderator
One question - are you really talking about validating Excel itself - or validating that the formulae on your spreadsheet do what they are supposed to do?
 

Kevin Mader

One of THE Original Covers!
Leader
Admin
ISO 13485 Medical,

What you experienced is becoming more common place and can lead to a major nonconformance. Generally speaking, the focus is more on database applications such as Microsoft Access, but Word and Excel also fall into this category. There are many bolt on solutions available to organizations that help to support a more compliant environment as Brad mentions, but some are better than others. Do your homework! I’ve purchased off the shelf QMS software along with the validation package that they sell and found that the software was reasonably useful while the validation package hopeless. In the end, the software manufacturer learned a great deal from our validation to improve both their product and their validation package. Still, we are moving to another software solution as the bugs in the program are numerous and the software manufacturer unwilling to improve the product to the extent it can be fully validated. Validations aren’t cheap and you cannot keep up an indefinite practice of continuing validation. An auditor will look for a single problem to declare the software was inadequately validated. FYI: this validation effort we currently are under is in its 8th month with dedicated resources.

Harry has pointed you in the right direction. Read all you can on the subject but expand your reading to include the principles of process validation as well. QMS software is the combination of the product itself as it interfaces with the humans in the business processes. Validation should be expanded to cover both aspects and keep in mind that you are trying to achieve a validated state from two perspectives: design controls and process controls. Also, keep Steve’s comments in mind as well: are you validating the software, a spreadsheet, or both? Create an internal procedure with a decision making algorithm that allows you to determine from a risk-based perspective how much validation is necessary. A simple spreadsheet with limited imbedded formulas might require very little to support its use as intended. However, keep in mind that FDA’s view on hybrid systems tends to place higher scrutiny and emphasis on the software as opposed to the manual QMS processes it may be coupled with.

Good luck on your endeavor!

Kind regards,

Kevin
 
V

vg_vvg

I would request for further clarification on validation of "COTS" category of software products.

as per the general definition, any configurable application implemented in GXP environment would be categorised as Category 4 [GAMP Definitions of Validations requirements ]

How do we draw the line between "configuration" and "customisation"?

Especially where applications like SAP and LIMS are providing with built-in platforms for coding/altering the application to suit user/business/organisation requirements
 

Kevin Mader

One of THE Original Covers!
Leader
Admin
vg_vvg,

Very good question – This could take a “group think” here to dial in on a good answer for you.

I tend to think about configuration in an application such as SAP as the process of establishing the pathway for decision making/routing information using the pre-established pathways/settings provided by the maker. Customization, however, is a more aggressive approach whereby the code within a supplier’s application must be rewritten in order to provide the user with the expected functionality/look. Many times this is done by folks/consultants not affiliated with the maker.

From an FDA perspective, the expectation is that the user performs the right amount of verification/validation at both the structural and functional levels. Many times we hear that we ought to use the software out-of-the-box without customization as anytime you change the structure of the software, you risk creating new ‘bugs’. This is probably good advice, but many times, impractical to the user. Additionally, many organizations do not want to disrupt their established workflows, especially if they have become fairly good at executing them. As a result, customization becomes more inevitable. That being said, the less you customize the software, the less validation (structural and functional) needs to be considered. However…

Many organizations do not do their proper homework. Using a software package out-of-the-box does not negate the need to perform validation (or consider it). Many operate under the assumption that since the ‘world is using this stuff’, validation isn’t necessary (you need to confirm that the supplier validated your scenario). The ‘world’, however, may not be using it in the same way you do. Such is the case with configurable software. Developed for a broader audience, they try to support a variety of functionality. However, not all software suppliers have thoroughly tested their product to the extent they should (as was the case I noted above). This should be an important note for those engaged in this thread should take away. Even with supplier supplied validation documentation and guarantees, the reality was different. This package was in no way customized for our use.

Performing a risk-based assessment helps to establish what and how-much validation is necessary. Taking this approach helps to save time, optimize resources and effort, and ultimately results in a better validation.

Back to the group...

Regards,

Kevin
 
P

Pudge 72

This is great topic.
In my mind, this is also a in many instances a great waste of time. No offense to the people who are doing it, but - let's think about this.
As this relates to second and third tier manufacturers and suppliers that provide Medical Device Companies with a products such as a machined component or a stamping, how many instances has Excel failed to correctly reproduce data that was originally entered thus causing a failure for a feature on a part?
The majority of what my customers seem to be telling me is that they need justification that a Microsoft application is not, assuming the data is entered correctly, manipulating the data and giving false output by which we are basing acceptance or rejection of a products feature on. To which I say - show me an instance where it did? I have never had any of them able to produce results of that nature. So, what are we doing here? We are now spending a lot of money and tying up resources because we have no instances by which to base true Risk Analysis? That's kind of like forgetting about SPC, FMEA and all of the other diagnostic tools in our toolbox and going on the gut isn't it? You can't show me a failure or impact, but you think the possibility is there, so, now we are going to do it?
It is mind boggling to me that as Quality Professionals we allow industry's to dictate excersises in futility and we just go along with it. Now, I have heard the story of the failure of the machine in the 80's that burned people from the inside out due to some software coding that was incorrectly interpreted in the programming. I do feel bad for the affected victims, but, give me an example of a device or a machine that failed based soley on the fact that data that was entered into Excel came out with a CPK Level of 1.35 instead of 1.28 because the background calculation that Microsoft uses failed thus causing a major issue in the field.
None of my Medical Customers can do it. In fact, none of them can provide me with any kind of a Risk Analysis that shows me how they arrived at deciding that Excel or any other Microsoft application needs validation as it relates to their product. These people should start to practice what they preach before they come and suck the resources out of our facilities.
As you can probably tell, this whole thing leaves a bad taste in my mouth and it certainly is not because I don't understand it - in fact, several of my customers have complimented me on my knowledge of GAMP, JETT, etc. etc. But, I still cannot find a document or instance which explains it without sounding like the tower of babble - which leads me to believe this is one of those CYA excersises in the Medical Device Industry - get out the pocketbooks everyone, but, just because it's here to stay doesn't make it right.
 
D

danpa

ISO 13485 - Medical,
What audit trail where you dinged for? Was it the tracking of changes to the formulas and spreadsheet structure, or the tracking of the data entry to the spreadsheet? I would handle the former via document control practices. I don't have a good general solution for the latter.

I think it is very important to keep Steve Prevette's piont in mind that generally people need to validate the particular formulas and final results of the spreadsheet and not the whole of Excel. Part of the validation can be having a review of the spreadsheet (looking at each formula to make sure it is written properly) and performing web searches to see if any of the functions used in your spreadsheet have any known defects that would effect your use of them. Follow this up with some testing of your particular spreadsheet and you have gone a long way at validating your use of Excel (IMHO).

One last point is to perform a safety risk analysis on your use of the Excel spreadsheet. The higher the risk the more in-depth your validation. For low risk applications a review and testing may be sufficient.

:2cents:
 

Doug Tropf

Quite Involved in Discussions
In a recent audit we were picked up for the use of spreadsheet in the GxP environment and that although we had some controls around them we ha not validated them and also that the audit trail was poor.

Can anybody help on how we would go about validating the spreadsheets and also to maintain an audit trial?
Many Thanks in Advance
The March 2008 MD&DI magazine has an excellent article on software validations.
 
Top Bottom