SBS - The best value in QMS software

Anyone with experience in open-source firmware in medical devices

Le Chiffre

Quite Involved in Discussions
#1
I'm looking for some guidance applying FDA's software validation recommendations to open-source software used in an embedded application within a class III medical device.

Much of the operating system and device drivers come for free, downloaded off the web, from the open-source community (controlled by organisations such as "kernel.org"). Another open-source community has provided adaptations to allow the software to run on non-Intel processors for embedded applications. The tools to build the kernel are also open-source and free, and in most cases are built specifically for the target from "standard" resources.

As you can imagine, there's a lot of software here that comes with no explicit warranty, and cannot be easily inspected or validated other than when tested as a system running our application. Our intention is to describe, review and test only the code that we've written. The rest would be tested during system validation. At the moment, even code reviewing our code appears unrealistic given the time it requires. Do you have any comments, or recommendations regarding this?
 
Elsmar Forum Sponsor

Al Rosen

Staff member
Super Moderator
#2
Le Chiffre said:
As you can imagine, there's a lot of software here that comes with no explicit warranty, and cannot be easily inspected or validated other than when tested as a system running our application. Our intention is to describe, review and test only the code that we've written. The rest would be tested during system validation.
The answer is in General Principles of Software Validation

When device manufacturers purchase "off-the-shelf'' software, they must ensure that it will perform as intended in their chosen application. For off-the-shelf software used in manufacturing or in the quality system, additional guidance is included in Section 6.3 of this document. For device software, additional useful information may be found in FDA’s Guidance for Industry, FDA Reviewers, and Compliance on Off-The-Shelf Software Use in Medical Devices.
 

Le Chiffre

Quite Involved in Discussions
#3
Thanks again Al,

I've read the FDA documents you mention but still felt uneasy treating open-source as conventional off-the-shelf since there's no responsibility for control.

Since we don't intend to code review the OTS software, the FDA's guidelines include audit's of the software vendor and vendor-supplied information. In the case of open-source, there is no vendor.

What are the expectations for code reviewing? 100% coverage of the device manufacturer's contribution, or is it possible to justify a lesser amount?
 
P

pldey42

#4
Hi (Bonjour) Le Chiffre,

I have experience of open source software, but not of medical devices. As I understand it, class 3 medical devices support or sustain human life, so I imagine you'd like the software not to fail ;-)

On open source software in general: software costs time and effort to maintain. Many companies that use open source software engage engineers to maintain it, because they cannot be sure the open source community will fix the bugs they care about and add new features in the timescales their customers demand. They find it's not really "free" and the costs of maintaining it, in terms of software engineering effort, can be as high as those of purchasing a commercial product. (I did the math once, it was scary.) That's not to say open source is bad, it can be good and there can be benefits; but the apparently-free downloads can be deceptive in terms of costs. What this means is you might want to identify one or a few engineers to ride herd on the open source stuff, and thoroughly validate new versions before incorporating them into your product.

On verifying it: you could inspect the code since it's open source, but I imagine that would be expensive and low in value because there would be too much to learn. I assume you'll test it. I would caution you on managing changes: when you download a new version, regression test it very carefully to make sure that everything still works properly. One open source product I dealt with a few years ago was notorious for being unstable: every new feature would break one or more existing features. The "goodness" of the software quality depends very much on the team leading the open source development.

On inspecting your own code: research indicates that code inspection is very cost-effective at finding bugs. In real time embedded systems, which are common in medical devices, there are bugs (to do with synchronising real time events and memory management) that simply cannot be found in testing, occur once in a blue moon (that once could kill someone) and can easily be found in code inspections (e.g. for every allocation of memory, did we return it to the heap when finished?). Cost-effective code inspection doesn't usually mean inspecting all the code. Typically I'd recommend sampling the complex code, the brand new code, and stuff written by novices; often, the engineers will know what they'd like someone to take a second look over. If you find errors, sample more of that kind of code; if not, less.

Here's an example of code inspection: will this program work?

main ()
{
printf ("hello world);
}

Almost everyone spots the missing closing quotes -- even people who can't read C.

A final thought: What's the cost if you let bugs out into the field? That would have some bearing on what it's worth to manage the software quality, I'd imagine.

Hope this helps,
Patrick
 

Le Chiffre

Quite Involved in Discussions
#5
Merci Patrick!

You're the first person who's suggested sampling when it comes to code inspections. But like any element of the QMS, the process should be justified. If a significant effort spent on code reviews shows nothing, I'd feel justified in reducing the scope.

:topic: This brings up another interesting debate. I once worked with an organisation that performed code inspections even before it was compiled - which IMO wasted experts time pointing out typos that the compiler would have found (like your hello world example). But you're absolutely right that by inspection, we can exercise hypothetical situations that may be extremely rare and difficult to synthesize in testing.

pldey42 said:
A final thought: What's the cost if you let bugs out into the field? That would have some bearing on what it's worth to manage the software quality, I'd imagine.
Unfortunately that's now irrelevant as the project costs have already been assessed (by the bean-counters) - our job is to do the best with what we have.
 
P

pldey42

#6
Le Chiffre said:
Merci Patrick!

You're the first person who's suggested sampling when it comes to code inspections. But like any element of the QMS, the process should be justified. If a significant effort spent on code reviews shows nothing, I'd feel justified in reducing the scope.
:)

Yes, absolutely. I've written code inspection procedures that have satisfied external ISO 9001 auditors where we used sampling. The documented rules were something like this:

Team leader determines which source files will be inspected. A minimum of x% of code per release will be inspected. Select files based on experience of developer, complexity of code. (Thus, code sampling was a repeatable process that could be audited.)

Then, to avoid spending effort when reviews found nothing, we noted in review records how much time we spent, how many bugs we found, and adjusted the sampling rules accordingly. At one point we even said something like, "If, after half an hour, you find no bugs, move to the next file."

So, for example, for some maintenance releases with small bits of easy code, we'd skip the reviews because we knew we had experienced people fixing simple stuff. (Well it should have been simple but, well, stuff happens.)

:topic: This brings up another interesting debate. I once worked with an organisation that performed code inspections even before it was compiled - which IMO wasted experts time pointing out typos that the compiler would have found (like your hello world example). But you're absolutely right that by inspection, we can exercise hypothetical situations that may be extremely rare and difficult to synthesize in testing.
Yes, I agree. All my inspections were done on code that compiles without errors or signficant warnings. In the old days, when it could take a day to compile and link stuff, it was worth it; but not now, surely.

Unfortunately that's now irrelevant as the project costs have already been assessed (by the bean-counters) - our job is to do the best with what we have.
Hmm ... I can understand that, but -- I don't know what you make so I'm guessing -- if there's a software failure that results in loss of life, the PR impact on the product will surely kill it, if the insurance claims don't. That might not be enough to increase your pile of pre-counted beans, but it might influence priorities on how they're spent? And, bean counters always have a secret stash of beans somewhere. Sometimes they can be persuaded to part with a few more if they think they'll lose a whole bean bag if they don't ;-)

Hope this helps,
Patrick
 
Thread starter Similar threads Forum Replies Date
D Does anyone have experience using open source software for PDM? ISO 13485:2016 - Medical Device Quality Management Systems 1
N Does anyone have experience of GB/T 34986-2017? China Medical Device Regulations 1
Z Does anyone have experience with EN ISO 17664 ? IEC 62366 - Medical Device Usability Engineering 9
M Honda Audits - Does anyone have any experience with the QAV audit? General Auditing Discussions 3
Q Does anyone have experience implementing a QMS without ISO certification? Quality Manager and Management Related Issues 2
QIE Anyone have experience with BPA Quality (QMS system based on Office365/Sharepoint) Quality Assurance and Compliance Software Tools and Solutions 1
W Does anyone have any experience with the Easy Metric System? General Measurement Device and Calibration Topics 2
A Anyone have experience with Transmille calibrators General Measurement Device and Calibration Topics 1
D Has anyone here had any experience with PQ-FMEA software? FMEA and Control Plans 1
S Has anyone completed IATF 16949 Certification - Share your Audit Experience? IATF 16949 - Automotive Quality Systems Standard 2
E Does anyone have experience implementing a Tiered QMS? AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 3
R Does anyone have any experience on change control? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
ScottK Does anyone have experience with how MDD unanounced audits of suppliers are going? EU Medical Device Regulations 21
A Does anyone have experience providing a summary of Pre-IDE discussions in the 510k US Food and Drug Administration (FDA) 1
A MDS software - Does anyone have experience with iPoint? RoHS, REACH, ELV, IMDS and Restricted Substances 8
M Does anyone here have experience implementing PCI DSS (Data Security Standard) IEC 27001 - Information Security Management Systems (ISMS) 10
GStough Rx-360 - Has Anyone Used This and What Was Your Experience? Supplier Quality Assurance and other Supplier Issues 3
J Does anyone have experience in submitting TSE Dossiers to EDQM? EU Medical Device Regulations 1
Colin Anyone with Experience of BS EN 1090? CE Marking (Conformité Européene) / CB Scheme 7
P Anyone have experience with GLOBAL GAP (Global Good Agricultural Practices)?? US Food and Drug Administration (FDA) 5
C South Korean KFDA's KGMP Audit- Anyone with Firsthand Experience to share? Other Medical Device Regulations World-Wide 4
E Anyone have experience with UL as ISO 13485/CMDCAS and MDD registrar Registrars and Notified Bodies 17
C Anyone have experience handling Feed Safety Management System (GMP+B2)? Other ISO and International Standards and European Regulations 1
J SFDA Product Quality Tracking Report - Anyone with experience to share? China Medical Device Regulations 2
J Integrated Management System - Anyone with experience dealing with ISOQAR? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
M Has anyone of you have experience with the World Class Manufacturing system? Lean in Manufacturing and Service Industries 6
DietCokeofEvil Anyone have any experience with a Helios UMG 50? General Measurement Device and Calibration Topics 1
K Does anyone has experience with determining if Lubricants and Greases are Food Grade? Food Safety - ISO 22000, HACCP (21 CFR 120) 6
I Does anyone here have experience with ISO 3951? Other ISO and International Standards and European Regulations 3
B Does anyone here have any experience with the Deming Prize? Quality Tools, Improvement and Analysis 1
K Has anyone had experience dealing with SEDEX (Supplier Ethical Data Exchange)? Customer and Company Specific Requirements 4
F Anyone have experience of a hiring a Social Anthropologist? Human Factors and Ergonomics in Engineering 9
N Does anyone have any experience or feedback using mind mapping software? Quality Tools, Improvement and Analysis 5
L Onsite Audit (Shenzen, China) - Anyone have any experience with a consultant? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 3
E Does anyone have experience with ProWorksinfo software for Work Instructions? Quality Assurance and Compliance Software Tools and Solutions 1
X Anyone w/experience - Internal part Chem Labels in a Sensitive Info. environment? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
G Polyester Biocompatibility standards - Anyone here have any experience? ISO 13485:2016 - Medical Device Quality Management Systems 1
C Anyone with experience working with/for Hyundai? Career and Occupation Discussions 2
I Does anyone have direct experience with BVQI (Brazil)? Registrars and Notified Bodies 3
H Anyone here with experience with American Institute of Steel Construction (AISC) Various Other Specifications, Standards, and related Requirements 1
R Anyone willing to share their experience in Knowledge Management (KM)? Lean in Manufacturing and Service Industries 3
Ajit Basrur ISO 15378 certification - Has anyone experience with ISO 15378? ISO 13485:2016 - Medical Device Quality Management Systems 4
D Has anyone had experience with a Portable XRF Analyzer? General Measurement Device and Calibration Topics 4
T Experience with 7i Datastream software? Anyone? Calibration and Metrology Software and Hardware 4
P FARO Arm & Laser Tracker - Anyone with experience? Software question General Measurement Device and Calibration Topics 6
K Has anyone had any experience with Rosettanet? Various Other Specifications, Standards, and related Requirements 1
T Has anyone had any experience with Kennedy Western University? Professional Certifications and Degrees 11
J Anyone with experience with the Chrysler APQP 10 point gate system? Customer and Company Specific Requirements 1
L Notified Body - Does anyone have experience using TUV Essex? Registrars and Notified Bodies 2
Hershal Anyone have experience with ISO/IEC 15189 for medical laboratories? US Food and Drug Administration (FDA) 12

Similar threads

Top Bottom