Anyone with experience in open-source firmware in medical devices

[Le Chiffre]

I'm looking for some guidance applying FDA's software validation recommendations to open-source software used in an embedded application within a class III medical device.

Much of the operating system and device drivers come for free, downloaded off the web, from the open-source community (controlled by organisations such as "kernel.org"). Another open-source community has provided adaptations to allow the software to run on non-Intel processors for embedded applications. The tools to build the kernel are also open-source and free, and in most cases are built specifically for the target from "standard" resources.

As you can imagine, there's a lot of software here that comes with no explicit warranty, and cannot be easily inspected or validated other than when tested as a system running our application. Our intention is to describe, review and test only the code that we've written. The rest would be tested during system validation. At the moment, even code reviewing our code appears unrealistic given the time it requires. Do you have any comments, or recommendations regarding this?

 

[Al Rosen]

As you can imagine, there's a lot of software here that comes with no explicit warranty, and cannot be easily inspected or validated other than when tested as a system running our application. Our intention is to describe, review and test only the code that we've written. The rest would be tested during system validation.

The answer is in General Principles of Software Validation

When device manufacturers purchase "off-the-shelf'' software, they must ensure that it will perform as intended in their chosen application. For off-the-shelf software used in manufacturing or in the quality system, additional guidance is included in Section 6.3 of this document. For device software, additional useful information may be found in FDA’s Guidance for Industry, FDA Reviewers, and Compliance on Off-The-Shelf Software Use in Medical Devices.

 

[Le Chiffre]

Thanks again Al,

I've read the FDA documents you mention but still felt uneasy treating open-source as conventional off-the-shelf since there's no responsibility for control.

Since we don't intend to code review the OTS software, the FDA's guidelines include audit's of the software vendor and vendor-supplied information. In the case of open-source, there is no vendor.

What are the expectations for code reviewing? 100% coverage of the device manufacturer's contribution, or is it possible to justify a lesser amount?

 

[pldey42]

Hi (Bonjour) Le Chiffre,

I have experience of open source software, but not of medical devices. As I understand it, class 3 medical devices support or sustain human life, so I imagine you'd like the software not to fail ;-)

On open source software in general: software costs time and effort to maintain. Many companies that use open source software engage engineers to maintain it, because they cannot be sure the open source community will fix the bugs they care about and add new features in the timescales their customers demand. They find it's not really "free" and the costs of maintaining it, in terms of software engineering effort, can be as high as those of purchasing a commercial product. (I did the math once, it was scary.) That's not to say open source is bad, it can be good and there can be benefits; but the apparently-free downloads can be deceptive in terms of costs. What this means is you might want to identify one or a few engineers to ride herd on the open source stuff, and thoroughly validate new versions before incorporating them into your product.

On verifying it: you could inspect the code since it's open source, but I imagine that would be expensive and low in value because there would be too much to learn. I assume you'll test it. I would caution you on managing changes: when you download a new version, regression test it very carefully to make sure that everything still works properly. One open source product I dealt with a few years ago was notorious for being unstable: every new feature would break one or more existing features. The "goodness" of the software quality depends very much on the team leading the open source development.

On inspecting your own code: research indicates that code inspection is very cost-effective at finding bugs. In real time embedded systems, which are common in medical devices, there are bugs (to do with synchronising real time events and memory management) that simply cannot be found in testing, occur once in a blue moon (that once could kill someone) and can easily be found in code inspections (e.g. for every allocation of memory, did we return it to the heap when finished?). Cost-effective code inspection doesn't usually mean inspecting all the code. Typically I'd recommend sampling the complex code, the brand new code, and stuff written by novices; often, the engineers will know what they'd like someone to take a second look over. If you find errors, sample more of that kind of code; if not, less.

Here's an example of code inspection: will this program work?

main ()
{
printf ("hello world);
}

Almost everyone spots the missing closing quotes -- even people who can't read C.

A final thought: What's the cost if you let bugs out into the field? That would have some bearing on what it's worth to manage the software quality, I'd imagine.

Hope this helps,
Patrick

 

[Le Chiffre]

Merci Patrick!

You're the first person who's suggested sampling when it comes to code inspections. But like any element of the QMS, the process should be justified. If a significant effort spent on code reviews shows nothing, I'd feel justified in reducing the scope.

This brings up another interesting debate. I once worked with an organisation that performed code inspections even before it was compiled - which IMO wasted experts time pointing out typos that the compiler would have found (like your hello world example). But you're absolutely right that by inspection, we can exercise hypothetical situations that may be extremely rare and difficult to synthesize in testing.

A final thought: What's the cost if you let bugs out into the field? That would have some bearing on what it's worth to manage the software quality, I'd imagine.

Unfortunately that's now irrelevant as the project costs have already been assessed (by the bean-counters) - our job is to do the best with what we have.

 

[pldey42]

Merci Patrick!

You're the first person who's suggested sampling when it comes to code inspections. But like any element of the QMS, the process should be justified. If a significant effort spent on code reviews shows nothing, I'd feel justified in reducing the scope.

Yes, absolutely. I've written code inspection procedures that have satisfied external ISO 9001 auditors where we used sampling. The documented rules were something like this:

Team leader determines which source files will be inspected. A minimum of x% of code per release will be inspected. Select files based on experience of developer, complexity of code. (Thus, code sampling was a repeatable process that could be audited.)

Then, to avoid spending effort when reviews found nothing, we noted in review records how much time we spent, how many bugs we found, and adjusted the sampling rules accordingly. At one point we even said something like, "If, after half an hour, you find no bugs, move to the next file."

So, for example, for some maintenance releases with small bits of easy code, we'd skip the reviews because we knew we had experienced people fixing simple stuff. (Well it should have been simple but, well, stuff happens.)

This brings up another interesting debate. I once worked with an organisation that performed code inspections even before it was compiled - which IMO wasted experts time pointing out typos that the compiler would have found (like your hello world example). But you're absolutely right that by inspection, we can exercise hypothetical situations that may be extremely rare and difficult to synthesize in testing.

Yes, I agree. All my inspections were done on code that compiles without errors or signficant warnings. In the old days, when it could take a day to compile and link stuff, it was worth it; but not now, surely.

Unfortunately that's now irrelevant as the project costs have already been assessed (by the bean-counters) - our job is to do the best with what we have.

Hmm ... I can understand that, but -- I don't know what you make so I'm guessing -- if there's a software failure that results in loss of life, the PR impact on the product will surely kill it, if the insurance claims don't. That might not be enough to increase your pile of pre-counted beans, but it might influence priorities on how they're spent? And, bean counters always have a secret stash of beans somewhere. Sometimes they can be persuaded to part with a few more if they think they'll lose a whole bean bag if they don't ;-)

Hope this helps,
Patrick