Verification and Validation as a Solo Engineer

[mr9000]

Hello everyone,
I recently started freelancing in the development of medical devices, and I'm currently preparing for my first major project: a complete overhaul of an existing product. This includes new electronics, new software, full documentation, and support in getting the company’s QMS ready for MDR compliance.
From an engineering perspective, I’m not concerned—the device itself is relatively simple, and it's not high-risk. In fact, it doesn’t even have essential performance as defined in the MDR; it's used to support healing after surgery or during chemotherapy.
However, I do have some concerns about the Verification and Validation (V&V) process, particularly because I’m working alone. My current plan looks like this:

• Develop a detailed V&V plan, covering unit/component, integration, and system-level verification
• At the component level, testing will mostly reference datasheets and manufacturer specifications
• For software unit testing, I plan to rely on clean compilation (no warnings/errors) and static code analysis (based on a modified MISRA rule set)
• Integration and system-level testing will be done using a Hardware-in-the-Loop (HIL) setup that allows simulation and measurement of all inputs and outputs
• Basic user-level testing to validate expected behavior (e.g., "press this button – does the system do what it should?")

Wherever possible, I’ll link test results to physical values measured using calibrated tools.

My main concern is how to validate the tests themselves. Since I’m the only (external) engineer, there’s no one available to review code or approve test protocols. Of course, I could ask my contact person at the company to sign off on everything, but they aren’t a technical person—it would just be a formality, and I’d rather avoid that kind of box-ticking.
I could write all kinds of unit tests, but at some point the question becomes: Who verifies the verification?
This isn’t about basic safety (e.g., IEC 60601-1 will be tested in a lab), but more about the software and overall system V&V process.

Do you have any advice or best practices for handling V&V as a solo engineer?

Best regards

 

[d_addams]

Hello everyone,
I recently started freelancing in the development of medical devices, and I'm currently preparing for my first major project: a complete overhaul of an existing product. This includes new electronics, new software, full documentation, and support in getting the company’s QMS ready for MDR compliance.
From an engineering perspective, I’m not concerned—the device itself is relatively simple, and it's not high-risk. In fact, it doesn’t even have essential performance as defined in the MDR; it's used to support healing after surgery or during chemotherapy.
However, I do have some concerns about the Verification and Validation (V&V) process, particularly because I’m working alone. My current plan looks like this:

• Develop a detailed V&V plan, covering unit/component, integration, and system-level verification
• At the component level, testing will mostly reference datasheets and manufacturer specifications
• For software unit testing, I plan to rely on clean compilation (no warnings/errors) and static code analysis (based on a modified MISRA rule set)
• Integration and system-level testing will be done using a Hardware-in-the-Loop (HIL) setup that allows simulation and measurement of all inputs and outputs
• Basic user-level testing to validate expected behavior (e.g., "press this button – does the system do what it should?")

Wherever possible, I’ll link test results to physical values measured using calibrated tools.

My main concern is how to validate the tests themselves. Since I’m the only (external) engineer, there’s no one available to review code or approve test protocols. Of course, I could ask my contact person at the company to sign off on everything, but they aren’t a technical person—it would just be a formality, and I’d rather avoid that kind of box-ticking.
I could write all kinds of unit tests, but at some point the question becomes: Who verifies the verification?
This isn’t about basic safety (e.g., IEC 60601-1 will be tested in a lab), but more about the software and overall system V&V process.

Do you have any advice or best practices for handling V&V as a solo engineer?

Best regards

no device can have no essential performance. Loss of essential performance does not result in harm/injury, it results in loss of benefit. Loss of essential performance results in unacceptable risk since there is no longer benefit to outweigh the risks. Thus, all devices must have essential performance. If not meeting a performance requirement results in a harm, it is safety performance characteristic not an essential performance characteristic.

I'd be open to hearing arguments to the contrary.

Regarding your direct question, have the client hire a 2nd independent reviewer? Won't you also need an independent reviewer for design reviews?

 

[Nichole F]

no device can have no essential performance. Loss of essential performance does not result in harm/injury, it results in loss of benefit. Loss of essential performance results in unacceptable risk since there is no longer benefit to outweigh the risks. Thus, all devices must have essential performance. If not meeting a performance requirement results in a harm, it is safety performance characteristic not an essential performance characteristic.

I'd be open to hearing arguments to the contrary.

Regarding your direct question, have the client hire a 2nd independent reviewer? Won't you also need an independent reviewer for design reviews?

Agreed. You will need a quantifiable clinical benefit to support your clinical evaluation. If your device is low risk, I doubt you will be conducting clinical investigations, but for MDR you still need to complete a clinical evaluation which requires clinical benefits and clinical claims.

As correctly pointed out, you need a competent independent reviewer for design control. That independent reviewer can also most likely verify the verification. Can the PRRC complete the verification? I'm assuming you are not the PRRC if you are freelancing.

 

[Peter Selvey]

Speaking as a designer in a one man business, the some practical approaches (may not be regulatory compliant): adding a forced delay between design work, testing and testing review as a proxy for a second set of eyes, say at least 2 weeks between each phase. Also writing up your logic and notes on decision helps to overcome any bias. In testing, always estimate the expected result, not just compare against the pass/fail criteria, and investigate differences even if the result is pass. For any mission critical issues (e.g. the main function of the device), figure out two or three independent ways to verify something. Testing is really only ever a spot check of good design (hardware or software), it's important not to read too much into positive test results. More than I care to admit, I've had a design mistake coupled with a test mistake that each cancel each other out, or two design mistakes that cancel each other out in a particular test condition, sounds bad but with a >1000 points to check in a design, it's going to happen from time to time, and especially coupled with a cognitive bias that assumes everything is OK. And if I'm brutally honest, the best "impartial" analysis really only occurs when something goes wrong and I don't know what it is. That causes me to pull apart the design in detail without any bias, or more to the point, a bias that suddenly flips and assumes everything is wrong. In those investigations, I find two or three weak points that have nothing to do with the issue that caused the trouble, but they are excellent opportunities to be honest about the design.

When standards refer to "validating" test set ups, this is really just an awareness that test set ups can often be ad hoc or have their own weak points, and it's worth to put through an intentional negative result or verification check of the set up itself. It depends on how important the test is. As mentioned above, for critical testing, alternate methods can be useful to increase confidence. However, it's case by case and not every test needs to be covered by a formal documented validation of the test method, it's not practical. For example, checking that the 5V rail is actually 5V.

I'd agree with above, the "no essential performance" is a slight of hand to keep things simple, but in truth any device that claims no essential performance is not a medical device. Generally, you expect "normal performance" under normal conditions, but may allow no performance for abnormal and fault conditions as appropriate, especially for a low risk device. For example, if your device is exposed to liquids in normal condition, then claiming "no essential performance" after a test simulating that exposure is flat out wrong, the device should continue to operate normally under normal condition. However if exposure to liquids is an abnormal condition, it's OK to say "no performance", only check for basic safety only after a test. And all tests for performance can be tailored for the test, you don't need to do a 100% essential performance assessment every time a test mentions essential performance, it can be anything ranging from no check (if it's clear the test could never affect performance), spot checks, inspection (e.g. water ingress), proxy tests, special software and so on. All common sense.

 

[mr9000]

I'd be open to hearing arguments to the contrary.

The risk of loss or degradation of the identified performance of the device is acceptable, therefore there is no essential performance as of 4.3 of 60601-1. I have met some products fulfilling that. Of course there is some kind of performance of the device, it's not a brick. It's just helping keeping the risk management file small.

Agreed. You will need a quantifiable clinical benefit to support your clinical evaluation. If your device is low risk, I doubt you will be conducting clinical investigations, but for MDR you still need to complete a clinical evaluation which requires clinical benefits and clinical claims.

There are clinical studies showing the benefit of the device. Do the studies for MDR need to be "new" using the new device or can one refer to older studies using the somewhat same device with similar performance parameter?


Yes, going the design control route and having a second person sign off everything would be the way it will go down, maybe I can get another somewhat technical person from production to also sign the documents. Since I know most of that will be just for the box ticking, I was wondering of reliable ways doing medical device development as a solo person. Throwing money at the problem always works (especially if it's not yours ;-) ), I was wondering if there are other ways to do it.

Speaking as a designer in a one man business, the some practical approaches (may not be regulatory compliant): adding a forced delay between design work, testing and testing review as a proxy for a second set of eyes, say at least 2 weeks between each phase. Also writing up your logic and notes on decision helps to overcome any bias. In testing, always estimate the expected result, not just compare against the pass/fail criteria, and investigate differences even if the result is pass. For any mission critical issues (e.g. the main function of the device), figure out two or three independent ways to verify something. Testing is really only ever a spot check of good design (hardware or software), it's important not to read too much into positive test results.

There will be a lot of completely different things to tackle during the project, a cool-down period between design, testing and (internal) review can surely be found. In my former job I often had to reject verification tests during my review since they were too shallow, did not test the right thing stated in the test description or sometimes didn't test anything at all. As you often say, using a lot of common sense is necessary. I'm also very confident using my engineering integrity that everything will be ok in the end and I will do the design and the verification to my best of knowledge and not using short cuts, but that is not proven unless objectively verified so here I'm back to my questions.

Thanks for your insights!

 

[Jean_B]

MDR doesn't define "essential performance". IEC 60601-1 does (for active devices), and clarifies stuff in its interpretation sheet 1 accessible here.
Iirc past ISO 20417 drafts talked of including it for non-active devices, but it's not present in the current version.

From the 60601-1 normal mode and single-fault mode must be considered, but it is still a risk assessment based aspect
A thing often forgotten is that dependent on the setting of use, another identical, similar, or alternative might be available. Thus essential performance reasoning in risk assessment should take into account how easy it is to obtain another identical, similar, or alternative item/treatment and the chance that that is not available (limited stock, e.g. trauma helicopter or intended for use in remote locations such as an oil drilling platform (who knows)).

Thus vitality-sustaining or -monitoring devices and life-saving devices almost by default are devices with essential performance as the chance of error and time-factor often precludes a decent chance to have the time needed to obtain identical, similar, or alternative item/treatments. (though what the essential performance is can take more work to define).
Then you have to look at whether the condition is deteriorating within a significant timespan, or stable. If intended for relatively stable conditions, then it can be argued to not have essential performance as the treatment timing takes place on an elective base, and not as an emergency matter.
etc.

Standardized risk management practices for medical devices occurs mostly at the level of a (singular) physical device, and not at the SKU (box of 10) or stock (a hospital has a decent chance of having more than one of the more generic kind of devices and not using all of them at once) or fleet (leasing constructions often temporarily replace a defective unit with a spare-pool unit) level. Let alone that before your device came they might have been using adequate, though not easy or comfortable, alternatives which might still be available.
A tongue depressor is a medical device; I do not regard that to have essential performance.



As for handling solo-work (V&V or otherwise) there's two aspects why you'd typically use more people: human error, and conflict of interest.
I'd have the legal manufacturer sign-off on at least the validation plans and the final validation report, with full insight in and access to all underlying/relevant documents, records (inc data), and other relevant materials (code etc.) planned and confirmed. They are responsible, and they should be the ones to call in a second opinion if they want to be (more) certain. You should make them capable of that, so they can execute their legal responsibility and cannot put it on you not providing them with the opportunity.

For human error, I would preferably draw on a second person. But you could minimize that aspect by implementing human-error reduction aspects in your own QMS:

• Enforced cooldowns between delivering work for review and performing review with recording time spent reviewing separately
• Annotated copies of work reviewed, with a default practice (i.e. a PDF copy of documents or code, with timecoded highlight marker of when you considered it reviewed). Or controlled copies with GDP notation for the same.
• Incorporate minuted "lay-man's explainer" sessions on work you finished. Even if the legal manufacturer is not and does not need to be an expert, they should have a "veteran operator's" working knowledge of the mode of operation and safety controls of their own device to assess vigilance issues. Make this work double-duty by having them as the application specialist (ISO 14971 requirement anyway) reviewing the response of your work item in certain situations.
• Demonstrably not exceed recommended/obligatory amounts of total work-time per day, and demonstrably taking recommend/obligatory breaks(/nourishment). Yes, this can break your 'flow-state' as a thought worker, but that is more a productivity efficiency thing than an effectiveness thing; safety trumps efficiency.
  • If needing to interrupt big chunks, mark interruptions and use the three steps back approach to double-cover those fade-out/fade-in aspects.
• Clear your schedule leading in to remove stray thoughts in working memory (perhaps include a physical work-out), have buffer-time at the end of scheduled reviews so you don't make time-pressured decisions.
• Make and use topic specific checklists to avoid working memory limitations and demonstrably cover non-linear review aspects such as interactions or holistic aspects.
• Change the 'view' between coding and reviewing. You even notice this for reviewers between digital and paper. They tend to find different and different amounts of faults in each, purely due to the change in interface during review and the ability to make notes. Regularly having pages next to each other during review (digitally: new window/split mode) already helps.
• DocControl more than only your final version. If you can show your drafts, with you answering your own questions you can evidence the development cycle it took and reasoning behind the final state.

Explain the above as part of your QMS for being a solo engineer. People can probably add more/their own practices but it's a free forum so this is a ¾-hour freebie.

 

[d_addams]

The risk of loss or degradation of the identified performance of the device is acceptable, therefore there is no essential performance as of 4.3 of 60601-1. I have met some products fulfilling that. Of course there is some kind of performance of the device, it's not a brick. It's just helping keeping the risk management file small.

this is a misinterpretation of the definition of essential performance and how essential performance is to be used in the design process. Again, devices cannot have no essential performance, there is an essential reason (the defined medical benefit as indicated in the intended use) the device is being used. The risk benefit in the error state cannot be positive or acceptable if not benefit can be derived in the error state. Defining essential performance intends to classify which behaviors are critical to deriving benefit and doesn't directly relate to the overall benefit/risk profile acceptability determination. Which is to say, when one says the risk is no longer acceptable if performance in this area is degraded (thus essential performance) one is not saying the overall risk/benefit for the product or therapy is unacceptable. One is just saying this characteristic is key to deriving medical benefit (fulfilling the intended use). The point of defining essential performance is to allow risk-based decision making, specifically identifying those areas where one should apply more rigorous evaluation of those characteristics compared to those areas that are not essential. Any individual feature may not be essential but there has to be at least one.

 

[Nichole F]

Do the studies for MDR need to be "new" using the new device or can one refer to older studies using the somewhat same device with similar performance parameter?

If there are new clinical studies, yes you should use them or you will need to justify why they were not used. You'll see "state of the art" mentioned in MDR, particularly in Chapter VI: Clinical Evaluation and Clinical Investigation and Annex I: GSPR. You need to use supportive data to show the benefit of the use of your device outweighs the risks. This claim needs to be supported by currently known information.

An over-simplified example: Take smoking for example. Back in the 1930s, tobacco companies advertised smoking as healthy. So for sake of example, lets say in the 1930s, the state of the art was smoking provided more benefit than risk. By today's knowledge, we know smoking is associated with lung cancer among other health risks. Using clinical evidence from today's studies, we know smoking is not health and the clinical risks far outweigh any benefits.

MDR requires the manufacturer to use state of the art studies because they don't want selectively including only data that supports the use of their device, especially when current knowledge may demonstrate otherwise. In the clinical evaluation, you are also required to include both favorable and unfavorable data (if available) . You also need to consider alternative treatment options in relation to your device. Be sure you are familiar with Article 61 and Annex XIV of MDR to ensure you have documented the information you will need for the clinical evaluation.

 

[Peter Selvey]

Years ago I did a thought experiment: if the on/off button of a digital thermometer doesn't work (the device won't turn on), is this a problem for "essential performance"? On the one hand, you could argue that there's no harm done. On the other hand, if the device doesn't turn on, you can't measure the patient's temperature which affect decision making, such as whether to take a child to hospital. On the third hand, society doesn't expect stuff to work all the time. And surely manufacturers can't reasonably be held responsible for a faulty switch indirectly causing harm because the temperature can't be measured? Finally, is it practical for safety test labs, EMC labs to be checking that level of functionality in tests that refer to "essential performance"?

But let's reframe the question: let's say a thermometer is subjected to storage and use testing in IEC 60601-1-11 (temp, press, RH, vibration, drop, IP testing etc.). Somewhere along the line, the device doesn't turn on. The manufacturer says it's a pass because it's not essential performance. Is this OK?

The answer to this puzzle is realising that there is a hidden assumption here: modern switches are normally reliable, so that a faulty switch is unlikely. So our instinct is not focus on that in EP related testing. And if the manufacturer is confident the switch failure was something unusual, offering data showing it's properly specified, showing in house test data or agreeing to repeat the test with several samples, it could be OK. Or if the test condition in the standard isn't really representative of normal use, more abnormal abuse. However, if it really is just a dodgy manufacturer using a cheap switch which is likely to fail often in real world normal use, it should be seen as a failure under essential performance regardless of what's written in the risk management file.

Jean mentioned that a tongue depressor should not have essential performance. In fact it does, but they are kind of obvious stuff similar to a on/off switch that you wouldn't expect a manufacturer to stuff up. They have to be suitable dimensions, no sharp corners or splinters, be able to take a degree of force with breaking when exposed to various temperatures, moisture etc expected under normal condition. Yeah, it's unlikely a manufacturer will stuff this up since it's important for sales, but still, it's not "EP free", it's more like "EP is obviously OK".

In reality essential performance should be a flexible concept, not a fixed definition, that takes into account the nature of the test itself, and the things that are checked should depend on the probability of the condition that test is intended to represent, and the probability the test condition will affect a particular functionality.

The bit where this gets all messed up is when standards start implying that test conditions are normal condition when they really are not, like 15kV ESD, or mandatory IPX2 tests on all home use equipment regardless of actual use. That forces manufacturers to claim "no EP" as a workaround.

 

[d_addams]

Years ago I did a thought experiment: if the on/off button of a digital thermometer doesn't work (the device won't turn on), is this a problem for "essential performance"? On the one hand, you could argue that there's no harm done. On the other hand, if the device doesn't turn on, you can't measure the patient's temperature which affect decision making, such as whether to take a child to hospital. On the third hand, society doesn't expect stuff to work all the time. And surely manufacturers can't reasonably be held responsible for a faulty switch indirectly causing harm because the temperature can't be measured? Finally, is it practical for safety test labs, EMC labs to be checking that level of functionality in tests that refer to "essential performance"?

But let's reframe the question: let's say a thermometer is subjected to storage and use testing in IEC 60601-1-11 (temp, press, RH, vibration, drop, IP testing etc.). Somewhere along the line, the device doesn't turn on. The manufacturer says it's a pass because it's not essential performance. Is this OK?

The answer to this puzzle is realising that there is a hidden assumption here: modern switches are normally reliable, so that a faulty switch is unlikely. So our instinct is not focus on that in EP related testing. And if the manufacturer is confident the switch failure was something unusual, offering data showing it's properly specified, showing in house test data or agreeing to repeat the test with several samples, it could be OK. Or if the test condition in the standard isn't really representative of normal use, more abnormal abuse. However, if it really is just a dodgy manufacturer using a cheap switch which is likely to fail often in real world normal use, it should be seen as a failure under essential performance regardless of what's written in the risk management file.

Jean mentioned that a tongue depressor should not have essential performance. In fact it does, but they are kind of obvious stuff similar to a on/off switch that you wouldn't expect a manufacturer to stuff up. They have to be suitable dimensions, no sharp corners or splinters, be able to take a degree of force with breaking when exposed to various temperatures, moisture etc expected under normal condition. Yeah, it's unlikely a manufacturer will stuff this up since it's important for sales, but still, it's not "EP free", it's more like "EP is obviously OK".

In reality essential performance should be a flexible concept, not a fixed definition, that takes into account the nature of the test itself, and the things that are checked should depend on the probability of the condition that test is intended to represent, and the probability the test condition will affect a particular functionality.

The bit where this gets all messed up is when standards start implying that test conditions are normal condition when they really are not, like 15kV ESD, or mandatory IPX2 tests on all home use equipment regardless of actual use. That forces manufacturers to claim "no EP" as a workaround.

your example of people justifying the failure because 'according to 60601 this isn't essential performance because the failure condition is acceptable thus not essential performance' is the exact thought experiment to demonstrate this is NOT how assigning a function as essential performance should be used. As you noted if people take this approach and are completing design verification they twist logic to now conclude that 10 of the 30 product we tested after environmental conditioning failed to report temperature. This is ok because this isn't essential performance and therefore not having 100% passing is acceptable. This is the exact extention of logic allowed when saying 'a product has no essential performance'.

If the product is supposed to do X to derive medical benefit (measuring the patients temperature), not measuring the temperature does not cause harm (the patients state is not changed by not measuring their temperature), but the product has now become useless and there is no benefit to placing it in their mouth, only risk. Exposing the patient to no benefit, only to risk is UNACCEPTABLE. This is not the same as saying the benefit/risk of the thermometer product is unacceptable. In this case the essential performance of the thermometer is to report the patients temperature. If the product can not do that, it only creates risk and there is no available benefit to offset the risk.