Search the Elsmar Cove!
**Search ALL of Elsmar.com** with DuckDuckGo including content not in the forum - Search results with No ads.

Risk Benefit Analysis - ISO 14971:2012 Requirements

#21
Time to jump start this conversation again... I recently went through an audit that resulted in these two risk related findings:

1. Risk Analysis is incomplete. Cannot asses if all 3 risk control options(D.P.I.) have been applied to each risk mitigation.
2. Risk Benefit Analysis was not applied for each risk individually.

1. a) My risk assessment has a checkbox for "Design, Process, and Information" for each risk. Some of the risks only have one of three, others two of three, and some have all checked. According to the auditor and their interpretation of 14971:2012 ZA #5 The manufacturer must apply all control options even if previous control options have reduced the risk to an
acceptable level. Why would you apply "labeling" for something if you've already designed it out and it no longer needs labeling as a mitigation? You wouldn't! So how can I check that box if I didn't apply labeling? I can't. So what do I do? Provide a justification each time I apply less than all three options? Can anyone show me how you are meeting this requirement and what evidence you are providing to auditors because i'm at a loss as to how i can meet this requirement.
:mg:
Your auditor is going overboard in my opinion. It sounds like you've documented the risk management process quite well...

Presumably what they are wanting to see is that you've reduced risk "as far as possible" - but to assume this means that ALL possible risk controls must be applied is absurd! (as you've pointed out)

Perhaps you could point to a risk management system procedure, or plan where it's stated that you will apply risk-controls until risk is reduces as far as possible? A documented explanation in the procedure/plan (e.g. "application of controls stops once residual risk is evaluated to be as low as possible") may suffice to explain what you're doing, and justify why not all options are applied in all cases...? :confused:

2. a) How is everyone applying RB to individual items? My only thoughts mirror what Yodon mentioned earlier in the post, to just do some handwaving by adding a column with a canned statement for each item that RB has been assessed by the members of the cross-functional risk analysis team and determined that the benefits outweigh the risks. Pretty silly since the overarching RB assessment/conclusion already accomplishes this. Or do i need to provide a justification of how the team came to the conclusion specific to each line item?
Agreed. I don't think it is necessary to provide a justification in every case (it'd probably just be a lot of copying and pasting essentially the same statement...).

Again, this may be overcome by documenting in a procedure/plan that "each identified risk is assessed for cost/benefit acceptability", and then simply include a checkbox on the documentation to show that you've done it.

It sounds like you have an overly-zealous auditor that wants to see some very specific format (even if it has questionable value). I guess we were lucky. We just grouped hazards into categories (e.g. materials, power, software...), and then included a short paragraph following the analyses in each category to the effect of "for each individual hazard in this category it has been assessed that the benefits to the patient outweigh the residual risks of harm...".
 
Last edited:

yodon

Staff member
Super Moderator
#22
According to the auditor and their interpretation of 14971:2012 ZA #5 The manufacturer must apply all control options even if previous control options have reduced the risk to an acceptable level.
The statement is kind of correct - you're expected to continue to apply controls to reduce to the greatest extent possible; i.e., not just drop down to an acceptable level.

But that certainly doesn't mean there should be 1 or more of each.
 
#23
The whole concept of risk/benefit is fundamentally flawed.

In the beginning, many regulators referred to "balancing risk and benefit", and this is still a popular phrase in literature and can still be found in the non-normative areas of ISO 14971. But this is illogical: there is no reason why we should neatly balance risk and benefit. It's actually quite stupid if you think about it.

Realising this, regulators moved to language to "benefits outweigh the risk", as used in ISO 14971 normative section. But again, this is equally stupid, as "outweigh" could mean that benefits that marginally outweigh the risks are acceptable.

In Europe they have realised "balanced" and "outweigh" are logically wrong. MEDDEV 2.7.1 (2016) uses an "acceptable risk/benefit profile". The MDD itself also refers to "acceptable risk/benefit ratio" and "acceptable risks when weighed against the benefits". While these are logically correct, they fail to provide an effective criteria or basis for making practical decisions, i.e. what is an acceptable profile or ratio.

In general, risk/benefit is useless for determining if something is OK or not. It is an extremely rare case where the risks and benefit are so closely matched that the amount of benefit influences the decision. Usually, residual risks are a small fraction of the benefits, which makes benefits irrelevant to the discussion.

Instead, the key decision point is the practicality of further risk reduction. Logic says we want to maximise benefit and minimise risk as far as reasonable, that is, within sensible limits of technology, cost and practicality.

Clause 6.5 in ISO 14971 does say risk/benefit can only be used if "further risk control is not practicable", but this fails to be effective as there is no requirement to keep any records. And it is an important point, as manufacturers often pull out the risk/benefit card when challenged, without discussing what solutions existed and why they were not implemented.

An example helps, which is based on real life: it turns out that 6% of hearing aid users actually get further hearing loss (harm) from the hearing aid itself.

But, manufacturers argue that the benefit outweighs the risk. Research shows that if the hearing aid is not used, the brain allocates hearing related sections to other tasks, meaning the patient will lose their hearing altogether without the hearing aid.

It is clear then that the benefits far outweigh the risks - but is this really OK?

No, at least not from perspective of common sense.

The correct question is to ask if there any reasonable cost, practical solutions that could reduce 6% further?

Maybe a simple software solution exists that monitors and limits the cumulative daily sound exposure, or limits the number of peak output events per day, or catches feedback ringing quicker.

If such a solution did exist, and this software reduced the incidence from 6% to 1%, and only increased the cost by an extra $0.50/hearing aid, surely logic would dictate that the solution should be used?

Clearly, using risk/benefit, whether balanced, outweigh, ratio or profile is the wrong criteria here.

A logical risk analysis would say: If the residual risk is significant, and further risk controls are not taken, analyse and record:
- what other solutions could have been used
- why those solutions were not considered practical
- why the residual risk is acceptable (yes, OK, this point might refer to the benefit...)
 
Last edited:
#24
...Instead, the key decision point is the practicality of further risk reduction. Logic says we want to maximise benefit and minimise risk as far as reasonable, that is, within sensible limits of technology, cost and practicality.
Agree 100%.
However, one of the problems is the explicit change of language in the Z Annex of EN ISO 14971:2012, which changes "as low as reasonably practicable" (similar to wording that you are - reasonably - presenting here), to "as far as possible".

...Clearly, using risk/benefit, whether balanced, outweigh, ratio or profile is the wrong criteria here.
A logical risk analysis would say: If the residual risk is significant, and further risk controls are not taken, analyse and record:
- what other solutions could have been used
- why those solutions were not considered practical
- why the residual risk is acceptable (yes, OK, this point might refer to the benefit...)
Again, I agree. But key here is your qualification "if the residual risk is significant". Under the EN ISO framework, this qualification doesn't seem to matter ("the manufacturer must undertake the risk-benefit analysis for the individual risk...in all cases"). Justifying "why those [other] solutions were not considered practical" is a tough case to make.

Take the example you've given recently in this thread, regarding the burn-warning label on a thermal printer where, like you say, any reasonable person would not maintain contact to the point of burns. How confident would you be that you could justify NOT applying the label under a "as far as possible" risk reduction framework?

Also, in principle (ISO 14971 requirements aside), I don't think justifying residual risk acceptability (by way of risk/benefit) should be necessary for each risk unless, as you say, the residual risk is significant.

This is because the justification is baked into the the whole risk management framework.

For example, in a quantitative system you might have established that risk is acceptable if the residual risk priority number is less than X (and this number is calculated from probability and severity). In other words, given a RRPN less than X, either:
a) the potential severity is acceptably low, so as to outweigh any probability of occurrence;
b) the probability is acceptably low, so as to outweigh whatever severity may manifest; or
c) the balance between probability and severity is deemed acceptable.

That is the justification. It is implicit in the calculation and acceptance of a residual risk priority value.

The idea that there is any added value - as is the original topic of this thread - in documenting risk-by-risk justification of acceptability (by way of risk/benefit) seems illogical to me...
 
#25
Yes, apologies - the original thread is more about the Annex Z wording than the broader context of risk / benefit.

In my opinion the whole model of acceptable risk (i.e. measure the risk, compare against limit, if below do nothing, if above take action or use risk/benefit) is flawed, so we are kind of floundering around trying to patch up issues around the edges when what is really needed is a new core.

But to defend Annex Z a little:
In the other thread I mentioned 3 types of "information for safety": obvious, lazy, and genuine. There is a 4th - irresponsible - where the risk is significant and but nothing is done except a warning, when there were other options on the table. That happens more often than you think, and seems to be the obvious trigger for Annex Z.

For example: digital thermometers take around 10 minutes to stabilise. We don't want to wait 10 minutes, so manufacturers use a predictive method: good devices watch for 30-60s and then use a software algorithm to estimate the final temperature. Then some manufacturers realised if they put "measures in just 10s" on the box, then they can sell more.

Problem is ... 10s only works under highly controlled conditions, which are rarely found out in the real world. High errors occur in practice which then influence clinical decisions. For example if your kid's real temp is 41.2°C you should go to the emergency dept., but if the 10s thermometer reads 39.7°C you might sweat it out at home. The kid dies at 3am. Because of a 10s thermometer. And you can buy these from your local pharmacy. A regulated medical device.

The manufacturers of the 10s products probably know that there are high errors in practice, but hide behind a warning which is documented as being acceptable risk.

Frustrated with situations like this, Europe pumped out Annex Z. The gist of Annex Z is to say: we don't care what you wrote in the risk management file, a 10s thermometer does not meet the requirements of the directive, get it off the market.

I agree that the wording in Annex Z seems to have overstepped the mark, especially the logic to say that risk/benefit is always required for individual risks. But again, a couple of key points:

First, it's an informative annex. Informative annexes are not mandatory or enforceable, the are not part legal framework which gives a presumption of conformity (harmonised standards)

Second: it does not indicate what records are required, which is the core point of this thread. ISO 14971 has several requirements where no records are required, and that's an important distinction. One example already mentioned is the trigger for using Clause 6.5, when solutions were "not practicable" - no records are required with respect to what solutions were considered and rejected or why. Annex Z is pointing out that you always need to keep the risk/benefit profile in mind, but there is no requirement to keep any records.

So, Annex Z does not change what you document in the risk management file. It's really a legal point that had to be made - without Annex Z, Europe appeared to be legally bound to accept the decisions in the manufacturer's risk management file, under Article 5, Section 1 of the MDD (presumption of conformity). Annex Z clarifies that EN ISO 14971 is only a partial standard, and compliance with EN ISO 14971 does not infer a legal presumption of conformity where Annex I refers to risk.
 
#26
Time to jump start this conversation again... I recently went through an audit that resulted in these two risk related findings:

1. Risk Analysis is incomplete. Cannot asses if all 3 risk control options(D.P.I.) have been applied to each risk mitigation.
2. Risk Benefit Analysis was not applied for each risk individually.

1. a) My risk assessment has a checkbox for "Design, Process, and Information" for each risk. Some of the risks only have one of three, others two of three, and some have all checked. According to the auditor and their interpretation of 14971:2012 ZA #5 The manufacturer must apply all control options even if previous control options have reduced the risk to an
acceptable level. Why would you apply "labeling" for something if you've already designed it out and it no longer needs labeling as a mitigation? You wouldn't! So how can I check that box if I didn't apply labeling? I can't. So what do I do? Provide a justification each time I apply less than all three options? Can anyone show me how you are meeting this requirement and what evidence you are providing to auditors because i'm at a loss as to how i can meet this requirement.

2. a) How is everyone applying RB to individual items? My only thoughts mirror what Yodon mentioned earlier in the post, to just do some handwaving by adding a column with a canned statement for each item that RB has been assessed by the members of the cross-functional risk analysis team and determined that the benefits outweigh the risks. Pretty silly since the overarching RB assessment/conclusion already accomplishes this. Or do i need to provide a justification of how the team came to the conclusion specific to each line item?
Hello,

I'm curious about how you managed to resolve this issue with your auditor.

In our case, the auditor also wants to see that we are following the standard word by word, particularly in applying all options for controlling risks (inherent safety, protective measures and information for safety). The problem we are facing is that some risks cannot be reduced by 'inherent safety', so we apply protective measures and information for safety and we are providing a risk/benefit rationale for every single risk.

I started this question this thread:

Hello Marcelo,

Thank you for the incredibly quick response to my question.

During our engineering discussions we have not (yet) found a way to mitigate this risk through ‘inherent safety’ and we are afraid that we won’t be able to do so in the near future. This leaves us with only two alternatives for mitigation, which of course are ‘protective measurements’ and ‘information for safety’.

However, failing to provide ‘inherent safety’ seems contrary to Section 2 of Annex 1 of Directive 93/42/EEC (please let’s forget about the oncoming MDR for a moment). As far as I understand, we must apply the ALL control options ‘cumulatively’ until additional endeavours do not improve safety.

In this regard, I am not quite sure if we can presume compliance by mitigating this risk only through ‘protective measurements’ and ‘information for safety’ and a good risk/benefit rationale.

By the way, it seems to me that this risk is quite common to all diagnostic devices that need to be in ‘good contact’ with the body of the patient. In my experience all of them use similar approaches to deal with this risk. I cannot identify any ‘inherent safe’ mitigation in these devices.
Cheers!
 

Marcelo Antunes

Addicted to standards
Staff member
Admin
#27
In our case, the auditor also wants to see that we are following the standard word by word, particularly in applying all options for controlling risks (inherent safety, protective measures and information for safety).
Please note that standards are voluntary. Following any standard word by word is not a requirement (outside the Brazilian certification process, some Chinese requirements and some from Korea).

Anyway, ISO 14971 does not require that you apply all 3 risk control option for everything, only that you evaluate in the priority order and apply which is feasible.
 
#28
Annex ZA I think is now more widely understood to be an informative Annex that does not have any requirements, and should not be used by auditors as a requirement. I recently planned to write an article on this but I found that a few Notified bodies had come to the same conclusion in published on-line material, so it should be easier to argue against any auditors that are treating it as "normative".

The key point is that ISO 14971 allows the manufacturer to set their own criteria for acceptable risk, while the directive states that risks have to be minimized. These are different criteria. So if you document a solution as having acceptable risk with perfect EN ISO 14971 records, there is no legal obligation for Europe to accept this as OK even though EN ISO 14971 is harmonized.

Annex ZA is just a bit of a long rambling justification for that position.

So again using the 10s electronic thermometer as an example: I know from experience these thermometers are hopelessly inaccurate in the hands of real people. But a clever manufacturer could write up everything as OK in a risk management file. The point is that an auditor can still ask: in light of the real world situation, have the risks been reasonably minimized? The obvious answer is no. But the auditor should not be quoting Annex ZA, but rather dealing with the individual issue, and only referring to Annex ZA if the manufacturer insists on using risk management to justify an unreasonable position.
 
Last edited:
Top Bottom