Relative vs absolute severity

[MrTetris]

Dear Community, I work for a medical company that produces many different medical devices from class I to class III. Some of them are linkable to high severity risks (failure can conduct to the death of the patient), others can be related to only very low severity risks (minor discomfort or delay).
We used to use the same severity table for all products, from 1= discomfort, delay to 5=death, which means that for some products, Severity 5 was never reached in the risk analysis. I always thought that this was a good approach, also because it allowed us to compare the severity of different products, but I recently received this comment from an auditor:
"You cannot use a single severity table for all your RMCs, you have to define a severity harm table for each of your devices, where 5 is always the highest possible severity for the specific device (so 5 could be "death", but also "discomfort", depending on the device)".
I tried to argue with the auditor, as I deeply disagree with the comment, and in the end it stayed as a simple "observation". What do you think about it? I am sure that the auditor will get back to it next year...

 

[Bev D]

Your auditor is an incompetent <many expletives deleted> person.

1). The entire idea of ratings is that there is a single scale. A relative scale only introduces confusion and misinformation. It actually treats less harmful products with more attention than more harmful products. THINK about it.

2). If mathematics on ordinal data (RPN or S*O calculations) is mathematically unsound then introducing a random scale value is even more scientifically unsound.

3) WHERE in the standard - any standard - does it say that a scale is to be relative? No where. NO WHERE. Because of 1 and 2 above…This why many industries publish scales to be used by their suppliers.

4). Report the auditor and get a new one. Don’t let this idiotic science denier back into your facility.

 

[MrTetris]

Your auditor is an incompetent <many expletives deleted> person.

1). The entire idea of ratings is that there is a single scale. A relative scale only introduces confusion and misinformation. It actually treats less harmful products with more attention than more harmful products. THINK about it.

2). If mathematics on ordinal data (RPN or S*O calculations) is mathematically unsound then introducing a random scale value is even more scientifically unsound.

3) WHERE in the standard - any standard - does it say that a scale is to be relative? No where. NO WHERE. Because of 1 and 2 above…This why many industries publish scales to be used by their suppliers.

4). Report the auditor and get a new one. Don’t let this idiotic science denier back into your facility.

I agree with 1, 2, 3. I will wait before proceeding with the 4, for now.

 

[Tidge]

"You cannot use a single severity table for all your RMCs, you have to define a severity harm table for each of your devices, where 5 is always the highest possible severity for the specific device (so 5 could be "death", but also "discomfort", depending on the device)".
I tried to argue with the auditor, as I deeply disagree with the comment, and in the end it stayed as a simple "observation". What do you think about it? I am sure that the auditor will get back to it next year...

I think the auditor has confused (at least) two different concepts in medical device risk management.

It is true that for each (family of) medical device, there is supposed(*1) to be an a priori individual assessment of overall acceptability for the (family of) medical device. This has often been represented as a table (an N x N matrix) of two ordinally-scaled rankings: Severity is one axis, the other is the "P1xP2" axis. The general idea is that different families of devices (with different indications for use) should have a priori different levels of risk acceptability. That is: a priori, we better tolerate the outcomes for patient injuries when the medical device is an AED than when the device is toothbrush.

Severities of harms should absolutely have the same relative ranking independent of the medical device. If they don't have a common ranking between devices, then there would be the absurd condition that (for example) a medical device that causes blindness would have the blindness caused to children be considered a different severity than blindness caused to adults (assuming the situation that one device has pediatric indications and the other does not). The medical device industry's practice is for each manufacturer of medical devices to maintain a "Master Harms List" that ranks the harms recognized by that manufacturer, and ranks them (typically with an ordinal scale). The Master Harms List is best attested to by a clinical person with certified medical training (i.e., not an engineer or salesperson).

(*1) This a priori assessment (found in the Risk Management Plan for a medical device) may be somewhat downplayed in an era of "reduce all risks as much as possible"-thinking. It is still valuable (and necessary!) because it is to be used in the final assessment of overall risk acceptability of the device, for its intended use, as well as the final benefit-to-risk assessment of the device.

 

[MrTetris]

I think the auditor has confused (at least) two different concepts in medical device risk management.

It is true that for each (family of) medical device, there is supposed(*1) to be an a priori individual assessment of overall acceptability for the (family of) medical device. This has often been represented as a table (an N x N matrix) of two ordinally-scaled rankings: Severity is one axis, the other is the "P1xP2" axis. The general idea is that different families of devices (with different indications for use) should have a priori different levels of risk acceptability. That is: a priori, we better tolerate the outcomes for patient injuries when the medical device is an AED than when the device is toothbrush.

Severities of harms should absolutely have the same relative ranking independent of the medical device. If they don't have a common ranking between devices, then there would be the absurd condition that (for example) a medical device that causes blindness would have the blindness caused to children be considered a different severity than blindness caused to adults (assuming the situation that one device has pediatric indications and the other does not). The medical device industry's practice is for each manufacturer of medical devices to maintain a "Master Harms List" that ranks the harms recognized by that manufacturer, and ranks them (typically with an ordinal scale). The Master Harms List is best attested to by a clinical person with certified medical training (i.e., not an engineer or salesperson).

(*1) This a priori assessment (found in the Risk Management Plan for a medical device) may be somewhat downplayed in an era of "reduce all risks as much as possible"-thinking. It is still valuable (and necessary!) because it is to be used in the final assessment of overall risk acceptability of the device, for its intended use, as well as the final benefit-to-risk assessment of the device.

Thank you for your contribution. I agree with your post, but we DO have an a priori individual assessment of overall acceptability for the (family of) medical device, based on risk severity. We do accept higher RPN numbers (and therefore higher probability of occurrence) for risks with lower severity, and vice-versa, so that is not the point... But thanks! That is another important point, which we are actually meeting.

 

[Ed Panek]

I dont agree. If that were the case an invasive medical device to treat a pulmonary embolism would have similar risk profile and controls to a device of a tongue depressor.

In fact, any regulatory body will first determine the risk classification of the proposed device. FDA, EU, etc all do this. The controls depend on that risk.

 

[Tidge]

We do accept higher RPN numbers (and therefore higher probability of occurrence) for risks with lower severity, and vice-versa, so that is not the point... But thanks! That is another important point, which we are actually meeting.

I'm having trouble parsing the part I bolded. The tenets of medical device RM generally lead to a situation where if a device is almost always causing (i.e. "High Occurrence") some level of low-level injury (i.e. "low severity") is "just as" UNacceptable as rare (i.e. "Low Occurrence") serious injury (i.e. High Severity").

The a priori table I mentioned is often leveraged (very early in the development/planning phase of a product life-cycle) to (somewhat) explicitly recognize that if a manufacturer is planning on entering a mature market(*1) with a device that offers relatively little medical benefit(*2) when used... we (the market, the industry) don't want that device generating a whole lot of low-severity injuries.

(*1) A mature market is one in which the underlying means that the device uses to meets its intended use is well studied/understood, and there exist devices to do what this new device does.

(*2) I'm not trying to step on anyone's marketing hype. If anyone wants to think their tongue depressor is a class III device that is radically improving the quality of life for users and patients... take that up with the regulators.

An example of a "low benefit" device that would be UNacceptable should they always cause low-level injuries would be the tongue depressors mentioned by @Ed Panek ; If 10% of users suffered a cut "Blood loss, minor" when using a manufacturer's tongue depressors... that should be recognized as UNacceptable by the manufacturer. Contrast this with a device & treatment that may be considered radical, but there is no alternative... such as ECMO for treating serious covid patients. The ECMO injuries from that sort of treatment are incredibly serious, and somewhat common, but more acceptable given that there is no alternate treatment and the alternative of not even trying the treatment is death.

 

[ThatSinc]

I don't think there's much more to add to the responses already made - I've worked with companies that have had severity rankings relative to the devices they manufacture, but the rankings were consistent within the company. i.e. the top severity was not death, but they had no devices that were assessed as being able to cause death.

Once the NB reviewed the scale they had a perplexed face but ultimately nowhere in the standard does it prescribe how you set it up.

I typically use a single matrix for Severity and Probability, however what is considered as acceptable within the matrix differs between devices and is defined in the plan - as per the policy on risk acceptability and preliminary clinical evaluation of the device type for establishing state of the art.

 

[Bev D]

An example of a "low benefit" device that would be UNacceptable should they always cause low-level injuries would be the tongue depressors mentioned by @Ed Panek ; If 10% of users suffered a cut "Blood loss, minor" when using a manufacturer's tongue depressors... that should be recognized as UNacceptable by the manufacturer. Contrast this with a device & treatment that may be considered radical, but there is no alternative... such as ECMO for treating serious covid patients. The ECMO injuries from that sort of treatment are incredibly serious, and somewhat common, but more acceptable given that there is no alternate treatment and the alternative of not even trying the treatment is death.

This is a very cogent example of why FMEA and hazard analysis should avoid any RPN or SO prioritization structure. Science, knowledge and logic are much better at prioritization of “risk mitigation” than any fake math. As a profession we need to stop enabling simplistic mathematical algorithms as a substitute for thinking.

 

[d_addams]

yes the reviewer is way off base. No scale can be appropriate or unappropriate based on how many items do/don't fall into any categories. As others have noted many companies with diversified products use a single categorization scheme even though the potential harms in products across their portfolio may not use all parts of the scale.

If a reviewer was going to question execution it could be on the use of the same acceptance criteria for each therapy. It is the risk acceptability that will change, not the categorization scale. I think part of this reviewer's confusion may come from some scales using 'judging' words in their scale descriptions (such as 'red' or the top scale risks being labeled 'unacceptable' and green risks being labeled 'acceptable'). Thus if only 'death' can result in an 'unacceptable' risk and you're product can't cause death, then no risks could be considered 'unacceptable'. So do yourself a favor and get rid of these type of judging words in the categorization descriptions if you are using them.

If you're not selling in the EU go ahead and set your a priori acceptability criteria based on risk classification (which 14971 allows), but if you intend to get CE mark, heed the content deviations and ensure all risks are reduced as far as possible before applying any risk acceptability criteria.