ISO Registrars/CBs citing for use of "Detection" in Risk Management

M

MIREGMGR

#21
Just curious, have you evaluated the devices human factors/usability to make sure that the devices does not induce the user to make a mistake?
Yes.

As a clarification, error rates with these products are very low and of low severity at worst. By and large, these are low risk products, even though some are US FDA Class II for legacy reasons.

However, as noted, we've considered substantially different design/ materials/ manufacturing approaches in the past. Customers have told us very clearly that given a choice between the traditional, industry-standard, low cost design and a new, more complicated design approach that would seek to eliminate the possibility of use error at a greater cost, uniformly they would buy the traditional, industry-standard, low cost design no matter what benefits were offered by the alternate product, because they don't regard the user error potential to be significantly associated with patient risk and thus won't accept any increase in cost.
 
Elsmar Forum Sponsor

Peter Selvey

Staff member
Super Moderator
#22
This is a long thread so not sure if this is already covered:

Detectability of an adverse situation in time to prevent harm is a valid consideration in risk estimation, but it should just be another factor in the overall probability estimation (as Sam L. pointed out).

I think the problem was that some manufacturers were adding a third parameter specifically for detection, and the impact of this quantity on risk estimation was far too high. A typical case might be where detection was rated from 1-6, and this factor was directly multiplied in the risk estimation. A certain notified body (who shall remain nameless) actually published a guidance around 15 years ago with such a scheme. They since removed it.

Normally, probability in risk is represented in logarithmic steps (e.g. 1, 0.1, 0.01, 0.001 etc events / year / device). In such a scheme, a risk control needs to be around 90% effective to change the risk by one step, 99% to change two steps, 99.9% to change 3 steps and so on.

Detection by the user is typically much less than 90%, and almost impossible to get 99% effective. A warning in a manual is typically only 5-20% effective. A value of 90% is usually only achieved if it is well known standard practice, or if special qualification / training is involved.

But such situations are rare and even then at best only one step improvement could be claimed.

Thus, typically detection by the user should not influence the decision by much. I guess that it what regulators / third parties might be worried about.
 
R

revolution_2006

#23
I happened upon this thread and although there have been multiple items discussed, specifically with regards to calculation of risk, the relevant parameters and the definition of detection, one element seems to be missing, so although this is late post, I hope it provides some value.

As I also work with Class II and III devices and utilize detectability in our FMEA, if challenged, as the OP was, I would relay on the following:
14971 G.4 - Failure Mode and Effects Analysis: does mention detectability (seems to relate to probability, as noted by others in this thread) however, it also references IEC 60812 for more information on the procedures for FMEA.

IEC 60812 - This standard clearly indicates (search for "detect") in multiple places that detectability can be considered in a risk analysis for both designs and processes. More specifically, section 5.3.2 provides a three parameter formula to determine the risk priority number as RPN=S x O x D.

As 14971 provides direction on the conduct of an FMEA, which further directs the user to 60812 for "more information on the procedures for FMEA", and said procedure provides a example in which detectability is one variable in a three parameter equation to determine risk, it would seem that the Notified Body errored in initial citation of a NC (although it sounds like they changed thier minds after further discussions). Just my :2cents:
 

Peter Selvey

Staff member
Super Moderator
#24
The use multiplying for risk estimation e.g. R = S x P x D should be debunked once and for all.

If real numbers are used (e.g. money to represent severity, actual probability of harm), then it is correct to multiply probability and severity to estimate risk.

However, most manufacturers use a roughly logarithmic scheme (e.g. 1-4 for severity, 1-6 for probability). These schemes are not numerically grounded, for example, a severity of "4" is not four times more severe than a severity of "1". The numbers are just a label for each level. Thus performing any mathematical operation on the "numbers" is a joke. Really.

Even if the numbers are carefully selected to be strictly logarithmic (e.g. S = 1 for $1, 2 for $100, 3 for $10,000, 4 for $1,000,000, P = 1 for 10-6, 2 for 10-5 etc), the mathematically correct approach would be to add the numbers, not multiply. Thus again making the above multiplication a real joke. A lawyer's picnic if it ever got into court.

Detection makes the situation even worse. Again if real numbers are used it's OK, detection is another factor in the probability. If we want to use the formula R = S x P x D, the parameter D should represent probability that the detection does not work. If real numbers are used, it quickly becomes apparent that the detection effectiveness needs to be very high (90% or more, so D = 0.1 or less) to have any significant reduction in risk. In a log scheme, D must follow the same log base as P (e.g +1 for each power of 10). If so, then the addition of R = S + P + D is correct.
Otherwise, any mathematical operation in meaningless.

Adding an arbitrary factor D which is then multiplied to determine the risk ala R = S x P x D, is beyond a joke, it should be "go to jail, do not pass go, do not collect $200".

Of course, the real story is that regardless of the scheme (even using real numbers), risk is almost always impossible to estimate. So the numbers are all fudged anyway, usually adjusted to support the status quo. So it doesn't really matter ...

My :2cents:
 

Peter Selvey

Staff member
Super Moderator
#26
For probability, a popular scheme uses the numbers 1-6 to represent from impossible (P ~0.000001) to frequent (P~1). The factor, or ratio of probabilities between different steps is roughly 10.

No matter what the scheme is, there is always a factor of at least 10 between steps or levels. Any smaller factor would need too many steps to cover the large range of probabilities which risk management must consider.

If the probability of harm is reduced through a risk control, the residual risk depends on the effectiveness of the risk control, which can be expressed as a probability E. The residual risk occurs when the risk control fails, or is not effective, i.e. a probability of (1-E).

The new probability with the risk control in place is then:

P(new) = P(old) x (1 - E)

If the minimum factor between probability steps is 10, then E has to be >0.9 (90%) in order to change one step in the scheme. If the scheme is strictly logarithmic, with a factor of 10 between each step, then an effectiveness of:

E = 90% = 1 step
E = 99% = 2 steps
E = 99.9%= 3 steps and so on

I find it crazy how often a risk management file includes a hazardous situation where a warning in the manual changes the probability by 2 or more steps or levels. That implies that the warning is 99% effective!
 

Ronen E

Problem Solver
Staff member
Moderator
#28
A very interesting thread - thanks to all contributors.

A few comments:

1) The way I see it, ISO 14971 doesn't promote "detection" at the conceptual level (though, as noted by others, it indirectly relates to it through the FMEA technique), but it also doesn't ban it. As long as all other requirements are met, I don't see a reason to cite a manufacturer for including detection in the process.

2) I would differentiate detection in the production line (especially automated detection, such as machine vision and other testing stations) from detection by the end user (or through the supply chain). These are, IMHO, 2 very different topics, each with their own set of concepts, tools and limitations. Discussing them together as if they were one only adds to the blur.

3) ISO 14971 may be more prescriptive up to the point where risk level is determined, but it leaves quite a lot of room when it comes to determining risk acceptability. It requires that the organisation has a policy for setting acceptability criteria (but doesn't say what it should be or how should it be set), and that such criteria (compliant with the set policy) are laid down at the beginning of the risk management round. Then, the established (or "established"...? :)) risk should be judged by the acceptability criteria. So, if an organisation orderly decides and documents that its acceptability criteria include, in some way, some detectability aspects, I don't see how that would go against the standard. Once the risk level has been designated, that risk's detectability level - among other factors - could influence its acceptability (which is a bit like what MIREGMGR described, I think). In such a system, improving detectability might improve acceptability, even if it doesn't lower the risk level, per se. How much? I do agree with Peter S. that simplistic numeric calculations are too easy to manipulate, and thus caution should be exercised; on the other hand, I don't think detectability should be completely abandoned (or is negligible) as a valid mitigation means.
 
Last edited:
Thread starter Similar threads Forum Replies Date
P List of ISO certificate registrars around the world - ISO certification databases ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
M Canada - Registrars that allow e-auditing for ISO 9001? Registrars and Notified Bodies 4
L Are ISO/TS Registrars also controlled or governed by ISO17021:2011? IATF 16949 - Automotive Quality Systems Standard 3
G Are there any ISO registrars that should not be selected and why? Registrars and Notified Bodies 9
J Registrars Doing Synchronized (Combined) Audits to AS9100 and ISO 13485 Registrars and Notified Bodies 8
C Changing ISO 9001 registrars (second triennial)? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
C Notified Body References and Reviews - Changing ISO 13485 Registrars EU Medical Device Regulations 8
T ISO 22000 Registrars in US Registrars and Notified Bodies 11
N Reputations of ISO 9001 Certifying Bodies (Registrars) - SGS, AJA, BVQI, SAI Global Registrars and Notified Bodies 5
X Where to start with ISO 13485 and Registrars near Seattle ISO 13485:2016 - Medical Device Quality Management Systems 2
Q Legitimacy of ISO 9001:2000 Registrars and Certificates Registrars and Notified Bodies 16
WCHorn ISO/IEC 17025 Registrars - I'm confused about "accredited" registrars for ISO 17025 ISO 17025 related Discussions 18
P Can I have 2 certificates under 2 different registrars for ISO 9001:2000? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 8
RoxaneB ISO 9001:1994 Certificates of Compliance - Are any registrars extending these? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 1
C Required Audit Days - ISO/IEC Guide 62 gives guidance to Registrars ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 15
H ISO 9001:2000 Registration Audit Findings by Registrars Registrars and Notified Bodies 30
A iso 17025 registrars ISO 17025 related Discussions 5
Marc ISO Guide 62 contains the mandays guideline used by registrars when auditing ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 0
S Need ISO 15189:2012 Documentation toolkit. Document Control Systems, Procedures, Forms and Templates 0
chris1price Archiving of paper records - ISO 9001 7.5.3.1b Records and Data - Quality, Legal and Other Evidence 4
M Transferring ISO 17025 from one company to another ISO 17025 related Discussions 1
D Common practices in ISO 9001 deployment ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 17
Q ISO 9001-2015 Internal audit finding Internal Auditing 12
B ISO 17025:2017 risk management Risk Management Principles and Generic Guidelines 0
P Audit check for IT company (ISO 9001) ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 7
M Label Making & Printing Standards ISO / ASTM ISO 13485:2016 - Medical Device Quality Management Systems 5
Sidney Vianna Interesting Discussion Should ISO 9004 be changed from a guidance standard to a requirements standard? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
Ed Panek ISO 13485:2016 Section 5.5.3 ISO 13485:2016 - Medical Device Quality Management Systems 3
Q Do these certificates of calibration meet ISO 9001 requirements for traceability to NIST? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 11
ebrahim QMS as per ISO 13485, Clause 4.2 Requirements for regulatory purposes for Medical Devices Authorized Representatives. ISO 13485:2016 - Medical Device Quality Management Systems 3
S ISO 2768-mk print call out Other ISO and International Standards and European Regulations 11
T ISO 17024, clauses 4.3.8. and 5.1.1. Other ISO and International Standards and European Regulations 4
C ISO 14001:2015 6.1.3 Compliance Obligations - Legal requirements monitoring ISO 14001:2015 Specific Discussions 0
C Requirement to link Quality Manual to ISO 9001 clause numbers? ISO 13485:2016 - Medical Device Quality Management Systems 13
D ISO 13485 scope (implantable) - Polymers for dental application EU Medical Device Regulations 9
W First time being audited (ISO 9001), asking for advice ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
John C. Abnet ISO 26262 ISO 26262 - Road vehicles – Functional safety 3
Marc ISO 26262- Road vehicles – Functional safety ISO 26262 - Road vehicles – Functional safety 0
John C. Abnet ISO 26262 IATF 16949 - Automotive Quality Systems Standard 0
A ISO/DIS 15223-1:2020 - Country of manufacture label (IEC 60417 No. 6049) - Which national law requires this symbol? Other Medical Device Related Standards 0
P ISO 14644 Class 8 Cleanroom Air Filter Requirements Other Medical Device Related Standards 4
K PDCA cycle and ISO processes alternative model Quality Management System (QMS) Manuals 14
N ISO 13485 7.3.9 Change control in medical device software ISO 13485:2016 - Medical Device Quality Management Systems 6
A ISO 13485 procedure change and reflect to legacy manufacture items ISO 13485:2016 - Medical Device Quality Management Systems 2
D ISO 13485 & CE Certification for Surgical Gloves CE Marking (Conformité Européene) / CB Scheme 0
S ISO 11137- Simulated product vs SIP Other Medical Device Related Standards 2
D Which ISO Standard to purchase? ISO 13485:2016 - Medical Device Quality Management Systems 7
V ISO 10360-5: 2020 Gap analysis and Action plan Excel .xls Spreadsheet Templates and Tools 1
Q ISO 9001 - Reseller Exclusions ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 7
S Inventory Listing and ISO 13485:2016 ISO 13485:2016 - Medical Device Quality Management Systems 3

Similar threads

Top Bottom