Should volume of sales be factored into risk probability assessments?

Bev D

Heretical Statistician
Staff member
Super Moderator
#11
Mark - you are misinterpreting probability and mis-using the term “likely”. I get what you are trying to say but it’s wrong. May I suggest that instead of trying to defend your position, you think about the feedback you are getting. you asked the question - we are answering it. Learning requires that we challenge our thinking not seek to confirm it.

First, the example of sales that you sight is an extreme and not a reflection of actual reality. Companies do not launch products thinking that they will only sell 10. Nor will their concern increase if they sell more. IF failures were randomly distributed, then the probability of a death in the first 10 is exactly the same as in the first million. That is just how probability works even if we wish it didn’t.

Secondly, failures are very rarely randomly distributed. This ‘assumption’ is a teaching ideal, not a reflection of actual reality. Failures are caused by physical mechanisms that are most often clustered and non-homogenous. They are not independent of each other...they do not ‘magically ‘ occur.

Most importantly the whole concept of a trying to determine the ‘probability’ of a failure before it happens is fundamentally flawed. It is impossible and is usually grossly underestimated under-guessed.

Like it or not the truth is that the probability of a death doesn’t change as you sell more. I sense that you are trying to use this to make some point in your organization - perhaps to get people to pay more attention to mitigating high severity/high hazard things? So, perhaps it would be more productive to discuss what your real point is? Why does this ‘probability’ matter to you? How would you use this personal interpretation?
 
Elsmar Forum Sponsor

Jean_B

Trusted Information Resource
#12
For it to matter it has to change decisions you make. In the base risk management of a single device's design it might not. But it might be of interest with regards to some topics that do not assume a single device to be risk managed, but its facilitating/supporting systems.
Number one: insurance coverage. The EU MDR explicitly demands you assess this, and whether you go insurance or piggy bank you want to not just have probability, but absolute number of events times payout as an input in your calculations.
Number two: support coverage. If your fleet grows, and that fleet needs maintenance you deal with an interaction with maintenance/service capacity. These are not one-to-one controlled, but lack of timely service and maintenance might affect occurrence of existing risks. This does not introduce a new risk, but the consideration of current fielded product needing maintenance versus available capacity might increase the probability that maintenance/service is delayed or skipped and should be considered in your regular reviews. State of maintenance/maintenance frequency intended/unintended could be seen as a characteristic that affects safety.
Number three: Traceability. With a larger amount of product in the field, often the production volume per period is also increased. A consideration on what amounts to acceptable traceability levels (material, component serial number/lot, component version, assembly serial number, production serial number/batch, production date etc.), methods (human readable, barcode, rfid) and systems (store as paper, excel, integrated into your ERP/MRP,Internet of Things level) can differ depending on the size and configuration complexity of your fleet. This might affect the design of those products and production processes where preparing for timely field safety action is seen as a necessary mitigation.
These can be hard to fit in your typical risk management file from a device standpoint, but the issues are not purely academic.
 

Mark Meer

Trusted Information Resource
#13
...May I suggest that instead of trying to defend your position, you think about the feedback you are getting. you asked the question - we are answering it. Learning requires that we challenge our thinking not seek to confirm it.
...I sense that you are trying to use this to make some point in your organization - perhaps to get people to pay more attention to mitigating high severity/high hazard things? So, perhaps it would be more productive to discuss what your real point is? Why does this ‘probability’ matter to you? How would you use this personal interpretation?
@Bev D, with all due respect, you're completely misreading me, and unless I'm misreading you, this post comes across as staggeringly condescending. :confused:
I like discussion, I like back-and-forth argumentation, I like to explore as many angles to an idea as possible....It's as simple as that.

For background, I was reviewing the EU MDR, and noticed that the regulations now require reports on the volume of sales. I wondered the rationale for the requirement. Does this have anything to do with public health or risk, and if so, how?

My first thought was that it might be that probabilities of people getting harmed scales with the number of users, and thought what better place to discuss than the Cove. ;) But as @Jean_B points out there are several other considerations to do with "risk", albeit not from a device design perspective, so perhaps a thread name change is in order. This is all in the spirit of friendly discussion; no ulterior motives.

... you are misinterpreting probability and mis-using the term “likely”...the probability of a death in the first 10 is exactly the same as in the first million...
Forgive me, but I honestly still don't understand where my thinking is going awry. If my goal is to get a coin to show heads, is it not more likely that I achieve this goal if I'm given 10 coin-tosses, versus if I have only one?
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#14
My response was not meant to be condescending at all. rather to move us past saying the same things to each other. and I apologize fro any offense.

Let me approach the coin tossing question as it is a classic one to address probability. Part of the confusion here is that most people think differently about probability and use statistical terms from a 'layman's' definition and not the statistician's definition of the phrase. "Confidence" and "Significance" have the same issue...

The short answer to the coin question that the probability doesn't change it is always 50% chance that you will get a head. (in a 'fair' coin, with a 'fair' toss). If you toss coin once there are 2 outcomes: a head or a tail. Sometimes you will get a head and sometimes you will get a tail. Over many tosses, you will get a head half of the time. If you toss a coin 10 times there are many outcomes and it is highly improbable that you would get no heads. From a numerical standpoint you will almost always get more heads when you flip the coin 10 times than if you flip it once. This is where most people misinterpret probability and say it is more likely to get "a" head if you flip a coins 10 times than if you flip it once. But the probability of getting a head in any given flip doesn't change. The way to show it to yourself is the old fashioned way. do the experiment yourself. Flip a coin once. record if it was a head or tail. then flip it 10 times record the number of heads. Repeat that experiment 20 times. let us know the answer. Jean_B approached the problem you raise correctly by addressing the cost of increased numbers of events. The number of events increases as the sales volume (area of opportunity) increases. and there are additional cost concerns or investments that must be made as the number increases...
 

Miner

Forum Moderator
Staff member
Admin
#15
This will probably throw some fuel on the flames, but here goes. Disclaimer: I am not in the medical field, but do deal with risk assessment in other disciplines.

Risk is usually defined as the likelihood of occurrence x the consequence of the occurrence. One of the things many find confusing is the lack of a standard definition for likelihood (in the context of risk). A few searches will show that some people equate likelihood with probability, in which case volume does not matter because, as Bev stated, the probability does not change. However, you will also find an almost equal number of people that equate likelihood with the number of occurrences over a given time period, in which case volume does matter a lot. My company uses the latter as it better reflects actual occurrences (# of times I will feel the pain). I don't know whether the medical community has a standard definition for likelihood, but in general other industries have not.
 

Mark Meer

Trusted Information Resource
#16
...From a numerical standpoint you will almost always get more heads when you flip the coin 10 times than if you flip it once. This is where most people misinterpret probability and say it is more likely to get "a" head if you flip a coins 10 times than if you flip it once. But the probability of getting a head in any given flip doesn't change.
Forgive my obstinance, but as an admitted layman I'm still not understanding the misinterpretation of "probability". If you could kindly clarify which of these statement is wrong, or where the term "probability" is being misused:
  • Probability of heads showing up in a given flip = 1/2.
  • Probability of no heads in 10 flips = 1/2^10 = 1/1024.
  • Probability of at least 1 heads in 10 flips = 1 - 1/1024 = 1023/1024
  • The probability 1023/1024 is greater than the probability 1/2. In other words, the probability of heads showing up in 10 flips is greater than the probability of heads showing up in 1 flip.
 

Tidge

Trusted Information Resource
#17
Forgive me, but I honestly still don't understand where my thinking is going awry. If my goal is to get a coin to show heads, is it not more likely that I achieve this goal if I'm given 10 coin-tosses, versus if I have only one?
If the coin is a medical device, before you marketed it you already determined that it was acceptable for it to come up heads! If coming up heads is unacceptable, it shouldn't be on the market.

This will probably throw some fuel on the flames, but here goes. Disclaimer: I am not in the medical field, but do deal with risk assessment in other disciplines.

Risk is usually defined as the likelihood of occurrence x the consequence of the occurrence. One of the things many find confusing is the lack of a standard definition for likelihood (in the context of risk). .
Let be the first one to introduce you to the standard (ISO 14971: 2019) definition of risk in the context of medical devices:

Risk is the combination of the probability of occurrence of harm and the severity of that harm.​

This is distinct from business risk. If your only prior exposure to the concept of risk comes from outside medical devices, I can understand your confusion. For example, if you attain certification through ASQ you will be exposed to risk in a different context that is not the same as the concept of risk in medical devices.

See next post:
 

Tidge

Trusted Information Resource
#18
Forgive my obstinance, but as an admitted layman I'm still not understanding the misinterpretation of "probability". If you could kindly clarify which of these statement is wrong, or where the term "probability" is being misused:
The distinction we are trying to make is that the probability of a result doesn't change with the number cases (throws of a die, tosses of a coin). This is the fundamental reason why "more cases, higher volume) can't affect the probability of an outcome, except to narrow the uncertainty around the probability. (i.e. more data gives better understanding).

In the assessment of risk for medical devices, you determine acceptability based on the probability of potential outcomes, not on the outcomes. If 1-in-1024 users will die from exposure to a toothbrush, that is almost certainly unacceptable. That doesn't necessarily mean that it is unacceptable that 1-in-1024 cardiac patients will die from the installation of a particular device. The medical device manufacturer is supposed to assess this risk prior to marketing the device, and have a mechanism for periodic evaluation of the risk assessment to make sure that the initial assessment was correct, and to make adjustments/corrections if it was not.
 

Mark Meer

Trusted Information Resource
#19
...If 1-in-1024 users will die from exposure to a toothbrush, that is almost certainly unacceptable. That doesn't necessarily mean that it is unacceptable that 1-in-1024 cardiac patients will die from the installation of a particular device. The medical device manufacturer is supposed to assess this risk prior to marketing the device...
But why is this, if number of users is not a factor in the risk assessment? If risk = probability & severity, and in this case severity is equal (dying), then on what basis do you conclude that this risk is "almost certainly unacceptable" in the case of toothbrushes, but may not be for a cardiac device?

Speaking philosophically for a moment, if I were a public policy maker or regulator, would it not be reasonable to expect a higher bar for safety if a device was expected to be in millions of households versus only a few hundred? My intuition tells me yes, but then I'm struggling to map this on to the way we currently approach safety in medical devices.

I guess, for the sake of discussion, we can leave probability aside, and simply ask: does consideration of expected number of users have any place in a medical device risk management framework?
 
Last edited:

Tidge

Trusted Information Resource
#20
But why is this, if number of users is not a factor in the risk assessment? If risk = probability & severity, and in this case severity is equal (dying), then on what basis do you conclude that this risk is "almost certainly unacceptable" in the case of toothbrushes, but may not be for a cardiac device?
It is incumbent on the manufacturer to establish the criteria for acceptability. A guiding principle is that for everyday, common health maintenance devices (e.g. toothbrushes) the threshold for unacceptable risks is lower, but that for devices used in acute life-saving situations the threshold for unacceptable risks is higher: that is, we (patients, users, manufacturers) accept more risk when the alternative to not accepting the risk is extreme.

I guess, for the sake of discussion, we can leave probability aside, and simply ask: does consideration of expected number of users have any place in a medical device risk management framework?
I tried to answer this in my first reply on the thread. The only immediate place I see where the expected (but not established *1) number of users has a place in a 14971-compliant process is establishing the risk acceptability limits (which are done at the very beginning of a risk management process). This is what I was implicitly considering in the toothbrush/cardiac device example.

I can imagine an argument for trying to apply the expected (but not established) number of users in a risk-benefit analysis, but doing so to accept risks that you had pre-determined to be unacceptable would have severe implications to my way of thinking.

(*1) I think it is important to distinguish between a large number of known users and a large number of hypothetical users. If you have actual data, the data can be used to inform certain elements of the risk management process. If it helps, think of the Bayesian approach to probability theory. In the absence of data, all you have are your prior beliefs.
 
Thread starter Similar threads Forum Replies Date
D Should low volume supplier PPM comparisons consider level of opportunity for error? Quality Tools, Improvement and Analysis 2
W Should we monitor all Significant part Characteristics with SPC? Low Volume CNC Statistical Analysis Tools, Techniques and SPC 15
Sidney Vianna Interesting Discussion Should ISO 9004 be changed from a guidance standard to a requirements standard? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
A Should I take an online course for a career in Occupational Health and Safety? Career and Occupation Discussions 2
J Should a Class 1 medical device with an option to measure body weight be considered Class 1m? EU Medical Device Regulations 0
K Should APQP/PPAP has its own section in a QM? Quality Management System (QMS) Manuals 1
S What should i choose for "testing procedure" characteristics? (N95) General Information Resources 0
P Should eIFU link per ISO 15223-1:2016 be added to labels out of scope of Reg 207/2012? EU Medical Device Regulations 1
S Which Sampling Plan(s) Should I Use? Inspection, Prints (Drawings), Testing, Sampling and Related Topics 7
A Document release vs its related training. Which should come first? ISO 13485:2016 - Medical Device Quality Management Systems 18
S Which department should prepare the control plan? could you show me a standard regarding to this matter. FMEA and Control Plans 17
J Help settle a disagreement: Should external providers of preventive maintenance be on your ASL? AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 5
N Master Samples - What should we be keeping? IATF 16949 - Automotive Quality Systems Standard 9
G Supplier delivered recent PPAP, should he deliver yearly layout inspection? IATF 16949 - Automotive Quality Systems Standard 4
John Broomfield Vote - Should ISO9004 Become a Requirements Standard? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 11
A Capability Study - in the beginning of your career what should you have known about the tool Quality Tools, Improvement and Analysis 11
J Should Loading and Unloading be Included in Cycle Times? Lean in Manufacturing and Service Industries 14
E Manufacturers should develop a testing device for covid19 Service Industry Specific Topics 0
T 510(k) submission - Which name should I use in the submission? Other US Medical Device Regulations 3
N ISO 19011:2018 - 5.4.2 "...audit program should engage in appropriate continual development..." Training - Internal, External, Online and Distance Learning 4
G Should I perform Gage R&R only at the beginning of a new project? Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 6
DuncanGibbons Should the requirements FAA/EASA Part 21 be addressed within the QMS and AS9100D quality manual? AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 5
M Should 510(k) Predicates be Actively Listed Devices? Other US Medical Device Regulations 12
B Why the Greek god Hephaestus should have done a design FMEA (DFMEA) on his giant robot APQP and PPAP 1
J On PFMEA for danger labels - Label always should be assigned severity 10 ? FMEA and Control Plans 3
H Who should be listed as the manufacturer/distributor on the box? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 15
M MDR, RED and LVD - Should our device comply with them? EU Medical Device Regulations 3
BeaBea How Many Processes should be created for each Department? Process Maps, Process Mapping and Turtle Diagrams 5
MrTetris Should potential bugs be considered in software risk analysis? ISO 14971 - Medical Device Risk Management 5
S Should safety checks be included in the Control Plan? IATF 16949 - Automotive Quality Systems Standard 5
M Which incubation condition should be selected to recover both bacteria and fungus effectively Miscellaneous Environmental Standards and EMS Related Discussions 3
D Is there a specific location for PPE such as safety glass holders and glove dispensers should be mounted Occupational Health & Safety Management Standards 10
Robert Stanley Which Registrar Should I Choose for ISO 9001:2015 registration? Registrars and Notified Bodies 10
M Who should receive the bills from suppliers and vendors, account payable or procurement? Consultants and Consulting 4
V IATF 16949 8.4.1 Control of externally provided processes, products and services - Should the CB be on our Approved Supplier List? IATF 16949 - Automotive Quality Systems Standard 10
A We are ISO 13485:2016 should we be audited to ISO 14971 ISO 13485:2016 - Medical Device Quality Management Systems 16
E Received a Major finding during IATF Surveillance audit for loss of BIQS Level 3 (more than 6 SPPS in 6 months)...how should we address SYSTEMIC CA? IATF 16949 - Automotive Quality Systems Standard 11
J Organization merger. Should we keep two separate ISO 13485 certificates? ISO 13485:2016 - Medical Device Quality Management Systems 6
S Companies that maintain your machine should be in ASL? AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 2
S Use of "Shall" versus "Should" in Procedures ISO 13485:2016 - Medical Device Quality Management Systems 26
D Class II medical device - When should a complaint be closed? Customer Complaints 6
Sidney Vianna IATF 16949 News Presentations from the latest IATF Stakeholder Event - Expectation that IATF 16949 certification should equate with product quality. Misguided? IATF 16949 - Automotive Quality Systems Standard 7
L Clause 0.4 of ISO 9001 and EHS - Where should I stop the inclusion of EHS in my QMS ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 8
Ed Panek Part 11 Self Certify Memo - What else should it cover? Qualification and Validation (including 21 CFR Part 11) 5
H Should I mention machine/Equipment password In SOP? Qualification and Validation (including 21 CFR Part 11) 4
D How long should we keep the spare parts available for our medical device, after we have stopped the production? ISO 13485:2016 - Medical Device Quality Management Systems 0
H Statistical Techniques Procedure - What should be included Document Control Systems, Procedures, Forms and Templates 4
Q How should I analyze measurement correlation between me and customer? Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 12
Sidney Vianna Interesting Discussion ISO 9001:2024 - What should be changed in the next Edition of ISO 9001? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 113
N Should it even be on the hazard analysis (software)? FMEA and Control Plans 2

Similar threads

Top Bottom