SBS - The best value in QMS software

KPIs or Metrics to Measure a New Complaint Handling Process

#1
We are working on developing and implementing a new medical device complaint handling process to assure that complaints are collected at the manufacturing sites, evaluated by them for the potential need to submit an MDR, and then forwarding the information to a centralized team to make the final MDR decision and to submit the MDR if needed.

I'm trying to define some KPIs or metrics that would help assure that the manufacturing sites continue to process complaints as required, and in a timely fashion. For example, if a staff member left who was responsible for reviewing and forwarding the complaint to the central office, is that still happening or is there a sudden drop-off in submissions.

Some thoughts I've had follow below. Can you suggest others that would help monitor the process?
  • Total number of Open complaints at month-end. Monitor average - maybe trigger points if we fall below a "normal" lower limit.
  • Break down by state of complaint record. (Received, Waiting Return, Under Investigation, etc.)
  • Total number of Closed complaints at month-end. Again, based on average volume is level of closed records "normal".
  • Average days to closure for records closed in the month.
  • Number of days from Aware Date to date the complaint was opened.
  • Number of days from Date Opened to 1st, 2nd, and 3rd attempts to have product returned for evaluation.
  • Number of days from date product received to date the investigation was completed. Are investigators, R&D, Engineering being timely with their evaluations?
  • Number of complaints closed without investigation – were rationales to do so valid?
  • Number of complaints miss-categorized upon initial intake – meaning how many were thought to be a minor issue (no harm) but further investigation revealed more serious situation and potential need for MDR.
 
Elsmar Forum Sponsor

John Predmore

Trusted Information Resource
#2
One approach which might generate fresh insights - consider a "value stream map" of how a complaint moves through your process from generation to resolution. Decide what aspects of the sequence you are interested in and how you might measure and monitor those in meaningful ways. Lean practitioners think about systems using two types of measures - speed by which items move through the sequence, and buffers (inventory) where items get held up. With complaints, do you always want FIFO or LIFO, or how is that best handled? Are there reject/rework loops for complaints? - you brought up mis-classification of majors as minors, for example.

Depending on how far upstream you want to go, you might also track aspects of the sequence like time between sale and problem noticed, or from date product is put into service to when complaint is received. Do your results vary significantly by the source of complaint, call center versus website?

I often find thinking in analogies is helpful for creative thinking, and a 10,000-foot view might lead you to consider aspects of the system which are important but less obvious.
 

RoxaneB

Super Moderator
Super Moderator
#3
My concern with metrics that are "number of..." is that they give little to no context. What if the volume of product being shipped increases? Odds are the "number of complaints" will increase. To be truly meaningful, I like to show both the number and the %...that's the story behind the process metrics.

Your list is rather long, in my opinion, and many of the items are more like control-items, not KPIs. For example, the "# of days..." - this speaks more to process flow and responsibility and maybe even resources. But are they truly KEY performance indicators.

Ask yourself...rather, ask the team...what are you trying to achieve with this new complaint process? Why is it new? Why was the old way considered obsolete? From that discussion, you may discover some true KPIs, while everything else is a driver towards those KEY metrics.
 

Ron Rompen

Trusted Information Resource
#4
I agree with Roxane, particularly about showing metrics as a percentage of ........, rather than just a number. Makes it much more relevant to what is actually happening.
I also agree that your list is far too long - pick out the 3 most important (to YOU and your company) metrics, and monitor those. After 6 months, revisit the list, and see if it needs to change. Your metrics don't need to be static - they should reflect what is really happening, and what you identify as important.
 
#5
Thanks for the input - great suggestions! I, too, was worried about the number of measurements (paralysis by analysis) and will work to pare them down from the team's first brainstorming session.
 

Jen Kirley

Quality and Auditing Expert
Staff member
Admin
#6
What matters to your customer? What would effectiveness and efficiency look like in customer complaint handling? What does failure look like?

Customer satisfaction can be maddeningly hard to measure, possibly more so while even more importantly so during complaint handling because the customer is already dissatisfied. This is your chance to keep them or else. Focus on what is needed to do that.

Then have a follow up contact with the customer to learn if it worked and how they feel about the interaction and problem solving process. Use that feedback to understand opportunities for improvement and make them.
 
#7
We are working on developing and implementing a new medical device complaint handling process to assure that complaints are collected at the manufacturing sites, evaluated by them for the potential need to submit an MDR, and then forwarding the information to a centralized team to make the final MDR decision and to submit the MDR if needed.

I'm trying to define some KPIs or metrics that would help assure that the manufacturing sites continue to process complaints as required, and in a timely fashion. For example, if a staff member left who was responsible for reviewing and forwarding the complaint to the central office, is that still happening or is there a sudden drop-off in submissions.

Some thoughts I've had follow below. Can you suggest others that would help monitor the process?
  • Total number of Open complaints at month-end. Monitor average - maybe trigger points if we fall below a "normal" lower limit.
  • Break down by state of complaint record. (Received, Waiting Return, Under Investigation, etc.)
  • Total number of Closed complaints at month-end. Again, based on average volume is level of closed records "normal".
  • Average days to closure for records closed in the month.
  • Number of days from Aware Date to date the complaint was opened.
  • Number of days from Date Opened to 1st, 2nd, and 3rd attempts to have product returned for evaluation.
  • Number of days from date product received to date the investigation was completed. Are investigators, R&D, Engineering being timely with their evaluations?
  • Number of complaints closed without investigation – were rationales to do so valid?
  • Number of complaints miss-categorized upon initial intake – meaning how many were thought to be a minor issue (no harm) but further investigation revealed more serious situation and potential need for MDR.
I measure #complaints/#shipments <- by site/region/globally, so its a ratio (or you can convert to a %). What is interesting is when you treat this statistically: a complaint = a defect The entire process of taking the order through billing can be viewed as a "manufacturing process". We finished the year at 0.7% which is a dpmo of 7,000 or ~4 sigma. Some of our complaints can be resolved quickly others require the customer to send a sample of the product back for analysis. I set a target of 21 working day average for all complaints AND no complaints >45 days. From previous data, I saw we could achieve the 21 day target if we eliminated the very long (forgotten....) complaints.
 
#8
There are two major metrics to consider, namely Wait Time and Cycle Time. Wait Time or Age is how long an action is pending, which if not met might lead to a non-compliance. Cycle Time involves how productive the team is and how well the team is utilized for the task (including mapping and getting things done where there are dependencies). There are lot of metrics that will need to implemented in complaint handling. However, I may not be able to post them in public forums. Hope the information I shared might be useful to you to some extent, think along these lines.
 

RoxaneB

Super Moderator
Super Moderator
#9
There are two major metrics to consider, namely Wait Time and Cycle Time. Wait Time or Age is how long an action is pending, which if not met might lead to a non-compliance. Cycle Time involves how productive the team is and how well the team is utilized for the task (including mapping and getting things done where there are dependencies). There are lot of metrics that will need to implemented in complaint handling. However, I may not be able to post them in public forums. Hope the information I shared might be useful to you to some extent, think along these lines.
I disagree with both of these metrics. Maybe they have not been explained properly, but:

- Wait time | What do you mean "might lead to a non-compliance"? There is already a customer complaint...there is nothing to "lead" to.
- Cycle time | Data that is too subject to human intervention. Receive complaint, immediately open, hit resolve button, complaint closed in under 10 seconds. An over-exaggeration, perhaps, of what could occur, but the cycle time is not a KEY indicator when it comes to the complaint process.

Again, it goes back to what is the intent of the complaint process? Is it to process as many complaints as possible in as short a time window as possible? ... OR ... is it to help identify trends with customer experiences in order to increase the overall satisfaction with the product/service? It cannot be both. One (I'd offer the 2nd option) is the WHY while the 1st is more the WHAT or HOW.

Using the complaint process to offer insight into items such the nature and cause behind the complaints is where the meaningful data lies.
 

Tagin

Trusted Information Resource
#10
Restructuring your sentences as below (which is how my mind reads what you wrote), it seems like you are attempting to compensate for weaknesses in the process being created with an abundance of metrics. Many of those metrics would be unnecessary if the process were more robust.

We are working on developing and implementing a new medical device complaint handling process to assure that complaints are:
  • collected at the manufacturing sites,
  • evaluated by them for the potential need to submit an MDR, and then
  • forwarding the information to a centralized team to
  • make the final MDR decision and to
  • submit the MDR if needed.
I'm trying to define some KPIs or metrics that would help:
  • assure that the manufacturing sites continue to process complaints as required, and
  • in a timely fashion.
Is it not possible in this day and age to create a central system that all the sites could enter their information into directly, and then the centralized team can take action on, once they are marked as 'evaluated'? This also gives the central team immediate visibility to newly-entered complaints, and gives someone the ability to monitor & report on time in each phase (entry>evaluate, evaluate>makedecision, makedecision> submit)...if ever needed for diagnostic purposes; however, monitoring overall time from entry>makedecision should suffice for a higher-level metric of responsiveness, to meet the goal of "in a timely fashion".

As for "For example, if a staff member left who was responsible for reviewing and forwarding the complaint to the central office, is that still happening or is there a sudden drop-off in submissions", this is a symptom of a weakness in the process that metrics won't solve. You could, as a crude example, require that the sites report a weekly summary of the # of complaints entered that week. If no complaints and no summary is reported for a site, then it will raise a flag to the central team that the site is unresponsive and they need to check that maybe that person has left or is sick, etc.; it also acts as a sanity check of the number the site thinks they submitted vs. the number the central team received. Build those controls into the process, rather than trying to control-by-measurement a weak process after the fact. :) Since this is a new process, I think there is an opportunity here for your group to think big, and think how the process design itself can create confidence that activities are being performed as intended.

As for KPIs (rather than metrics), I agree with @RoxaneB - if you really want KPIs, then you really need to look at is what the purpose and value of this process.
 
Thread starter Similar threads Forum Replies Date
B Laboratory KPIs ISO 17025 related Discussions 13
R Monitor production quality - Internal KPIs Manufacturing and Related Processes 5
C Calculating your KPIs ISO 13485:2016 - Medical Device Quality Management Systems 7
qualprod Proposed Kpis ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
P Received a minor for not having good measureables/goals. Need help with KPIs. IATF 16949 - Automotive Quality Systems Standard 52
K IATF16949 inputs/outputs & KPIs Internal Auditing 3
Q Corrective Action - Assigning KPIs to Non-Guilty Processes ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
Q Suggested KPIs for a Manufacturing Production Business ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 1
Q How to align a Business Strategy to Operative KPIs ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 12
Q KPIs - Lead and Lag Indicators - ISO 9001:2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
A If 4.1 is a process, how to assign KPIs for this process? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 12
Buckyb Are KPIs a requirement to determine Process Effectiveness? AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 14
J Lean Appropriate KPIs for a Distribution Centre Lean in Manufacturing and Service Industries 1
P New to AS9100 Rev. C - Setting KPIs and FAI Reports AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 14
S Process Map and defining KPIs Misc. Quality Assurance and Business Systems Related Topics 5
K Can you Suggest KPIs for a Business Process Improvement Team? Pls. Benchmarking 4
R Document Control Person - Setting KPIs, Development Plan, and Career Path Quality Manager and Management Related Issues 6
R Key Process Indicators (KPIs) for ISO 9001:2000-certified Service Organization ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
R KPIs (Key Performance Indicators) for Telecom Operators TL 9000 Telecommunications Standard and QuEST 1
B Is there a necessity of two sets of KPIs for Delivery Process Quality Manager and Management Related Issues 15
M First Draft of Policy, Objectives and KPIs - Your feedback appreciated ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 10
I Are KPIs (Key Performance Indicators) considered as Quality Objectives? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 7
A Aircraft Maintenance Quality & Safety KPIs - Industry Trend or Database EASA and JAA Aviation Standards and Requirements 3
A Aircraft Maintenance Quality & Safety KPIs - Industry Trend or Database Quality Manager and Management Related Issues 4
R Quality for Telco Companies - Quality in Customer Experiences & KPIs Quality Tools, Improvement and Analysis 5
0 Problem with setting up KPIs via Excel Pivot Table Misc. Quality Assurance and Business Systems Related Topics 6
Q Training KPIs (Key Process Indicators) Training - Internal, External, Online and Distance Learning 8
A Department KPIs - Plant, Machineries and vehicles (PMV) ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 1
Jim Wynne "Business Coach" Touts Basing Pay on KPIs Philosophy, Gurus, Innovation and Evolution 43
S Differences Between KPIs and Objectives/Targets ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 11
S Do Key Performance Indicators (KPIs) really make a difference in the QMS? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
D Key Performance Indicators (KPIs) in the Construction Industry ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
G KPIs required by TOP Management for strategic planning Statistical Analysis Tools, Techniques and SPC 2
D Key Performance Indicators (KPIs) book - Reviews and suggestions wanted Statistical Analysis Tools, Techniques and SPC 9
S KPIs (Key Performance Indicators) and Risk Management Process (ISO14971:2007) ISO 14971 - Medical Device Risk Management 5
B Defining Physical Asset Maintenance KPIs (Key Performance Indicators) Benchmarking 6
Ajit Basrur Customer Satisfaction Survey Content - Which KPIs employed? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
J Key Performance Indicators (KPIs) for Management Review process Management Review Meetings and related Processes 37
J Document Control Metrics Document Control Systems, Procedures, Forms and Templates 3
bruceian Software Quality Metrics Software Quality Assurance 11
D Engineering Change Order (ECO) metrics to track ISO 13485:2016 - Medical Device Quality Management Systems 6
M Quality Metrics Questions Lean in Manufacturing and Service Industries 2
CPhelan Best metrics for monitoring low volume (<1000) parts per vendors Supply Chain Security Management Systems 1
G Defining performance metrics for DFMA implementation Design and Development of Products and Processes 2
W Core Process Metrics AS9100D for Product Planning? Quotes and sales? AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 1
R Visually impacting graphics for Nonconformance status metrics for Management Report Nonconformance and Corrective Action 5
Ed Panek User Feedback both negative and positive and acting upon those metrics 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 4
andika_untoro What are a good Critical to Quality (CTQ) metrics for a biotech manufacturing production? Lean in Manufacturing and Service Industries 0
D Need Resource for Industry Data - standard metrics for machine shops Lean in Manufacturing and Service Industries 3
V Information Technology Dashboard - IT Department Performance Metrics IEC 27001 - Information Security Management Systems (ISMS) 2

Similar threads

Top Bottom