SBS - The Best Value in QMS software

Software Quality Metrics

D

DRAMMAN

#1
Any opinions or experience developing SW quality metrics? I am looking ot establish SW quality metrics that monitor performance durring the development process and post release.
 
Elsmar Forum Sponsor

yodon

Staff member
Super Moderator
#2
Software quality metrics are challenging and can lead to encouraging bad behavior if not watched closely. For example, if you proclaim that number of lines of code per day per developer is good, you'll get lots of lines of code but not necessarily quality. If you proclaim the number of bugs fixed per day per developer is good, you'll get plenty of bugs fixed but not necessarily improving quality. If you proclaim the number of bugs found by test is good, they'll find bugs, but many that aren't too useful.

Static analysis tools can give good insight (e.g., code complexity) but don't get religious over the numbers. There are often very good reasons for added complexity in code.

Clearly, the best measure of software quality is the number of problems reported from the field. But that's more of a systemic measure than software quality; i.e., why was the bug introduced in the first place, what was the earliest point where it should have been detected (phase escape), etc.

The absolute worst thing you can do is to try to use the metrics to assess performance. As soon as there's any inkling of management doing that, you can kiss any thought of getting rational data out. Developers will quickly slip into the habit of making the numbers look good rather than focusing on quality. Along the same lines, don't punish developers if bugs do get into the field. If that does occur (and it will - there is no bug-free code), a number of factors had to occur. Presuming the requirements and design were correct and the developer did introduce the bug, why wasn't there a review that caught the error? Why didn't unit and/or integration testing catch the error? Why didn't system testing catch the error? It would be like blaming a goalie in hockey for goals allowed (where was the rest of the defense?).

Since the goal for metrics should be for eventual improvement, I would (instead) set up a feedback / improvement process. I would start with problems reported from the field. Do an analysis on each as to why the problem occurred and where the problem should have been caught. If you have a large number of bugs reported, you may need to employ a prioritization scheme to help focus. Generally, developers get on board if the focus is kept on improvement and not for punitive actions.
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#3
expanding from what yodon said:

Every metric can be manipulated and abused; a best practice is to have a set of metrics that are based on people (morale or engagement), quality, delivery and cost. Each of these are of equal importance and since this is a system, if you work on one you will affect the other 3. so it's important to understand that improvements (or at least no negative affect) in all 4.

An approach I've tried in these types of cases is to get the most recent field failures / complaints, investigate as to why they occured and why they escaped. Then intiate 'fixes' and improved processes to better detect, correct and prevent. during this process ask the developers what metrics they think will help keep them on track and be viewed as great evidence of their improved performance....
 
D

DRAMMAN

#4
Thanks for the feedback. We are in the early stages of implementing metrics and processes to drive continuous improvement in the SW portion of our products. Do any of you define a difference between "issues", "bugs", "field problems", etc? Or do people tend ot use the terms interchangeably.
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#5
well, as a colleague of mine says "people have issues, equipment has problems". So I tend to not use the term 'issue'.

I use "Customer complaint" which can be an imprecise description of the problem as they experienced it.

I will also use "error" - if there is a specific error 'code' that is displayed. Sometimes the term "fault" if there isn't an error code...it may also be referred to as the "failure mode" => some function failed in a specific manner. The key here is that error or fault should be a precise description of the Problem from the effect side. It should not involve descriptions of the cause.

I reserve the term 'bug' for the cause of the failure. I do this becuase Grace Hopper coined the phrase when someone found that a 'bug' (moth?) had caused a short in an electrical component of a computer that casued it to malfunction. so 'bug' is a cause and not an effect.

I do recommend the "apollo method" for a comprehensive description of the cause and effect relationship.

however, you can use any terminology that you wish as long as you clear about the definitions and people in your organization stick to the definitions. otherwise confusion shall result!
 

yodon

Staff member
Super Moderator
#6
well, as a colleague of mine says "people have issues, equipment has problems". So I tend to not use the term 'issue'.
Such discussions make me think of one pundit's approach. He suggested calling things that weren't right "spoilage." Can't say I think that's a good idea nor do I buy in to being overly 'politically correct' but, as Bev points out so well, words have multiple meanings. What things are called could certainly have effects, depending on the company culture. Personally, I use "issue" and have instilled that it's never personal; individuals are never blamed. The culture here is quite accepting of this approach.
 

michellemmm

Quest For Quality
#7
Thanks for the feedback. We are in the early stages of implementing metrics and processes to drive continuous improvement in the SW portion of our products. Do any of you define a difference between "issues", "bugs", "field problems", etc? Or do people tend ot use the terms interchangeably.
I always become concerned when organizations' approach for setting metics starts at micro level and top management set it at macro. "issues" or "bugs" might be significant to some and "defect density" might be more meaningful to others.


No matter what type of metics you select, it should measure effectiveness Vs efficiency of the process, driven from company's goals and objectives . Somehow micro and macro should relate.
 
D

DRAMMAN

#8
michelle.....completly agree with you.

Anny....Do you have any examples or thoughts on what sftware metrics have worked well in your organizations? Or even which have not.
 
F

flyin01

#9
michelle.....completly agree with you.

Anny....Do you have any examples or thoughts on what sftware metrics have worked well in your organizations? Or even which have not.
My two cents on this is that you should separete the metrics into two parts.

1.Measure the process itself. Could be how efficient the developers are in terms of submitting patches, handling feedback/issues/bugs (responding to them, tagging them, assigning them etc). Time measurements, numbers of items.

2.Measure the output of the process. I e the SW blob itself that come out of your developers hard efforts. What is important to the customer? Don?t guess, ask! Is it how snabby the SW is (ms), number of bugs/crashes (#, %). Have a few metrics that make sense and prioritize them, then focus on these. Don?t go for the score card with 57 metrics. This will only cause head ache. Don?t make the formulas too complex. I am quoting Einstein here, if you cannot explain it to a 6 year old... :D

Sometimes you do not yet have constructive feedback from the customers in terms of what went wrong. (Maybe you have not release a final product yet) Then it may be really tricky to find out what matters, but you can benchmark the competition.

I hope it helps! :bigwave:
 
Thread starter Similar threads Forum Replies Date
bruceian Software Quality Metrics Software Quality Assurance 11
Ajit Basrur Dashboard Software for representing Quality Metrics like DPPM, On Time Delivery, etc. Quality Tools, Improvement and Analysis 4
O Any info on release date of FDA “Computer Software Assurance for Manufacturing and Quality System Software” document? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 0
S SOP for ISO 13485:2016 Quality related Software validation ISO 13485:2016 - Medical Device Quality Management Systems 9
W Air Quality Measurement Hardware and Software General Measurement Device and Calibration Topics 11
K Templates for software development quality audit Document Control Systems, Procedures, Forms and Templates 1
R ISO 13485 Software validation procedure and Quality Objectives Monitoring wanted Document Control Systems, Procedures, Forms and Templates 1
T Software quality training course Training - Internal, External, Online and Distance Learning 2
R Quality Management Software Recommendations ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
S ERP Software and Quality Department Procedures Integration Misc. Quality Assurance and Business Systems Related Topics 4
P Need suggestion on Software or App for Tracking Quality Plan Status Software Quality Assurance 1
A Quality system vs SDLC (Software Development Life Cycle) (or ALM)? Other US Medical Device Regulations 2
G QMS Quality Plan Documents Software Document Control Systems, Procedures, Forms and Templates 4
P Why does a SQA (Software Quality Assurance) standard be implemented in our company Software Quality Assurance 4
A Software Tools for managing a QMS (Quality Management System) ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
S Comparing Quality System Software Features Quality Assurance and Compliance Software Tools and Solutions 18
A Suggest Computer Aided Quality Assurance (CAQ) software for Medical Devices Quality Assurance and Compliance Software Tools and Solutions 3
A Quality Software TQS-9000 Family of Products Quality Manager and Management Related Issues 3
R Moving to an Electronic System - Quality System Software Recommendations ISO 13485:2016 - Medical Device Quality Management Systems 5
M Has anyone used "Paradigm 3" software to Control their Quality or Management System? Quality Tools, Improvement and Analysis 2
M Software for our Quality System - IPI Solutions Experiences/Reviews wanted Quality Assurance and Compliance Software Tools and Solutions 3
T Quality Compliance Software to meet ISO 13485 QMS and 21 CFR 820 Requirements Quality Assurance and Compliance Software Tools and Solutions 3
M Auditing Test Management Software such as HP Quality Center (HPQC) Test Tool Internal Auditing 1
D Has anyone used Mango as their Quality Software? Quality Assurance and Compliance Software Tools and Solutions 9
P Software Quality Assurance - Projects developed without a Contract Software Quality Assurance 10
michellemmm Software Product Development Quality Assurance (PDQA) Audit Checklists Software Quality Assurance 12
D TQM (Total Quality Management) Diagram Software Quality Assurance and Compliance Software Tools and Solutions 4
T Software for Quality Management suggestions? Quality Assurance and Compliance Software Tools and Solutions 14
D Switching to a new Quality Management software - Seeking recommendations Quality Assurance and Compliance Software Tools and Solutions 4
Marc What software (if any) do you use to manage your quality system? Quality Assurance and Compliance Software Tools and Solutions 57
C Software Quality Assurance Plan Template Software Quality Assurance 7
R Supplier Quality Investigation - Software and Hardware Issues Supplier Quality Assurance and other Supplier Issues 8
M Cost of Software to Calculate the Cost of Quality Quality Assurance and Compliance Software Tools and Solutions 10
C Comprehensive Quality Management Software (database) Recommendations wanted Quality Tools, Improvement and Analysis 13
M Has anyone used Quality Software from ION called Simple Trak Quality Assurance and Compliance Software Tools and Solutions 7
A From Supplier Quality Assurance to Software Quality Assurance Software Quality Assurance 5
S Software Quality Audit - Internal Audit Checklists Software Quality Assurance 6
S A Sample Quality Manual for Software Industry Software Quality Assurance 2
E Quality Software - Who uses software to manage your Quality System? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
E Quality System Software with TS 16949, Does it good? IATF 16949 - Automotive Quality Systems Standard 1
Sidney Vianna New ISO/IEC standard: quality of IT system / software engineering life cycle process Other ISO and International Standards and European Regulations 1
B ETQ Quality & Compliance Software - Opinions or Reviews wanted Quality Assurance and Compliance Software Tools and Solutions 35
A CDRH Guidance Document - Software bugs submitted in 510(k): Quantity or Quality? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 3
S Information regarding the Software quality systems Software Quality Assurance 3
G Report template for quality process audits of the software product teams Internal Auditing 1
F Seven Quality Tools diagrams and charts software - Excel alternative? Quality Tools, Improvement and Analysis 9
B Quality Training Software Recommendations Training - Internal, External, Online and Distance Learning 1
S Quality Defect Management Software - Discussion and Recommendations Misc. Quality Assurance and Business Systems Related Topics 3
S Internal Quality Audit Software - IQA Dash Board Internal Auditing 1
T QFD (Quality Function Deployment) Software recommendations Quality Assurance and Compliance Software Tools and Solutions 3

Similar threads

Top Bottom