Sampling strategies/techniques for software QA

#1
Hi,

Are there any published industry standards (or guidelines) on sampling strategies for software QA? A representative use case would be measuring compliance to a code review process; how do I determine how many code review records I need to examine to be able to say that compliance to the review process is x% with a margin of error of y% ? I've generally used a sample size of 10 - 15% (which seems to be widely-used sample size), but I'd like to be able to refer to a standard or guideline that would give this approach more validity.

Thanks in advance!
 
Elsmar Forum Sponsor

yodon

Staff member
Super Moderator
#2
I've never heard of sampling for software QA. Everywhere I've been and everywhere I've seen, all data are used.

I'm curious how many data points you have that you feel sampling would be needed?

I'm also curious how you would use the % compliance to code review metric (irrespective of sampling). Will that be used in conjunction with other metrics?
 
#3
It sounds like you're conducting an internal audit for a process issue? Not specifically using sampling to inspect software outputs.

If you're sampling process, I would suggest sticking to the record sampling methods outlined in audit standards, irrespective of whether the record is related to the software domain. Maybe a better question: Is there no way to force conformance to the review process? i.e you're not allowed to merge code without review of X authority person?
 

Tidge

Trusted Information Resource
#4
Specific to code reviews of (product) software against an established coding standard: I have always had each code review make an assessment of every one of all identified deviations from the coding standard.

I have always done this specifically so that we have the raw data for either evaluating either/both of the following:
  1. The technical ability of our (individual, collective) programmers to follow the coding standard
  2. The appropriateness of discrete elements of the coding standard
EDIT: I should add that much of my recent focus has been specifically with (product) software for medical devices. In this field we track all anomalies from the software development process to support the launch of the medical devices.

In practice, I have only ever seen revisions to the coding standard. In my judgement, such revisions have always been in the area of 'pet peeves' rather than any actual improvements in the coding standards. (I always think about the difference between a 'computer scientist' and a 'computer artist'.) For example, we had a coding standard with a strict requirement in the area of "blank spaces vs. TABs" that was cluttering up the anomalies list for code reviews and that criterion eventually got dropped.

If the question is strictly an (internal) audit for assessing "how well are we following our established code review process?", any standard (process) audit process can be followed: scope/objectives, plans, criteria, records (from both source records and systems that those records can feed into like NCR, CAPA, deviations), audit reports.

Depending on the scope and criteria you may need technical resources. The specific effectiveness of Code Reviews themselves can be tricky, because the 'denominator' (for lack of a better word) in establishing study designs can be extremely confusing. Even a simple metric like 'lines of code' (LOC) can muddy the studies. For example: a LOT of lines of code can be generated automatically and would be unlikely to violate 'coding standards' so a sampling plan that includes such code would be highly unlike to expose defective code reviews.
 
Thread starter Similar threads Forum Replies Date
S Strategies for reducing Sampling on incoming bulk material - low volume, ingots Inspection, Prints (Drawings), Testing, Sampling and Related Topics 2
A Class IIb implantable-TD sampling plan EU Medical Device Regulations 0
M How to Develop a Sampling Plan for Drug Substance? Inspection, Prints (Drawings), Testing, Sampling and Related Topics 0
S Stability sampling for Class 1 medical devices ISO 13485:2016 - Medical Device Quality Management Systems 1
J Internal audit random sampling methodology Internal Auditing 2
D ANSI/ASQC Z1.9 VS MIL-STD-1916 for Continuous Sampling Lean in Manufacturing and Service Industries 2
M Detection rank for sampling vs 100% inspection & 15x vs 400x magnification FMEA and Control Plans 4
J IVD sampling sizes for buyer Reliability Analysis - Predictions, Testing and Standards 0
S Which Sampling Plan(s) Should I Use? Inspection, Prints (Drawings), Testing, Sampling and Related Topics 13
C Sampling - ISO 16269-6 vs ISO 2859 Other ISO and International Standards and European Regulations 0
L Sampling Plan Risks AQL - Acceptable Quality Level 6
A Sampling plan for in-process QC (medical devices) Inspection, Prints (Drawings), Testing, Sampling and Related Topics 13
M Minitab Capability of the Population (no sampling) Using Minitab Software 11
W LTPD, AQL, Ppk and Cpk validation sampling plan table Inspection, Prints (Drawings), Testing, Sampling and Related Topics 0
Q Sampling method for Distributor verification activities EU Medical Device Regulations 1
E Change in control plan - Do I have to do sampling? IATF 16949 - Automotive Quality Systems Standard 1
S Is using ANSI/ASQ Z1.4-2008 the correct sampling plan to determine Pass/Fail of Apparel measurements? AQL - Acceptable Quality Level 4
N Sampling Plan for Internal Audits - ISO 2859 or 3951 - Or Neither? Internal Auditing 6
F Product audit sampling plans IATF 16949 - Automotive Quality Systems Standard 12
R Reduced sampling plan for sterial products APQP and PPAP 0
A ISO 2859-1 - 9.3.3.2 Switching score - Sampling Plans and PPM AQL - Acceptable Quality Level 0
R BS EN 12060:1998 - Zinc and zinc alloys Method of sampling Specifications Manufacturing and Related Processes 4
S High voltage testing - ISO 17025 - 7.2.2 Validation of methods and 7.3 Sampling ISO 17025 related Discussions 3
8 Activity Sampling Exercise Lean in Manufacturing and Service Industries 11
T Defining sampling plan for different AQL AQL - Acceptable Quality Level 3
B AS9100D Sampling Plan AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 2
R Design verification for interim design outputs - sampling rationale ISO 13485:2016 - Medical Device Quality Management Systems 2
Nicole Desouza Sampling plan for a customer who wants AQL 1.0 (per ANSI Z1.4) AQL - Acceptable Quality Level 5
V How to interpret AQL sampling tables AQL - Acceptable Quality Level 5
A AQL sampling bags of parts Inspection, Prints (Drawings), Testing, Sampling and Related Topics 5
C Applying Weibayes to Attribute sampling Reliability Analysis - Predictions, Testing and Standards 0
I Sampling processes - Who must define the AQL level? AQL - Acceptable Quality Level 9
E Sampling for capability studies for variables and attributes Capability, Accuracy and Stability - Processes, Machines, etc. 4
O Statistical justification of sampling size in V&V tests ISO 13485:2016 - Medical Device Quality Management Systems 5
M Sampling Plan for Alumin High Pressure Die Castings Manufacturing and Related Processes 0
E Sampling plan for orthopedic implant - Process Validation Inspection, Prints (Drawings), Testing, Sampling and Related Topics 4
S Inspection/sampling economics Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
S How to make a Sampling Plan - Conveyor belts Inspection, Prints (Drawings), Testing, Sampling and Related Topics 4
V Statistical basis and justification while comparing / changing sampling plans Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 11
J AQL for Effectiveness Check - What AQL tiers to use on this type of sampling? Nonconformance and Corrective Action 9
N The question asks about sampling plans and AQL Reliability Analysis - Predictions, Testing and Standards 5
S Help develop sampling plan for prototype samples (medical devices) Statistical Analysis Tools, Techniques and SPC 7
A ISO 2859 Single Sampling - Clarification regarding the sampling table Inspection, Prints (Drawings), Testing, Sampling and Related Topics 4
C Are zero defect sampling plans relevant to AS9120B Distributor AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 4
R DCMA Sampling of CSI Parts AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 4
D Sampling method that is representative of the devices supplied by that distributor EU Medical Device Regulations 5
T ISO 13485: 2016 Internal Audit - Is sampling on projects allowed? ISO 13485:2016 - Medical Device Quality Management Systems 6
M Zero Defect Sampling Plan Standards Inspection, Prints (Drawings), Testing, Sampling and Related Topics 4
L ISO 2859-2 LQL sampling plan linked to %confidence/%reliability Inspection, Prints (Drawings), Testing, Sampling and Related Topics 2
K Sampling Plan for transparent tubular plastic parts from 32 different cavities Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3

Similar threads

Top Bottom