Sampling strategies/techniques for software QA

#1
Hi,

Are there any published industry standards (or guidelines) on sampling strategies for software QA? A representative use case would be measuring compliance to a code review process; how do I determine how many code review records I need to examine to be able to say that compliance to the review process is x% with a margin of error of y% ? I've generally used a sample size of 10 - 15% (which seems to be widely-used sample size), but I'd like to be able to refer to a standard or guideline that would give this approach more validity.

Thanks in advance!
 
Elsmar Forum Sponsor

yodon

Leader
Super Moderator
#2
I've never heard of sampling for software QA. Everywhere I've been and everywhere I've seen, all data are used.

I'm curious how many data points you have that you feel sampling would be needed?

I'm also curious how you would use the % compliance to code review metric (irrespective of sampling). Will that be used in conjunction with other metrics?
 
#3
It sounds like you're conducting an internal audit for a process issue? Not specifically using sampling to inspect software outputs.

If you're sampling process, I would suggest sticking to the record sampling methods outlined in audit standards, irrespective of whether the record is related to the software domain. Maybe a better question: Is there no way to force conformance to the review process? i.e you're not allowed to merge code without review of X authority person?
 

Tidge

Trusted Information Resource
#4
Specific to code reviews of (product) software against an established coding standard: I have always had each code review make an assessment of every one of all identified deviations from the coding standard.

I have always done this specifically so that we have the raw data for either evaluating either/both of the following:
  1. The technical ability of our (individual, collective) programmers to follow the coding standard
  2. The appropriateness of discrete elements of the coding standard
EDIT: I should add that much of my recent focus has been specifically with (product) software for medical devices. In this field we track all anomalies from the software development process to support the launch of the medical devices.

In practice, I have only ever seen revisions to the coding standard. In my judgement, such revisions have always been in the area of 'pet peeves' rather than any actual improvements in the coding standards. (I always think about the difference between a 'computer scientist' and a 'computer artist'.) For example, we had a coding standard with a strict requirement in the area of "blank spaces vs. TABs" that was cluttering up the anomalies list for code reviews and that criterion eventually got dropped.

If the question is strictly an (internal) audit for assessing "how well are we following our established code review process?", any standard (process) audit process can be followed: scope/objectives, plans, criteria, records (from both source records and systems that those records can feed into like NCR, CAPA, deviations), audit reports.

Depending on the scope and criteria you may need technical resources. The specific effectiveness of Code Reviews themselves can be tricky, because the 'denominator' (for lack of a better word) in establishing study designs can be extremely confusing. Even a simple metric like 'lines of code' (LOC) can muddy the studies. For example: a LOT of lines of code can be generated automatically and would be unlikely to violate 'coding standards' so a sampling plan that includes such code would be highly unlike to expose defective code reviews.
 
Thread starter Similar threads Forum Replies Date
S Strategies for reducing Sampling on incoming bulk material - low volume, ingots Inspection, Prints (Drawings), Testing, Sampling and Related Topics 2
C Statistics/sampling plan for medical device Inspection, Prints (Drawings), Testing, Sampling and Related Topics 10
G Acceptance sampling by variables - Do Minitab use one sided k only? Why? Statistical Analysis Tools, Techniques and SPC 0
Bev D Acceptance Inspection Sampling Plan Calculators Misc. Quality Assurance and Business Systems Related Topics 0
Q Control Plan and the frequency of sampling FMEA and Control Plans 11
F Attribute sampling plan ISO 13485:2016 - Medical Device Quality Management Systems 7
M What is the Risk of Using Obsolete Versions of C=0 & ANSI/ ASQ Z1.4 Sampling Plans? ISO 13485:2016 - Medical Device Quality Management Systems 8
R Customer feedbacks sampling size EU Medical Device Regulations 4
S Mixing criteria and sampling Qualification and Validation (including 21 CFR Part 11) 1
W What would you consider "representative sampling" for PT/ILC? ISO 17025 related Discussions 2
G Calculating Ppk for Design Verification - Variable Sampling Design and Development of Products and Processes 15
J Medical Device Component Change - Testing, Sampling Criteria ISO 14971 - Medical Device Risk Management 3
D Is sampling allowed for MDR Class III products? EU Medical Device Regulations 0
B Inspection of polymer palletes - sampling plan Inspection, Prints (Drawings), Testing, Sampling and Related Topics 0
A Class IIb implantable-TD sampling plan EU Medical Device Regulations 0
M How to Develop a Sampling Plan for Drug Substance? Inspection, Prints (Drawings), Testing, Sampling and Related Topics 0
S Stability sampling for Class 1 medical devices ISO 13485:2016 - Medical Device Quality Management Systems 1
J Internal audit random sampling methodology Internal Auditing 2
D ANSI/ASQC Z1.9 VS MIL-STD-1916 for Continuous Sampling Lean in Manufacturing and Service Industries 2
M Detection rank for sampling vs 100% inspection & 15x vs 400x magnification FMEA and Control Plans 4
J IVD sampling sizes for buyer Reliability Analysis - Predictions, Testing and Standards 0
S Which Sampling Plan(s) Should I Use? Inspection, Prints (Drawings), Testing, Sampling and Related Topics 13
C Sampling - ISO 16269-6 vs ISO 2859 Other ISO and International Standards and European Regulations 0
L Sampling Plan Risks AQL - Acceptable Quality Level 6
A Sampling plan for in-process QC (medical devices) Inspection, Prints (Drawings), Testing, Sampling and Related Topics 13
M Minitab Capability of the Population (no sampling) Using Minitab Software 11
W LTPD, AQL, Ppk and Cpk validation sampling plan table Inspection, Prints (Drawings), Testing, Sampling and Related Topics 0
Q Sampling method for Distributor verification activities EU Medical Device Regulations 1
E Change in control plan - Do I have to do sampling? IATF 16949 - Automotive Quality Systems Standard 1
S Is using ANSI/ASQ Z1.4-2008 the correct sampling plan to determine Pass/Fail of Apparel measurements? AQL - Acceptable Quality Level 8
N Sampling Plan for Internal Audits - ISO 2859 or 3951 - Or Neither? Internal Auditing 6
F Product audit sampling plans IATF 16949 - Automotive Quality Systems Standard 12
R Reduced sampling plan for sterial products APQP and PPAP 0
A ISO 2859-1 - 9.3.3.2 Switching score - Sampling Plans and PPM AQL - Acceptable Quality Level 0
R BS EN 12060:1998 - Zinc and zinc alloys Method of sampling Specifications Manufacturing and Related Processes 4
S High voltage testing - ISO 17025 - 7.2.2 Validation of methods and 7.3 Sampling ISO 17025 related Discussions 3
8 Activity Sampling Exercise Lean in Manufacturing and Service Industries 11
T Defining sampling plan for different AQL AQL - Acceptable Quality Level 3
B AS9100D Sampling Plan AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 2
R Design verification for interim design outputs - sampling rationale ISO 13485:2016 - Medical Device Quality Management Systems 2
Nicole Desouza Sampling plan for a customer who wants AQL 1.0 (per ANSI Z1.4) AQL - Acceptable Quality Level 5
V How to interpret AQL sampling tables AQL - Acceptable Quality Level 5
A AQL sampling bags of parts Inspection, Prints (Drawings), Testing, Sampling and Related Topics 6
C Applying Weibayes to Attribute sampling Reliability Analysis - Predictions, Testing and Standards 0
I Sampling processes - Who must define the AQL level? AQL - Acceptable Quality Level 9
E Sampling for capability studies for variables and attributes Capability, Accuracy and Stability - Processes, Machines, etc. 4
O Statistical justification of sampling size in V&V tests ISO 13485:2016 - Medical Device Quality Management Systems 5
M Sampling Plan for Alumin High Pressure Die Castings Manufacturing and Related Processes 0
E Sampling plan for orthopedic implant - Process Validation Inspection, Prints (Drawings), Testing, Sampling and Related Topics 4
S Inspection/sampling economics Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3

Similar threads

Top Bottom