SBS - The Best Value in QMS software

Independence between the development and testing

Hirvo

Starting to get Involved
#1
We have a developer team of two persons and they review each others code. Then we have an automatic testing procedure (regression test) after every implemented feature. Do you think this is an appropriate level of independence between the development and testing? (Safety Class B cloud based SaMD, agile process)
 
Elsmar Forum Sponsor

yodon

Staff member
Super Moderator
#2
What's driving this question? In Annex C of 62304 is:

This standard makes no recommendation concerning independence of people responsible for one software ACTIVITY (for example VERIFICATION) from those responsible for another (for example design).

That said, independence is, IMO, a very good thing. Developers tend to be pretty myopic. Peer reviews of code is a good idea (assuming someone other than the developer is the reviewer). Scripted (automated) testing, to me, is only as good as the script and doesn't really afford the opportunity for assessing robustness. Developers will tend to do whatever is needed to make the test pass. I *highly* recommend some level of exploratory testing. Developers have a "make it work" mindset and testers have a "how can I break it" mindset. That "tension" (presumably managed correctly) will lead to better software.
 

Hirvo

Starting to get Involved
#3
Yes, I agree independence in testing is a good thing, but as long as it is not a requirement of regulation, it is a business decision. My role is to check what does the standard demand :).
 

Tidge

Trusted Information Resource
#5
I want to provide some conversation about the point about 'is independence in testing necessary'. As @yodon writes, 62304 requires system level testing (for all software system safety classifications) to guarantee (software) requirements are met. This level of testing requires pre-approved test methods and pass/fail criteria (5.7.1, also 5.7.4) so it isn't as if you can have testers simply hunt-and-peck to make an assessment of if the requirements are met.

There is a further point that needs to be considered: An attempt by external 3rd parties to make a general assessment of design verification (such as under 13485 7.3.6, or 21 CR 820.30(f)) for software development can quickly lead a developer into trouble if the evidence/tests performed don't explicitly align with the software requirements. It is (in my experience) both rare and unnecessarily constricting to write software requirements that specifically identify implementation details that will ultimately be tested; yet this is a common expectation for the (uninformed) 3rd-party auditor.

I have witnessed external auditors prepare to 'lower the boom' for 'improper testing' for cases where all of the test methods were rigorously vetted and approved (per 62304 5.7.1 (a), (b)) simply because the auditor couldn't hold up a specific test (with predetermined acceptance criteria) and make a 1-to-1 match against a correspondingly worded software requirement. My experiences were quite painful: We literally had to drag an (only-partially) curious auditor (*1) through the complete 62304 development process AND provide them the equivalent of a "survey of modern technology" to get them to realize that all was in order. I wouldn't recommend trying to endure a similar experience if the testing only consisted of code reviews.

(*1) Frankly: Most auditors, no matter how well trained to audit a process against a standard, can recognize when they have become mis-calibrated (about how the process maps to the standard) when they see a specific process output that doesn't look like what they expect. Software is very susceptible to this sort of 'mis-calibration' (not just by 3rd party reviewers, but also developers themselves) because of the large universe of potentially acceptable design solutions and the well-loved (but potentially dangerous) concept of a 'trace matrix'. The observed danger with the XLS trace matrix is the attitude "ok, I'll pick a cell in a column at random, scan several rows across... now show me how those two cells relate."
 

John Broomfield

Staff member
Super Moderator
#6
Surely each process has objectives and the auditor ascertains these objectives from members of the process team instead of auditing against their personal (auditor) expectations?

I’ve never seen an audit objective that refers to my expectations as the auditor.
 

Tidge

Trusted Information Resource
#7
Allow me to write this:

Software is a much younger field of engineering than most other fields, and the 'barrier to entry' of exposure to software is much lower than other engineering fields. Auditors (as well as other 3rd-party reviewers that may not have a precise audit plan) have demonstrated all sorts of bad (my term for misaligned expectations described above) behavior that wouldn't be tolerated in most other fields of engineering.

I've had almost all 3rd party groups (acting officially as an auditor or inspector) ask to see source code, but I've only once had a 3rd party reviewer actually want to witness a piece of manufacturing equipment while reviewing process validations. Similarly, outside of NRTLs during testing, I've never seen a third party want to see a specific design output or physical instance of a design output (such as a battery, or a PCBA). I have had 3rd party folks from NRTLs witness manufacturing test processes and certain specific outputs such as labels... but those instances have always been per their established plan.

I have a not-yet-disproven hypothesis is that some of the folks I've encountered are either frustrated programmers or have some sort of internal self-assessment that they could have been programmers. From my past, the gentlemen who wanted to see the manufacturing equipment used to work with that sort of EQ in his past and just wanted to see our setup.

There is a subset of 3rd parties who will say (during an audit) things like "software is black magic" or "I don't get software". The folks that say stuff like this don't realize how unprofessional those statements are and how the statements are revealing of(at best) how they are the wrong person for the job or (at worst) they have all sorts of inappropriate biases. I wouldn't want to have a medical doctor or a physician's assistant that casually dropped a gem like "I just don't get women" or "pain management is black magic".
 
Thread starter Similar threads Forum Replies Date
T ISO 13485 - 5.5.1 Responsibility and authority - Small Company Independence ISO 13485:2016 - Medical Device Quality Management Systems 13
C Independence of DQA (Design Quality Assurance) 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 1
J ISO 17020 concerns of independence & impartiality of services Other ISO and International Standards and European Regulations 1
lyobovnik Matters being audited, independence, participation and direct responsibility 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 3
D How to ensure independence and authority necessary to perform QM tasks ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 1
A ISO 13485 - Internal Auditor Independence and Process Owners ISO 13485:2016 - Medical Device Quality Management Systems 3
Ronen E Independence on Notified Bodies, inter alia of Manufacturers they Assess EU Medical Device Regulations 7
P Happy Independence Day - India Coffee Break and Water Cooler Discussions 1
R On Auditing Internal Audit Process - How Independence can be Established Internal Auditing 4
Howard Atkins Happy Indian Independence Day Covegratulations 3
Marc Thomas Jefferson, co-author of the United States Declaration of Independence Coffee Break and Water Cooler Discussions 1
T Quality Staff - Independence (ISO 13485 Clause 5.5.1) ISO 13485:2016 - Medical Device Quality Management Systems 1
Ajit Basrur India celebrates 65th Independence Day World News 6
sagai Organizational Independence of the Quality Division 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 10
A Mexico is Celebrating 200 years of Independence and 100 of revolution Coffee Break and Water Cooler Discussions 8
Stijloor Charlotte, NC - Our Independence is in trouble... Funny Stuff - Jokes and Humour 17
somashekar Independence Day - India - August 15th Coffee Break and Water Cooler Discussions 8
A Internal auditors independence & integrity Internal Auditing 9
M Internal Auditing Independence Rule - Can someone provide a good interpretation Internal Auditing 20
L Registrar Customer Learning Session-Independence Ohio Coffee Break and Water Cooler Discussions 2
W Happy Independence day America Coffee Break and Water Cooler Discussions 4
E Being IQA Team Leader and auditee - Auditor Independence General Auditing Discussions 9
G 4th OF JULY - USA's Independence Day Coffee Break and Water Cooler Discussions 15
Raffy Internal Auditing - Showing independence of process being audited Internal Auditing 4
P Internal auditor independence Internal Auditing 4
A What is different between Field Safety Notice and Advisory Notice in U.S and EU? Document Control Systems, Procedures, Forms and Templates 6
R Whats different between these position CE Marking (Conformité Européene) / CB Scheme 10
G What's the difference between 1.32 and 1.33 Cpk? Capability, Accuracy and Stability - Processes, Machines, etc. 18
U How to pick between ANSI Z1.4 vs. ISO2859-1 for AQL Inspection? What are the major differences? AQL - Acceptable Quality Level 4
Moumen H Variations between ASTM A29 Standard for steel bars and Mill test certificates specs Manufacturing and Related Processes 1
N R&R for Differences between 2 measurements Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 4
S Distinction between a critical supplier and a Virtual manufacturer EU Medical Device Regulations 2
I Сorrespondence between hazards and risks ISO 14971 - Medical Device Risk Management 2
W What is the difference between TYPE B and TYPE BF? IEC 60601 - Medical Electrical Equipment Safety Standards Series 2
T The difference between ISO 14644-3:2005 and ISO 14644:2019 Other Medical Device Related Standards 2
Q Terminal Lugs sizes - Difference between 225/24 vs. 275/24 lugs Manufacturing and Related Processes 2
T Relationship between ISO 9001 and ISO – IEC BS EN 870079- 34 2020 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
M Difference between "Production Trial Run" and "Run at Rate" IATF 16949 - Automotive Quality Systems Standard 8
Ron Rompen Surface Finish Correlation between Ra, Rz and Tp (bearing surface ratio) General Measurement Device and Calibration Topics 3
L MRA between EU and Switzerland - 1/2021 EU Medical Device Regulations 2
D Difference between Test Method Validation and Gage R&R Qualification and Validation (including 21 CFR Part 11) 18
K Joint approval between OEM and Manufacturer on Design Documents ISO 13485:2016 - Medical Device Quality Management Systems 4
C Is my software an accessory? Telecommunication between HCP and patients EU Medical Device Regulations 10
K Verify Software Architecture - supporting interfaces between items IEC 62304 - Medical Device Software Life Cycle Processes 2
E ASTM F2118 - Fatigue testing of bone cement - Changes between the 2003 and the 2014? Other Medical Device Related Standards 1
A What is the difference between Design Process, Process Design and Design Control? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 2
R What's the major difference between Green Belt and Black Belt in term of training and project Six Sigma 3
DuncanGibbons How is the arrangement between Design and Production organisation envisaged? EASA and JAA Aviation Standards and Requirements 4
M Risk Analysis Flow - Confusion between ISO 14971 and IEC 62304 IEC 62304 - Medical Device Software Life Cycle Processes 8
T Difference between a subcontractor and a supplier ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 21

Similar threads

Top Bottom