IEC 62304 - evaluation of integration and system testing

[PaulG]

In IEC 62304:2006/AMD1:2015, the updated clause 5.6.5 simply states "The manufacturer shall evaluate the integration test procedures for adequacy." Based on the standard's definition of "evaluation" (a systematic determination of the extent to which an entity meets its specified criteria) does this mean that each test procedure should be run with sample cases to demonstrate the outcomes are as expected? Or is a qualitative assessment/rationale for the test sufficient?

Similarly, 5.7.4 states that "The manufacturer shall evaluate the adequacy of verification strategies and test procedures." I'd be interested to hear how people have interpreted and satisfied these requirements in practice.

 

[kreid]

I think this can be covered by evidence that the test procedures have been reviewed and approved before running the tests.
I have seen several instances where this is not done and reviews only take place on the completed test results document.

 

[yodon]

I mostly agree with @kreid but we don't typically approve (in the traditional sense) integration tests. We do review them "for adequacy" and document the review (with a conclusion that they are adequate... with justification). We also do the same for software system test protocols although they do go through a formal review / approval process). You could argue that the review process covers requirement for appropriateness but we expand the documented review to assert that the protocol will drive capture of the required test record contents (5.7.5).

 

[PaulG]

OK, this makes sense. We are planning to capture the testing in a test plan document that will be reviewed and approved beforehand, so I think this will cover it. Thanks for everyone's feedback.

 

[Tidge]

In my experience, less mature (i.e. reactionary) ME software development teams struggle with the concept of integration testing more than any other element of 62304. If a team struggles with this concept, it can look like they aren't living up to this particular part of the standard. The folksy advice I offer about integration testing is that it shouldn't appear to be nearly indistinguishable from either unit or system-level testing, and that the purpose of the testing is to identify defects that aren't otherwise found at the unit level before they become issues at the system level.

We are planning to capture the testing in a test plan document that will be reviewed and approved beforehand, so I think this will cover it.

I have found that a formal review and approval of the integration tests prior to execution (as suggested by kreid) to both satisfy the OP's concern and leads to more useful testing, at all levels. I typically don't expect test plans to go into the level of detail necessary to demonstrate adequacy of the specific tests at the discrete levels. For example, if the ME software was software safety class C: the test plan may be as simple as identifying the need for unit, integration, and system level testing of the software, and then describing the general approaches to be used at the different levels. This would be sufficient to begin the planning process of dedicating resources, etc. but would not require revision when issues begin to crop up in more specified detail test protocols.

If the software development plan which governs a specific software package (with a previously approved, dedicated test plan) has prescribed which tests are to be executed during integration and has predetermined the appropriateness of those tests, then I feel like the development team wouldn't have to revisit those formally prescribed tests. My experience with software development and testing is that it is often ad hoc and/or the development team bristles at "being told the way to do things", so the formal review and approval of test protocols is usually necessary.