Cloud-based SaMD Validation

[RedDevil_AK]

Hi Cove-dwellers,

I (no software development experience) am working at a startup with a Class II (moderate LoC) SaMD designed to run on a cloud-based service hosted in AWS. The platform code provides an SDK package for device software to enable the use of an API to launch docker containers with algorithms with the use of AWS resources within AWS, track the progress of their execution and store information about such jobs in the database in order to allow monitoring and reporting.

I was hoping to get clarity on a couple of questions regarding V&V for this platform -

a. For AWS validation, what else would be required outside of their validation documentation on Part 11 compliance and data confidentiality and integrity? The device uses the platform to make API calls to AWS, so would validation consist of making dummy API calls and making sure that input/output is repeatable?

b. For the platform itself, would unit tests comprising of design and peer review, testing written into the code (Unit tests check that all the code paths work as expected (e.g. perform specific activities depending on the input and intermediate results, fail quietly with or without the log message if it is acceptable or raise exception and propagate it up the stack to alarm about the failure, etc.).) and functional testing (covering integration + system tests) be enough?

Thanks! Appreciate all responses.

 

[yodon]

From the description, I'm having a hard time seeing how this is a medical device, but let's assume so.

a. For AWS validation, what else would be required outside of their validation documentation on Part 11 compliance and data confidentiality and integrity? The device uses the platform to make API calls to AWS, so would validation consist of making dummy API calls and making sure that input/output is repeatable?

Validation shouldn't be just about testing. How will you know that the platform is stable and changes are rolled out in a controlled manner (allowing you time to ensure your software still works on it)? Not sure how Part 11 fits in here. Certainly you need to ensure data confidentiality and integrity (and retention). This pulls in backup and recovery as well.

b. For the platform itself, would unit tests comprising of design and peer review, testing written into the code (Unit tests check that all the code paths work as expected (e.g. perform specific activities depending on the input and intermediate results, fail quietly with or without the log message if it is acceptable or raise exception and propagate it up the stack to alarm about the failure, etc.).) and functional testing (covering integration + system tests) be enough?

62304 uses the concept "unit verification and acceptance." One of the means to do unit verification and acceptance is unit testing (and there's a little baggage that goes along with that). You define what the means for unit verification and acceptance. We do rely heavily on static analysis and code inspections for our unit verification and acceptance. We generally only use unit testing if there are some safety considerations in the unit. Anyway, your SW Dev Plan needs to define what you WILL do for unit verification and acceptance. What you describe should be acceptable. (Side note: in the last few years, FDA has taken a keen interest in static analysis - would definitely recommend that.)

Most likely, you can cover "Software System Testing" (62304) / Software Requirements Verification through your functional testing. There may be additional considerations regarding overall system testing (may not be anything else). Risk analysis may drive a need for additional efforts to demonstrate effectiveness. If you're complying with IEC 62366, you may have some additional usability testing.

And certainly don't forget cybersecurity. That's pretty much front-and-center these days when it comes to reviews.

 

[RedDevil_AK]

Thanks yodon!

From the description, I'm having a hard time seeing how this is a medical device, but let's assume so.

should have been more detailed - the platform is used to host the software that does the medical device part (image segmentation)

How will you know that the platform is stable and changes are rolled out in a controlled manner (allowing you time to ensure your software still works on it)?

while I understand what the intent is, I am unable to understand how it pertains to AWS as the third-party tool. I'm sorry, could you please rephrase this?

 

[yodon]

while I understand what the intent is, I am unable to understand how it pertains to AWS as the third-party tool. I'm sorry, could you please rephrase this?

I'll try. You are relying on AWS for certain things. They may change those things. They may deprecate or change an interface. Will these changes pull the rug out from under you or will there be a transition period to allow you to adapt?

 

[erkan]

I was just about to post a related question, so perhaps I will just add it here.

Regarding IEC 62304 8.1 Configuration Identification. What types of Configuration Items must one record for software using a cloud like AWS?

 

[yodon]

I don't expect there would be any CIs for you. You don't have any control over what they do. I presume there's an API spec that you can cite but, again, not a CI for you.

 

[erkan]

Thanks, Yodon, perhaps I can re-phrase. We are building a cloud-based SaMD, which uses AWS. Won't my S3 or other infrastructure I use be considered CIs?

 

[ECHO]

Let me try to clear a few things up a bit.

The intention of IEC 62304 8.1 is to track the changes in the software, including libraries and the environment the software runs in. Therefore, when your software quality engineer says, "Engineering Build V2.1.113 passed verification", as a team, you know exactly how to recreate a deterministic system.

The statement above implies that the system will use a fixed version of libraries and environment. Now, if you are using AWS, you don't really have control over what Amazon does to their backend.

The changes AWS would/could make doesn't really impact you if you are just using S3 to simply store an image. But if you are using EC2 to do very time sensitive calculations, how do you know that the change AWS made isn't going to add a microsecond to your calculation and throw off the entire functionality?

Below is an example of what you can do.
Automate your testing so you can catch potential problems.
When you run these automated tests, make sure to record your SW version number (or just git hash for section 8.1.1), spit out the dependent libraries (title, manufacturer and version for section 8.1.2) and similar info for the environment. Save this log (for section 8.1.3).

The frequency of the tests and what you do when you find an issue will depend how you implemented the rest of IEC 62304.

 

[yodon]

Won't my S3 or other infrastructure I use be considered CIs?

I had to chat with our software / cloud expert on this. According to him, the answer is "sort of." What we apparently do is to capture all this in build / setup instructions. So in that respect, yes, that would be a CI.