You are going to make business decisions based on the validation results, isn't it? Obviously prototypes have a critical function as part of the product design validation. So, from a risk mitigation perspective, why wouldn't you want to ensure that you are getting parts with the adequate level of quality, so you can trust the results of the prototype testing?
Without a doubt, the type, nature and extent of supplier qualification for this type of products can (and likely should) be very different from the qualification process used for volume production parts, but, in my opinion, irrespective of what 13485:2016 says, from a business perspective, yes, I would qualify such suppliers.