I also do not recommend outsourcing to a contract manufacturer that is not ISO 13485 certified.
I think this option may appeal to startups worried about their burn rate, because these contract manufacturers may bid substantially lower than ISO 13485 certified contract manufacturers, if they do not appreciate the burdens that meeting ISO 13485 requirements will place on them. (And, as Shimon has noted, they will have to meet these requirements, whether they obtain certification or not.) Or they might appreciate them, but realize that the startup does not, and they can bring in the business with a low bid up front and leave the startup to bear the brunt later.
Since the contract manufacturer is not ISO 13485 certified, the device company must create an agreement that essentially replicates ISO 13485, or the contract manufacturer will not be obligated to meet these requirements. Then, when the findings start rolling in, it can say (often correctly) that the work it will have to do to address the findings is outside the scope of the original agreement, and therefore must be done at an additional cost. So, in the end, the device company incurs the same costs but had to go through a lot more headaches than if it had gone with an ISO 13485 certified contract manufacturer in the first place.
Beyond that, if it is a one-device startup, I don't think it makes any sense for a contract manufacturer to take on the infrastructure burden equivalent to an ISO 13485 certified manufacturer in order to manufacture a single medical device. Under this scenario, the contract manufacturer should actually have to charge the startup more, not less, than an ISO 13485 certified contract manufacture that specializes in medical devices. That's because the latter can spread its ISO 13485-related infrastructure costs across its numerous medical device clients, where the former must include all of these costs in what it charges the one medical device client to manufacture its medical device.
When everyone reaches the point that all of the above has become painfully obvious, I don't think either the device startup nor the contract manufacturer are likely to be very happy with how things turned out.