We (my employer) normally do an on-site periodic audit only of those suppliers that provide us with sterilization services, and/or that do contract/component manufacture of products that have sterile barrier functionality and/or that they sterile barrier package.
We also do site visits by qualified specialist engineers, but not formal on-site audits, for suppliers of sterile barrier packaging materials.
Other suppliers are periodically re-qualified by review of quality records, reference to updated certifications, questionaire responses and the like.
We, like many other companies, were burned by the Pharmaceutical Innovations fiasco...in our case, to a very limited extent because we'd already cut them off due to uneasiness plus the difficulty of working with their owner, but still it was a sobering lesson. We had not done a full audit of them, including review of validations. Part of that was due to their being ISO 13485 certified, and our believing their certificate, and part of it was that what they supplied us was under their name, and we just kit-packed it. And, we had some "shortage of resources, have to prioritize" thinking. We also don't have an in-house deep knowledge of the oddball dry-heat-through-foil sterilization approach they were using, and had been using forever including through previous FDA reviews. Big mistake. We won't do that again.