Agree with advice so far regarding re-assessing the need for audit.
However, if you determine it is appropriate (perhaps because it's a critical supplier, or because of the very isolated lots), I'd suggest revising your supplier approval criteria to associate the audits with orders. Perhaps some criteria like "a supplier audit shall be performed within X months of placing an order".
This way, audits are associated with purchases (which will mitigate supplier reluctance - supplier has something to gain), and you can confirm that all the quality controls from the last order are still in place (e.g. you can verify that they haven't changed any suppliers, equipment, processes in the interim).