Not necessarily, Section 820.50 (Purchasing controls) states
Each manufacturer shall establish and maintain procedures to ensure that all purchased or otherwise received product and services conform to specified requirements.
(a)Evaluation of suppliers, contractors, and consultants. Each manufacturer shall establish and maintain the requirements, including quality requirements, that must be met by suppliers, contractors, and consultants. Each manufacturer shall:
(1) Evaluate and select potential suppliers, contractors, and consultants on the basis of their ability to meet specified requirements, including quality requirements. The evaluation shall be documented.
(2) Define the type and extent of control to be exercised over the product, services, suppliers, contractors, and consultants, based on the evaluation results.
(3) Establish and maintain records of acceptable suppliers, contractors, and consultants
So, you set up your inspection to ensure the supplier is supplying the correct parts and you have defined this to be the extent of your controls (and can justify based on risk), that is all you have to do.
We have many suppliers that we do not “audit”, but we evaluate and have controls set up to satisfy the requirement. It's all about making the system work for your company.