Let me throw a little gasoline in this dying fire:
Don't forget the requirements from ISO 9001 and TS 16949:
The organization shall determine:
a) requirements specified by the customer, including the requirements for delivery and post-delivery activities,
b) requirements not stated by the customer but necessary for specified or intended use, where known,
c) statutory and regulatory requirements applicable to the product, and
d) any additional requirements considered necessary by the organization.
The standard does require an organization to go beyond the minimum, contractually specified customer requirements. I do agree, however, that if a customer does not inform a supplier that a "Supplier Quality Manual" exists nor invokes it anywhere in the T&C's, the supplier might plea blissful ignorance.
What should a good auditor do, in a situation like this? Besides planning and communicating with the auditee beforehand, the moment an organization resists to the idea of being audited and expected to comply with the supplier manual, the auditor should have called someone at the customer SQA function and, in the presence of the auditee, enquire:
Hi, I am So&so, auditing supplier ABC, on behalf of CB XYZ. The supplier tells me that they have never been informed nor required to comply with this manual. Can you clarify where and how was compliance with this manual flown down to this supplier?
If the customer SQA can not provide a clear location where such expectation was conveyed, then the auditor should not use that manual as a source of auditable requirements.