From an FDA perspective:
I assume that you'll be doing further manufacturing on the product/material that your customer is having their supplier ship directly to you.
If your relationship with your customer is as a Contract Manufacturer, your contract with them should clearly state that status, and that they are either the Specifications Developer, or the Manufacturer if they do further operations on the product before it's marketable.
The Specifications Developer or Manufacturer has primary regulatory responsibility for the final product. A Contract Manufacturer nonetheless has co-responsibility for all aspects of the final product that are related to those elements of its production over which they have substantial control.
I would think that an FDA inspector under QSIT procedures readily would conclude that, if the supplier ships directly to you, you have full responsibility for Incoming Inspection and for verification of purchasing conformance, because you have substantial control over received goods and your customer is not physically present...unless they have personnel at your facility, or your contractural relationship calls for you to notify them to send such personnel, to perform those responsibilities whenever something arrives that they purchased.
As a Contract Manufacturer, it's your responsibility under Contract Review to ensure that your contract with your customer adequately defines all regulatory aspects of your relationship. Your customer is responsible for offering you a contract that adequately defines regulatory responsibilities, and for controlling your actions. They're really dropping the ball if their regulatory people aren't discussing with you what you have to do to make it possible for them to meet FDA requirements.