I'm definitely aware of FDA inspectors reviewing each of the letter-steps in 21CFR 820.30...QMS procedures, then how the plan, inputs, outputs, reviews, verification/validation and transfer are consistent with those procedures. The QSIT manual requires such a stepwise inspection procedure...it's right in the inspection flowchart. An inspector that only looked at outputs and the DHF would not be following procedure.
No doubts here. It's just that this review is normally a review of the resulting documentation / evidence, not of the actual design control processes as they happen; the plan, inputs, outputs, reviews, verification/validation and transfer are all required to produce documentation / evidence. Therefore the review can technically occur wherever the evidence is present, and staff are available to explain / answer questions.
That pretty much has to be done at the site where the design work is done, and that site pretty much has to have a cGMP QMS.
It's obvious that this would be the preferred set-up, but I would say it can still be done wherever the documentation is available (either the specification development service provider site, or the finished device manufacturer site - provided that they made provisions for that). Regarding the second half of your statement, I don't think that part 820 mandates components suppliers to have a part-820-compliant QMS. As I stated earlier, relevant requirements (including relevant portions of the QSR) could be enforced by the finished device manufacturer - for their own compliance - but I don't see how part 820 directly requires anything from an entity that does not deal with a complete device (or with something that can be perceived as such).