Generate test boards with known (manufactured) defects, then tune the machine to identify the defects - use IPC-A-610 as a guideline basically your machine should agree with an inspector
Calibrate regularly against your test boards, maintaining the boards in a vacuum sealed ESD bag with a sachet of dessicant to preserve the solder joints whenever they are not in use.
Taking this to the Nth degree, you could consider having a master board for each type of PCB you manufacture, which can be run through as a first and last validation test - if you are using expensive multilayers with blind/buried vias etc, then talk your PCB manufacturer they should be able to provide you with a simple non-functional board which can be used for this purpose - for expensive IC's, you can likewise obtain dummy components for placement testing which can be used for the AOI validation.
Basically, determine your validation critera, then cut your cloth to suit.
Also bear in mind that AOI equipment suppliers should be providing some sort of calibration boards which can be used, but those I have seem tend to be pretty basic - what you use should ideally be representative of your product.
