the other cautions I have are that different operators can effect the positioning of the part in the fixture. if using a CMM, this positional difference can lead to 'measurement error' (the tip hits the part at different locations and/or angles or the shank hits before the tip does...) If you have within piece variation, the positional change can have a substantial effect on measurement error. Remember you are not testing the CMM, you are testing the whole system.
the same holds for any ATE from electrical board testing to NDE and destruct testing of physical properties to vision systems to functional tests of any kind
I would recommend a standard multiple part, multiple readings, multiple operators design.