I tend to suggest to my clients that they consider how existing systems can be used to demonstrate training effectivity before they design all new systems. For example, I use a method whereby reject issues are reviewed to determine the cause (operator, equipment, management oversight, customer). Any issue that is identified as "operator" suggests an issue involving poor instructions or a training issue. By trending the number of "operator" issues, the effectivity of training can be demonstrated. I also suggest that the company include a training review during personnel reviews (it's surprising how many companies overlook this prime place for measuring training effectiveness!) Essentially, the supervisor evaluates how well the employee is trained in evaluating performance. It's also a prime opportunity to determine training needs.
Beyond this, there may be places to use on the job testing, especially with higher level jobs. For example, I've used a final operational test as a measure to make an operator of a heat treat furnace full class. A supervisor witnessed the operator performing loading and unloading of the equipment, then had the operator perform an emergency shut down of the equipment, then had them start it up again. All this was documented on a checklist that went into their training file.
I don't advocate a performance test for everything, but they do have their place in many companies.