What about running a pareto of the processes or elements with most issues, CAR's vs. OI's. We do issue OI's and follow-up on them in subsequent audits. It won't measure effectiveness of the process directly, but it would possibly show you the most troublesome areas.
I would normally say to do it by element, but I think Doc Control will always be at the top of that list....
What do you think or the pros and cons of this?
You could do something like this, but I'm not certain it tells you much about the audit process, itself. Giving management a set of graphs to look at isn't going to do much good. I also believe that there have been two vitally important components missing from this thread, IMHO:
1) the planning of audits
2) the fact that audits are part of the "Measurement, Analysis and Improvement" section of ISO 9001.
The effectiveness of audits is a function of planning them. If you are going to do something to measure the effectiveness of audits, you're probably going to have to do something to make the planning really effective. Just like any other process, setting a 'goal' for it is important so you can judge progress/performance.
So, on that basis, (plus we already have debated that you can't really set goals around the number of NC's, OFi's etc.), you will have to draw up something which is likely to be understood by management (without too much interpretation).
Why not do the cost analysis? Develop a standard cost for an audit 'day' and look at the results to see what the audit findings 'pay back'. The goal would be to at least break even!
You could also look at how many audits are 'pushed' on a process and how many are requested by management (if they get the reasons why we should do audits, they will want you to audit their process) and set a goal of 40 or 45% (say) this year, 50 - 60% next year and so on. Ultimately, you might get to 95 or more. That's improvement in my book!
The point about the section of the standard that audits is found in, is a clue, IMHO, that the audits are there to report on effectiveness, be a monitoring action and help with improvements.
As a business, the focus should be on improving the customers' perceptions and the main value adding processes as a priority. When your organization has begun to 'flatten out' on improving those two business aspects, then I'd turn the attention to other things like other areas of the standard (audit, calibration, etc. etc.), but not as a priority.
From a business sense it makes no sense for you to be working and reporting on those aspects, when others might need your help......