First, you need to identify what makes a "qualified" auditor. You might consider some form of grandfathering to get you started. Of course, CQA and/or Lead Auditor training should not be discounted.
Now that you have "qualified" auditors, the mentoring is a must. Define your minimum expectations, for example assist a lead auditor on a minimum of two audits, then lead the next with the lead auditor assisting. The lead auditor observes and provides go/no-go feedback.
How will you measure "effectiveness"? If it's by number of audits accomplished, you risk hasty and poor audits. If it's by number of findings, God help the auditee. If it's by the effectiveness of the corrective actions to a given audit finding, then are you judging the auditor or the auditee? As you can see, I am having difficulty with this one. I prefer to judge performance to an agreed-to schedule, including all phases of the audit process. This includes notifications, compilation of the audit team, preparation, opening meeting, adherence to schedule to perform the audit, applicability of findings to cited standard, closing meeting, timeliness of audit report, follow-up on corrective actions or lack thereof, through to closure of the audit and corrective action.
The benchmarking is a worthy goal. I have not had much success in trading auditors. Perhaps you will be more fortunate.
My "qualified" auditors were given two years to become CQAs or loose their status. (This is only effective for your direct reports.) (It should also have some financial consequences or rewards.)
Rewards? See the above. And, for auditors from outside the quality discipline, it provides an opportunity to see more of the company and other areas, on company time. Besides, if someone from another area gets to come in and audit you, wouldn't you like an opportunity to return the favor? Keep graphics, such as % of audits accomplished on schedule. Give public kudos for those at or above a target percentage (ie 95%).
As for every auditor presenting their results in front of the executive staff, is that a cost-offective way to do it? The executive staff is not free. The auditors are not free. Unless this is specifically what the executive staff wants, a more cost-effective method is to provide an executive summary. In fact, that's more of what a registrar would expect to see when judging 4.1.
There's no right nor wrong here. Just some ideas for you to consider. Good luck.