C
Re: Measurables for Document Control and Internal Audits
Hi Covers,
I've been following this thread with much interest, and, after careful thought, here is my pennyworth. If I may, I'd like to address the comments made by some of you and then suggest some of my own:
There were some very good examples of audit metrics, but I disagree with the one to do with the auditor finding problems that may become huge issues. This may, of course, be an outcome of the audit, but this seems to guide the auditor to only look for the big fish. What if there are lots of minor errors that suggest the process is not in control. Should these be ignored? My view is that an audit is intended to look at the effectiveness of systems and processes and is based on a sample, and all bad AND good findings are passed to management for action. End of story. There should not be any influence on the auditor, unless there is a for cause audit.
An audit of the effectiveness of actions is good to cover during an audit, but can it be a metric? The only one I can think of is, has the effectiveness of all actions has been audited? Some auditors don't look at the results of previous audits, and some orgs have the effectiveness of actions built into the CAPA system, so it's not the responsibility of the auditor?
The time to closure is a metric but not one for the auditor surely? Maybe one for the auditees management?
The number of opportunities for improvement and the potential number of NCRs surely depends on the process being audited and not a metric for the auditor? What if the process is so good you cannot find fault? So you say in your audit report, based on my audit sample, the process was great. I
I think we need to define the scope of the audit process before we apply metrics to it. For example, in some organisations, the audit process is separate, but feeds into, the CAPA process. In small orgs, like the one I work for, the auditor does the audit, does the audit admin, seeks CAPA, and tracks and trends action plan completion. Also, CAPA plans are usually reviewed and approved by Quality so we have an opportunity to review effectiveness and cost saving benefits of the action plans, but this may not be the responsibility of the auditor. So I would create metrics just for performing and reporting audits.
I would suggest the following metrics may be more appropriate to measure the effectiveness of the audit process:
The real value of the audit process is two-fold:
1. Designing your audit plan/checklist/strategy cleverly so that your sample encompasses the effectiveness of the whole system or process and
2. Performing the audit with the intention of identifying the opportunities for improvement and reporting these to management.

Hi Covers,
I've been following this thread with much interest, and, after careful thought, here is my pennyworth. If I may, I'd like to address the comments made by some of you and then suggest some of my own:
There were some very good examples of audit metrics, but I disagree with the one to do with the auditor finding problems that may become huge issues. This may, of course, be an outcome of the audit, but this seems to guide the auditor to only look for the big fish. What if there are lots of minor errors that suggest the process is not in control. Should these be ignored? My view is that an audit is intended to look at the effectiveness of systems and processes and is based on a sample, and all bad AND good findings are passed to management for action. End of story. There should not be any influence on the auditor, unless there is a for cause audit.
An audit of the effectiveness of actions is good to cover during an audit, but can it be a metric? The only one I can think of is, has the effectiveness of all actions has been audited? Some auditors don't look at the results of previous audits, and some orgs have the effectiveness of actions built into the CAPA system, so it's not the responsibility of the auditor?
The time to closure is a metric but not one for the auditor surely? Maybe one for the auditees management?
The number of opportunities for improvement and the potential number of NCRs surely depends on the process being audited and not a metric for the auditor? What if the process is so good you cannot find fault? So you say in your audit report, based on my audit sample, the process was great. I
I think we need to define the scope of the audit process before we apply metrics to it. For example, in some organisations, the audit process is separate, but feeds into, the CAPA process. In small orgs, like the one I work for, the auditor does the audit, does the audit admin, seeks CAPA, and tracks and trends action plan completion. Also, CAPA plans are usually reviewed and approved by Quality so we have an opportunity to review effectiveness and cost saving benefits of the action plans, but this may not be the responsibility of the auditor. So I would create metrics just for performing and reporting audits.
I would suggest the following metrics may be more appropriate to measure the effectiveness of the audit process:
- Adherence to audit schedule
- Reports submitted on time
- Turnaround time for CAPA plans
- Regular reports to management on CAPA plans due and CAPA plan completion
- Trending of audit data by observation category, numbers of observations, numbers of CAPAs etc
- Strengths reported to management with a view to application in other areas
- Review of audit frequency based on effectiveness as measured by audit eg. your process is in control in 2 sequential audits so instead of annual audit, move to biennial audit?
The real value of the audit process is two-fold:
1. Designing your audit plan/checklist/strategy cleverly so that your sample encompasses the effectiveness of the whole system or process and
2. Performing the audit with the intention of identifying the opportunities for improvement and reporting these to management.