Measureables for Document Control and Internal Audits

C

curryassassin

#41
Re: Measurables for Document Control and Internal Audits

Hi Covers,
I've been following this thread with much interest, and, after careful thought, here is my pennyworth. If I may, I'd like to address the comments made by some of you and then suggest some of my own:

There were some very good examples of audit metrics, but I disagree with the one to do with the auditor finding problems that may become huge issues. This may, of course, be an outcome of the audit, but this seems to guide the auditor to only look for the big fish. What if there are lots of minor errors that suggest the process is not in control. Should these be ignored? My view is that an audit is intended to look at the effectiveness of systems and processes and is based on a sample, and all bad AND good findings are passed to management for action. End of story. There should not be any influence on the auditor, unless there is a for cause audit.

An audit of the effectiveness of actions is good to cover during an audit, but can it be a metric? The only one I can think of is, has the effectiveness of all actions has been audited? Some auditors don't look at the results of previous audits, and some orgs have the effectiveness of actions built into the CAPA system, so it's not the responsibility of the auditor?

The time to closure is a metric but not one for the auditor surely? Maybe one for the auditees management?

The number of opportunities for improvement and the potential number of NCRs surely depends on the process being audited and not a metric for the auditor? What if the process is so good you cannot find fault? So you say in your audit report, based on my audit sample, the process was great. I

I think we need to define the scope of the audit process before we apply metrics to it. For example, in some organisations, the audit process is separate, but feeds into, the CAPA process. In small orgs, like the one I work for, the auditor does the audit, does the audit admin, seeks CAPA, and tracks and trends action plan completion. Also, CAPA plans are usually reviewed and approved by Quality so we have an opportunity to review effectiveness and cost saving benefits of the action plans, but this may not be the responsibility of the auditor. So I would create metrics just for performing and reporting audits.

I would suggest the following metrics may be more appropriate to measure the effectiveness of the audit process:

  • Adherence to audit schedule
  • Reports submitted on time
  • Turnaround time for CAPA plans
  • Regular reports to management on CAPA plans due and CAPA plan completion
  • Trending of audit data by observation category, numbers of observations, numbers of CAPAs etc
  • Strengths reported to management with a view to application in other areas
  • Review of audit frequency based on effectiveness as measured by audit eg. your process is in control in 2 sequential audits so instead of annual audit, move to biennial audit?
I think we need to bear in mind that conducting an audit is tough enough (all your senses working/skills/personality) for the auditor without the background pressure of meeting what may be inappropriate metrics.

The real value of the audit process is two-fold:

1. Designing your audit plan/checklist/strategy cleverly so that your sample encompasses the effectiveness of the whole system or process and
2. Performing the audit with the intention of identifying the opportunities for improvement and reporting these to management.
:D
 
Elsmar Forum Sponsor
V

vanputten

#42
Re: Measurables for Document Control and Internal Audits

1. In my opinion, the metrics I offered are for measuring the process and are not individual performance metrics. If one choses to view or use those metrics for individual / employee performance, then you have missed the point. These are audit process metrics.

2. In general, an audit process does not fix anything. Audits provide actionable data as an input into corrective action. Corrective action process is the fix or improvement.

3. I don't think the language could be any softer... suggested metrics to consider. No mention of requirements, that this is the only way, dogma, or that all have metrics have to be used.

4. To me, the following comment seems a little harsh from a Froum Moderator, "Most of these metrics are for audit/auditors managers who have nothing better to do - and we have already established that in the main, most of these folks have a real daytime job........." Are the principles and metrics underlying internal and external audits really different?

5. Measure what you want or what you think you can measure. Maybe we shouldn't even do audits if it is not a "real daytime job."

Regards,

Dirk
 

Stijloor

Staff member
Super Moderator
#43
Re: Measurables for Document Control and Internal Audits

1. In my opinion, the metrics I offered are for measuring the process and are not individual performance metrics. If one choses to view or use those metrics for individual / employee performance, then you have missed the point. These are audit process metrics.
Well articulated Dirk!

2. In general, an audit process does not fix anything. Audits provide actionable data as an input into corrective action. Corrective action process is the fix or improvement.
Absolutely, amen to that!

3. I don't think the language could be any softer... suggested metrics to consider. No mention of requirements, that this is the only way, dogma, or that all have metrics have to be used.
I agree!

4. To me, the following comment seems a little harsh from a Forum Moderator, "Most of these metrics are for audit/auditors managers who have nothing better to do - and we have already established that in the main, most of these folks have a real daytime job........." Are the principles and metrics underlying internal and external audits really different?
I agree!

5. Measure what you want or what you think you can measure. Maybe we shouldn't even do audits if it is not a "real daytime job."

Regards,

Dirk
Thank you Dirk for hitting the nail on the head!

Stijloor.
 

pondo

Registered Visitor
#44
Re: Measurables for Document Control and Internal Audits

1. In my opinion, the metrics I offered are for measuring the process and are not individual performance metrics. If one choses to view or use those metrics for individual / employee performance, then you have missed the point. These are audit process metrics.

2. In general, an audit process does not fix anything. Audits provide actionable data as an input into corrective action. Corrective action process is the fix or improvement.

3. I don't think the language could be any softer... suggested metrics to consider. No mention of requirements, that this is the only way, dogma, or that all have metrics have to be used.

4. To me, the following comment seems a little harsh from a Froum Moderator, "Most of these metrics are for audit/auditors managers who have nothing better to do - and we have already established that in the main, most of these folks have a real daytime job........." Are the principles and metrics underlying internal and external audits really different?

5. Measure what you want or what you think you can measure. Maybe we shouldn't even do audits if it is not a "real daytime job."

Regards,

Dirk
Meanwhile...I am going to hang-out in the real world. :truce:
 

Stijloor

Staff member
Super Moderator
#45
Re: Measurables for Document Control and Internal Audits

Meanwhile...I am going to hang-out in the real world. :truce:
Pondo,

In the real world, ALL processes that make up the quality management system have clear objectives and associated measurements. This will allow us to determine if the processes are delivering the expected results. If not, corrective action will be taken. Oh yes my friends, this includes Internal Auditing and Control of Documents! This is exactly in line with "Monitoring and Measurement of Processes." The objectives and measurements may be simple, but that is far better than having nothing. Having processes and NOT knowing how they perform would go against the principle of process control.

Stijloor.
 

Jim Wynne

Staff member
Admin
#46
Re: Measurables for Document Control and Internal Audits

Pondo,

In the real world, ALL processes that make up the quality management system have clear objectives and associated measurements. This will allow us to determine if the processes are delivering the expected results. If not, corrective action will be taken. Oh yes my friends, this includes Internal Auditing and Control of Documents! This is exactly in line with "Monitoring and Measurement of Processes." The objectives and measurements may be simple, but that is far better than having nothing. Having processes and NOT knowing how they perform would go against the principle of process control.

Stijloor.
After watching this thread go back and forth between two polarized points, I think I know what the problem is. On the one side we have, "Don't worry about it, it's not a big deal," and on the other there's "It's a PROCESS!!! It MUST BE MEASURED!!!" The right answer is somewhere in the middle, imo. To say that something should not be overtly measured isn't the same thing as saying that no one cares whether or not the thing is working as intended. We usually know whether or not the document control system is working without having to hold a yardstick up against it. The same applies to internal audit, for the most part. We know that if a problem develops in one of those areas (which will almost always be personnel issues, btw) we'll know it, and act accordingly. When we start making up objectives where the objectives should be bloody obvious, nothing of substance is likely to be gained. Document control and internal audit are important, and need to be monitored (that's what managers are for), but I've never seen an instance where written objectives and measurement against them were helpful.
 
B

Brian Dowsett

#47
Hello Xfngrs

For doc control, define what docs need control then use your audits to count how many of those type of docs are under doc control. Before you put in your system that number would be large. Now it should be small - if your system is effective.

For internal audits the important thing again is "are they effective" - meaning do people fix the problems that you identify. The measure of this is the number of repeat non-conformances you get. Believe me if you never get the same non-conformance repeated at a later audit you are doing very well.

Cheers

Brian


Miss Lola: "Tell me. Tell me about my dear dear Daddy. Is it true he's dead?
Stan: "Well we hope he is.. they buried him"
MIss Lola: "Oh it cant be! What did he die of??"
Stan: "I think he died of a Tuesday, or was it a Wednesday?"
Way Out West - 1937
 

Stijloor

Staff member
Super Moderator
#48
Hello Xfngrs

For doc control, define what docs need control then use your audits to count how many of those type of docs are under doc control. Before you put in your system that number would be large. Now it should be small - if your system is effective.

For internal audits the important thing again is "are they effective" - meaning do people fix the problems that you identify. The measure of this is the number of repeat non-conformances you get. Believe me if you never get the same non-conformance repeated at a later audit you are doing very well.

Cheers

Brian


Miss Lola: "Tell me. Tell me about my dear dear Daddy. Is it true he's dead?
Stan: "Well we hope he is.. they buried him"
MIss Lola: "Oh it cant be! What did he die of??"
Stan: "I think he died of a Tuesday, or was it a Wednesday?"
Way Out West - 1937
For our International Covers:

http://www.imdb.com/title/tt0029747/
http://en.wikipedia.org/wiki/Way_Out_West_(1937_film)

Stijloor.
 
C

curryassassin

#49
Re: Measurables for Document Control and Internal Audits

but I've never seen an instance where written objectives and measurement against them were helpful.
The audit process shows if the QS is effective. The audit process has to show that the required audits were performed at the required frequency and that the results were reported to management in a timely manner. So there's a couple of helpful measures.

We need documents to tell us what to do and to control what we do, so measuring a turnaround time for writing, reviewing and approving documents is a helpful measure of the effectivenss of the document control process. With an efficient DC process, making the product or providing the service proceeds without delay.

I agree, we must always use SMART measures.

The UK Health Service, amonsgt other public sector services, is currently grumbling that their performance is now being measured to see if patients are being treated in a timely manner for one example. For years each successive government (both sides) have poured money in and local health authorities have been charged with managing their budgets and have just overspent wildly (I know there are other politics involved). Now the government is saying 'measure up' and 'balance your books'. Any other business would have died a long time ago with the same poor performance.
 
C

CliffK

#50
Re: Measurables for Document Control and Internal Audits

What I have to say is possibly a reaction to some of the over-complicated, over-thought, over-documented and over-killed systems I have seen recently. If this post sounds like a soap-box rant, that's why.

That said,
The audit process shows if the QS is effective.
Sorry, it doesn't. It could, I suppose, but as currently practiced it's nothing more than a quality police action.

We need documents to tell us what to do...
We don't need nearly as many as we generally think, especially when it comes to procedures and work instructions.

I'm thinking of two companies with which I have worked. In the first, the shipping department packs and crates delicate electronic equipment. Each system unique; transport may be by truck, sea or air. This shipping department functions successfully without benefit of detailed packaging instructions. The second has work instructions for stacking bags on a pallets and loading pallets on trucks. Their success rate is no higher than the first company. So what is the value of their work instructions?

to control what we do
There is no control, only motivation.

so measuring a turnaround time for writing,reviewing and approving documents is a helpful measure of the effectivenss of the document control process.
People keep forgetting that data are expensive. You can monitor a process at far less cost. One way to do that is to see if the process causes or contributes to nonconformities. If the document control process isn't causing problems with a core process, leave it alone. Surely there are more important problems to solve.

To go back to AndyN's very useful metaphor about the dashboard, you need gages (measurements) for your core processes only. All the others should be relegated to warning light status. Else you run the risk of providing too much information.
 
Thread starter Similar threads Forum Replies Date
P Received a minor for not having good measureables/goals. Need help with KPIs. IATF 16949 - Automotive Quality Systems Standard 52
A Measureables to Monitor an Integrated Management System Capability IATF 16949 - Automotive Quality Systems Standard 3
A Corrective Action Tracking - What Measureables and How to Track Status ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 11
J Lack of Measureables in our Quality Objectives ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 12
J Quality Objective Measureables - Please Review ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 14
N Quality Objective - Scrap Measureables ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 13
J Measureables to determine efficiency and effectiveness of the COP Processes IATF 16949 - Automotive Quality Systems Standard 4
H Ergonomics Metrics (Measureables) - Human factors engineering Lean in Manufacturing and Service Industries 39
D Design and Development Measureables - ISO9001:2000 Clause 7.3 Design and Development of Products and Processes 5
G Statistical Techniques Measureables for Application Development Company Statistical Analysis Tools, Techniques and SPC 3
1 Made it through Phase One - Measureables for Management Review - Need list or form Management Review Meetings and related Processes 9
K Business Plan - Quality Objectives and Measureables - TS 16949 Para 5.4.4 IATF 16949 - Automotive Quality Systems Standard 8
A Internal Audits - Design Process Measurement (Measureables) Design and Development of Products and Processes 6
Icy Mountain Quality Objectives Redux - Setting Actual Numbers (Measureables) IATF 16949 - Automotive Quality Systems Standard 15
L Measureables: Efficiency - Effectiveness of COP (Customer Oriented Process) and KSP ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 8
7 ISO 14001 Measureables - Plastics related industry Miscellaneous Environmental Standards and EMS Related Discussions 1
Marc Measuring and Improving Quality - Choosing Measureables Quality Tools, Improvement and Analysis 3
J Document Control Metrics Document Control Systems, Procedures, Forms and Templates 3
A Document Review and Document Approval --- 2 Signatures needed acc. §820.40? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 3
I Document Control Software Document Control Systems, Procedures, Forms and Templates 2
K 25-year lifetime of medical device - document storage period EU Medical Device Regulations 1
J UDI - Where to document it? EU Medical Device Regulations 8
T Controlling Expandable Forms in Paper-Based Document Control System Document Control Systems, Procedures, Forms and Templates 10
S Changing revision status of a reviewed document Manufacturing and Related Processes 4
shimonv Document Control Procedure Header Content Document Control Systems, Procedures, Forms and Templates 3
A Document release vs its related training. Which should come first? ISO 13485:2016 - Medical Device Quality Management Systems 18
Q Document ownerships Document Control Systems, Procedures, Forms and Templates 14
E Document Change Request forms (or no forms)? ISO 13485:2016 - Medical Device Quality Management Systems 9
O Any info on release date of FDA “Computer Software Assurance for Manufacturing and Quality System Software” document? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 0
B Document Approval Matrix Benchmarking Document Control Systems, Procedures, Forms and Templates 3
J Document Approval Signature Order Document Control Systems, Procedures, Forms and Templates 10
L How to add exemption or statement to control of document procedure? ISO 13485:2016 - Medical Device Quality Management Systems 5
M What is the proper way to document measurements Measurement Uncertainty (MU) 1
P Equipment 21 CFR 820.70(g) - User Requirements Document for Off the shelf equipment 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 7
B Do IFU designs have to be document controlled under ISO 13485? Document Control Systems, Procedures, Forms and Templates 2
C Certificate of Conformance Form - COC for each customer a controlled document? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
A Document "Correspondence IATF 16949 vs ISO13485" available? IATF 16949 - Automotive Quality Systems Standard 0
R Document Retention - Discard hard-copies after scanning? ISO 13485:2016 - Medical Device Quality Management Systems 2
P Mylar plot suppliers in accordance with D6-51991 document AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 4
A Document Change Notice vs complete System re-write Manufacturing and Related Processes 4
Q Document approval through SharePoint (without signature) Records and Data - Quality, Legal and Other Evidence 4
S What is considered a "core algorithm"? (From an FDA guidance document) Medical Information Technology, Medical Software and Health Informatics 4
U Document Approval - Software company ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
O Please, help with document on ISO/ TS 17758 and (IDF 87: 1979). Thank you. Manufacturing and Related Processes 1
K Proper document of SMPS used in infant warmer for IEC 60601-1 testing IEC 60601 - Medical Electrical Equipment Safety Standards Series 1
I Nitpicking on document released dates ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
I Document Control on Log Files ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 6
Q Refusal to Document Complaints Customer Complaints 39
Q Tracking Procedure Revisions (Document Control) Document Control Systems, Procedures, Forms and Templates 9
C Document Control Stamps - Does anyone still stamp their documents? Document Control Systems, Procedures, Forms and Templates 24

Similar threads

Top Bottom