Measureables for Document Control and Internal Audits

Stijloor

Staff member
Super Moderator
#51
Re: Measurables for Document Control and Internal Audits

To go back to AndyN's very useful metaphor about the dashboard, you need gages (measurements) for your core processes only. All the others should be relegated to warning light status. Else you run the risk of providing too much information.
Sorry Cliff, with all due respect.....

When the "warning light" comes on, you may already be too late. The engine may have already been damaged beyond repair. (Ask my daughter....). Are internal audits not meant to be preventative in nature?

Stijloor.
 
Elsmar Forum Sponsor
J

JaneB

#53
Re: Measurables for Document Control and Internal Audits

Great discussion, all.

When the "warning light" comes on, you may already be too late. The engine may have already been damaged beyond repair. (Ask my daughter....).
Indeed, so. A friend of mine was driving down to the coast late one evening when a red light on the dash came on. I asked her what she did. She said she didn't know what it meant, so she just kept driving! Aargh. (Seized the engine, very expensive repair of course).

Are internal audits not meant to be preventative in nature?
Certainly one of their functions in my mind.

I'm definitely of the 'keep it simple, & don't go overboard with a whole lot of data that management won't want to be burdened with' school, but I can't go along with not having any assessment of things like IA or doc control at all. Sounds like the critical point here is how can we assess/measure/monitor effectively and usefully?

The fact that it hasn't been done yet doesn't mean it can't be done or shouldn't be attempted, IMO.
 
C

CliffK

#54
A few points...

Stijloor: I don't see how devising some measurement scheme for internal audits will make internal audits preventive in nature.

Do you think the warning light came on after the damage was done, or do you think the warning light was not noticed or was ignored? Do you really think a gage would have made any difference?

And aren't measurements after-the-fact also? Well, run charts have some predictive value, I suppose. But I don't think we're ready to advocate plotting run charts for data as slippery as internal audit measurements, are we?

That said, I will admit that measurements of the internal audit program might be useful for the guy that runs the program, as an indicator of its performance and to drive improvement. ISO 19011 suggests that the audit programme (sic) manager should continually improve the programme, after all. But report that measurement to executive management? Never.

Jim Wynne: Theoretically, the analysis of audit data might give you a clue to preventive action, but I have yet to see it happen. From what I've seen, problems show up in other ways long before they raise their heads in the internal audit process.

Sometimes an internal audit can verify that a corrective action remains in place after the initial focus. But I don't see that as preventive, either. Just another quality police action.

JaneB: I never said you should not monitor things like internal audits or document control. Measuring them is a waste of time unless they have been identified as critical areas. In an earlier post I suggested a way of monitoring these processes...I'm sure there are others
 
#55
Re: Measurables for Document Control and Internal Audits

Sorry Cliff, with all due respect.....

When the "warning light" comes on, you may already be too late. The engine may have already been damaged beyond repair. (Ask my daughter....). Are internal audits not meant to be preventative in nature?

Stijloor.
Using the dashboard analogy, would putting a gauge on the dash - showing the actual oil pressure data from the engine - have made any difference? I don't think so. The warning light is simply to tell you to take action, because some maintenance activity or similar hasn't been done effectively.

Since a 'twist' has been put on this story, I'd like to suggest that (continuing the analogy) the vehicle wasn't subjected to an effectively implemented preventive maintenance process, which would have revealed low pressure, sludging, or lack of oil (through leakage, consumption by the engine etc.). The use of audit, continuing the analogy, would have had someone (other than the user of the car) look at the oil change records, service records, interview the car user to see if they knew what their responsibility for maintenance was. This would have revealed the lack of effective maintenance, or responsibility (awareness, competency etc) with the user.

IMHO, it's only the core process needs to be monitored and measured - in this case maintenance. Audits are used to verify the process is being effectively implemented and report on its status - not to have measurement applied to it for its own sake. The audit should be able to confirm that the maintenenace process demonstrated that oil was changed to schedule, oil level checks were performed when there were signs of oil being burnt/leaked by the engine, sludging was occuring etc.

Audits per se don't have to be measured, the audit results have to be used in conjunction with the core process performance (results) to know if audits are effective.

Since the engine blew up we now know that the maintenance process was ineffective..........Effective audits would have confirmed this, perhaps before the catastrophy............
 

xfngrs

Quite Involved in Discussions
#57
What about running a pareto of the processes or elements with most issues, CAR's vs. OI's. We do issue OI's and follow-up on them in subsequent audits. It won't measure effectiveness of the process directly, but it would possibly show you the most troublesome areas.

I would normally say to do it by element, but I think Doc Control will always be at the top of that list....

What do you think or the pros and cons of this?
 
C

CliffK

#58
What about running a pareto of the processes or elements with most issues, CAR's vs. OI's. We do issue OI's and follow-up on them in subsequent audits. It won't measure effectiveness of the process directly, but it would possibly show you the most troublesome areas.

I would normally say to do it by element, but I think Doc Control will always be at the top of that list....

What do you think or the pros and cons of this?
Sorry, no value. Too much variability from audit to audit, resulting in an unacceptable uncertainty of measurement.
 
#59
What about running a pareto of the processes or elements with most issues, CAR's vs. OI's. We do issue OI's and follow-up on them in subsequent audits. It won't measure effectiveness of the process directly, but it would possibly show you the most troublesome areas.

I would normally say to do it by element, but I think Doc Control will always be at the top of that list....

What do you think or the pros and cons of this?
You could do something like this, but I'm not certain it tells you much about the audit process, itself. Giving management a set of graphs to look at isn't going to do much good. I also believe that there have been two vitally important components missing from this thread, IMHO:

1) the planning of audits
2) the fact that audits are part of the "Measurement, Analysis and Improvement" section of ISO 9001.

The effectiveness of audits is a function of planning them. If you are going to do something to measure the effectiveness of audits, you're probably going to have to do something to make the planning really effective. Just like any other process, setting a 'goal' for it is important so you can judge progress/performance.

So, on that basis, (plus we already have debated that you can't really set goals around the number of NC's, OFi's etc.), you will have to draw up something which is likely to be understood by management (without too much interpretation).

Why not do the cost analysis? Develop a standard cost for an audit 'day' and look at the results to see what the audit findings 'pay back'. The goal would be to at least break even!

You could also look at how many audits are 'pushed' on a process and how many are requested by management (if they get the reasons why we should do audits, they will want you to audit their process) and set a goal of 40 or 45% (say) this year, 50 - 60% next year and so on. Ultimately, you might get to 95 or more. That's improvement in my book!

The point about the section of the standard that audits is found in, is a clue, IMHO, that the audits are there to report on effectiveness, be a monitoring action and help with improvements.

As a business, the focus should be on improving the customers' perceptions and the main value adding processes as a priority. When your organization has begun to 'flatten out' on improving those two business aspects, then I'd turn the attention to other things like other areas of the standard (audit, calibration, etc. etc.), but not as a priority.

From a business sense it makes no sense for you to be working and reporting on those aspects, when others might need your help......
 

Stijloor

Staff member
Super Moderator
#60
Hello Andy,

I appreciate your valuable insights. It appears that you and I are not that far apart! I believe that you and I have the success of the audit process (and the documentation control process) in mind. We can differ on methods and approaches on how to accomplish this. I'm a strong objectives & measurements guy. The reason for this is that I have seen too much subjective-guesswork-assuming BS. In other words, no substance, just talk. Objectives and measurements? Yes, but not to the extent where this becomes a meaningless, non-value added activity. I know better.

Thanks Andy.

Stijloor.
 
Thread starter Similar threads Forum Replies Date
P Received a minor for not having good measureables/goals. Need help with KPIs. IATF 16949 - Automotive Quality Systems Standard 52
A Measureables to Monitor an Integrated Management System Capability IATF 16949 - Automotive Quality Systems Standard 3
A Corrective Action Tracking - What Measureables and How to Track Status ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 11
J Lack of Measureables in our Quality Objectives ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 12
J Quality Objective Measureables - Please Review ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 14
N Quality Objective - Scrap Measureables ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 13
J Measureables to determine efficiency and effectiveness of the COP Processes IATF 16949 - Automotive Quality Systems Standard 4
H Ergonomics Metrics (Measureables) - Human factors engineering Lean in Manufacturing and Service Industries 39
D Design and Development Measureables - ISO9001:2000 Clause 7.3 Design and Development of Products and Processes 5
G Statistical Techniques Measureables for Application Development Company Statistical Analysis Tools, Techniques and SPC 3
1 Made it through Phase One - Measureables for Management Review - Need list or form Management Review Meetings and related Processes 9
K Business Plan - Quality Objectives and Measureables - TS 16949 Para 5.4.4 IATF 16949 - Automotive Quality Systems Standard 8
A Internal Audits - Design Process Measurement (Measureables) Design and Development of Products and Processes 6
Icy Mountain Quality Objectives Redux - Setting Actual Numbers (Measureables) IATF 16949 - Automotive Quality Systems Standard 15
L Measureables: Efficiency - Effectiveness of COP (Customer Oriented Process) and KSP ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 8
7 ISO 14001 Measureables - Plastics related industry Miscellaneous Environmental Standards and EMS Related Discussions 1
Marc Measuring and Improving Quality - Choosing Measureables Quality Tools, Improvement and Analysis 3
J Document Control Metrics Document Control Systems, Procedures, Forms and Templates 3
A Document Review and Document Approval --- 2 Signatures needed acc. §820.40? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 3
I Document Control Software Document Control Systems, Procedures, Forms and Templates 2
K 25-year lifetime of medical device - document storage period EU Medical Device Regulations 1
J UDI - Where to document it? EU Medical Device Regulations 8
T Controlling Expandable Forms in Paper-Based Document Control System Document Control Systems, Procedures, Forms and Templates 10
S Changing revision status of a reviewed document Manufacturing and Related Processes 4
shimonv Document Control Procedure Header Content Document Control Systems, Procedures, Forms and Templates 3
A Document release vs its related training. Which should come first? ISO 13485:2016 - Medical Device Quality Management Systems 18
Q Document ownerships Document Control Systems, Procedures, Forms and Templates 14
E Document Change Request forms (or no forms)? ISO 13485:2016 - Medical Device Quality Management Systems 9
O Any info on release date of FDA “Computer Software Assurance for Manufacturing and Quality System Software” document? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 0
B Document Approval Matrix Benchmarking Document Control Systems, Procedures, Forms and Templates 3
J Document Approval Signature Order Document Control Systems, Procedures, Forms and Templates 10
L How to add exemption or statement to control of document procedure? ISO 13485:2016 - Medical Device Quality Management Systems 5
M What is the proper way to document measurements Measurement Uncertainty (MU) 1
P Equipment 21 CFR 820.70(g) - User Requirements Document for Off the shelf equipment 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 7
B Do IFU designs have to be document controlled under ISO 13485? Document Control Systems, Procedures, Forms and Templates 2
C Certificate of Conformance Form - COC for each customer a controlled document? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
A Document "Correspondence IATF 16949 vs ISO13485" available? IATF 16949 - Automotive Quality Systems Standard 0
R Document Retention - Discard hard-copies after scanning? ISO 13485:2016 - Medical Device Quality Management Systems 2
P Mylar plot suppliers in accordance with D6-51991 document AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 4
A Document Change Notice vs complete System re-write Manufacturing and Related Processes 4
Q Document approval through SharePoint (without signature) Records and Data - Quality, Legal and Other Evidence 4
S What is considered a "core algorithm"? (From an FDA guidance document) Medical Information Technology, Medical Software and Health Informatics 4
U Document Approval - Software company ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
O Please, help with document on ISO/ TS 17758 and (IDF 87: 1979). Thank you. Manufacturing and Related Processes 1
K Proper document of SMPS used in infant warmer for IEC 60601-1 testing IEC 60601 - Medical Electrical Equipment Safety Standards Series 1
I Nitpicking on document released dates ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
I Document Control on Log Files ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 6
Q Refusal to Document Complaints Customer Complaints 39
Q Tracking Procedure Revisions (Document Control) Document Control Systems, Procedures, Forms and Templates 9
C Document Control Stamps - Does anyone still stamp their documents? Document Control Systems, Procedures, Forms and Templates 24

Similar threads

Top Bottom