Measureables for Document Control and Internal Audits

[xfngrs]

When we first certified to ISO/TS 16949 we set up some pretty vague measurables. Now we are trying to pin down some more useful ones.

I am the Process Owner for "Document and Quality Records Control" and "Internal Audits".

For the first we are measuring the TAT between when a DCR (engineering change) is requested and when it begins routing and the TAT between when it is routed and when approved.

I have a problem with these measurables because the numbers they are based on are not really constants.

The time between when a DCR is requested and when it is routed is dependent upon whether the DCR submitted makes sense and is complete, whether attachments are included or I have to ask for them, whether the person submitting it hands me a marked up hardcopy I have to edit for them first, whether there is one document or ten, etc.

The time between when routed and when approved is dependent upon whether the approvers are travelling and it maybe out 2 days here, 4 days there, etc. Sometimes we use alternates, but often not. Sometimes there is one document to review, sometimes an entire PPAP package or new specifications.

For Internal Audits are measurable is no External Audit findings.

I have been looking through the standard, but I'm not finding anything that would work better. Can somebody suggest something or tell me the types of things you've been using.

Pretty, pretty please?

 

[Stijloor]

Re: Measurables for Document Control and Internal Audits

For Internal Audits are measurable is no External Audit findings.

Pretty, pretty please?

Hello xfngrs,

I'll tackle internal audits....

I believe that the internal audit process must add value (I am sure that many will argue how you should calculate this....). In addition to what you already identified as a valid measurement ("No external audit findings"), think about this:

• The number of (true) opportunities for process and product improvement identified
• The number of potential nonconformities (risks) identified
• Internal (auditee) customer satisfaction scores
• What do others in your organization want from the internal audit process?

I'll think about the other one....

Stijloor

 

[Duke Okes]

Re: Measurables for Document Control and Internal Audits

Ask yourself (the organization) who the customers are of those two processes. Then go talk to the customers and ask them what is important about the outcomes of those processes to them, which measurable outcomes are most important, and what improvements they perceive would be most useful. Then establish metrics around the ones that make more sense to measure and set targets for improvement.

 

[AndyN]

Re: Measurables for Document Control and Internal Audits

Don't bother!

Audits are part of the measurement of any process and there's no meaningful metric you can put on them. It's a waste of your time and there's better things to be doing with audits than trying to find some way of measuring them, honestly. Comparing them to external audits is next to useless.

Document control is not something that management are interested in. It's not a core process that affects much that the customer sees (except in unusual circumstances). management should only care about $$$ (some kind of document, I guess)

Use the audits to find out if the document controls are working.

I have seen people try to find purpose in measuring these and not yet found anything which is really useful. If you can, work on something which is going to get some attention.........

 

[Stijloor]

Re: Measurables for Document Control and Internal Audits

Don't bother!

Audits are part of the measurement of any process and there's no meaningful metric you can put on them. It's a waste of your time and there's better things to be doing with audits than trying to find some way of measuring them, honestly. Comparing them to external audits is next to useless.

I strongly disagree. Auditing is a process that has (internal) customers. Managers (process owners) have certain expectations (see my previous post) that can be measured.

 

[AndyN]

Re: Measurables for Document Control and Internal Audits

I strongly disagree. Auditing is a process that has (internal) customers. Managers (process owners) have certain expectations (see my previous post) that can be measured.

Audits wouldn't be done if it wasn't for 'ISO' (and I mean by that registration). Management have only one expectation - do them, don't involve me and make it as painless as possible.

I have yet to meet a manager who really understands the benefits of audits. Sure, they smile sweetly and make approving noises, but if it wasn't for ISO, they wouldn't be supported.

The measurements you stated are subject to too much variation to be of use. One auditor may find more than another. What does that mean? If you factor in that most auditors are not familiar with the process they're auditing (they're from another, 'independent' dept) won't that affect their findings? Are you suggesting that auditors be given a quota? Do/don't find more than X nc's? "Make the audit score card look good!"

Isn't the size of organization (processes, headcount, product volume etc), complexity of organization, processes, maturity (of auditors, audit program and QMS) all influences on audit results? Won't they have bearing on the audit results? There's too much to consider which will prevent meaningful assignment of numerical values to the audit(s).

Believe me, I've been asked more times than I can remember about the number of non-conformities found during audits. Is 1 good, is 10? Is none? There are no answers to any 'measurement' questions regarding audits.

Internal customer satisfaction is inversely proportional to the number and type of findings. The less an auditor finds, the happier management are because they have no actions to work on.

I have sat through countless Management Reviews, including Ford's supplier QOS metrics and there has never been any substantial discussion about audit metrics. So why waste time working on something which is underwhelming to the vast majority of an organizations management and supervision??

There is only one conceivable number (that hasn't been mentioned) that is of any interest to anybody - the cost benefit of doing audits. They cost a lot to do - training, disruption to the business day, taking people out of their productive work to perform the audit.

So maybe (just maybe) the measurement should be what did the auditors find/report on that was at least equivalent to the cost of the audit?

 

[pldey42]

Re: Measurables for Document Control and Internal Audits

I've seen two good measures of internal audits:

Cost of audits (and corrective actions) compared to savings, an ROI-like calculation. It makes sense: if audits aren't saving money, what are they doing? (Sometimes audit findings and corrective actions will save time, or reduce reject and return rates, or improve on-time delivery; these are all things that can be measured in money somehow.)

The costs are annualized, and estimated if they can't be measured cheapy, and audited by internal process auditors often drawn from the accounts department - thereby engaging the financial people in process management, an added bonus.

The benefits of this measure are that it concentrates auditors on findings that add measurable value to the business, and it provides excellent justification that cost-conscious managers understand for continuing the audit programme.

I don't see the point of measuring # of external findings with the target of using internal audits to make it zero, because that means the internal auditors are doing the external auditors' job - duplication of effort, expensive. Of course it's common, because too many organizations regard audit findings as 'a bad thing'. But they aren't, they're a problem waiting to happen, better fixed. Nobody loses their certificate because of a non-conformity report, it's not fixing it that will do that. So since organizations pay the external auditors to audit and report non-conformities -- and good things too, hopefully -- why not let them do it?

The other measure I've seen in a few mature organizations, where managers regard audits as providing valuable insights into their organizations, is customer satisfaction. Managers are regarded as customers for audits and given customer satisfaction surveys with questions like, "Was the auditor prepared?" and "Were the findings relevant and helpful?"

For document control I've yet to see a decent measure. I'd like to see someone measure the 'cost of confusion' - in other words, the costs associated with working to wrong copies of documents, or of poor imact analysis and change control. The neat thing about this measurement would be that, ideally, it would also sniff out useless documentation, or stuff that meets the formal requirements yet nobody understands or uses it.

Just 2c,
Pat

 

[Kales Veggie]

Re: Measurables for Document Control and Internal Audits

Interesting discussion.

IMHO, the metrics that Stijloor mentions are subjective and probably all audit metrics are subjective. So, if you then set targets for the subjective measurables, get what happens. The audit team will audit so that the targets are being met. This would not be a true representation of the process or system being audited. (WE have to meet the target.....). If you want 0 N/Cs, N/Cs will be written as OFIs or recommendations. If our goals is 5 improvement ideas per audit, that is what we will get (or pretty close to it).

I believe that audit metrics should be limited to measuring open N/Cs by months and the aging of N/Cs. If the organization is actively managing the quality system, corrective actions for N/Cs will be implemented timely (which is an TS requirment by the way).

Metrics/Audits results.
I have worked with organizations (US/Japanese joint venture, large bearing manufacturer, privately held automotive suppliers) where metrics are actively used, managed and acted upon. In Quality reviews (with VPs, plant managers and quality managers present), metrics are reviewed/discussed monthly, quality managers report on status of C/As and make predictions on where the metrics are heading for the next few months. In one instance, there were 3 plants making similar products for FoMoCo. We created an internal competition for maintaining the highest Ford Q1 score. The Q1 scores were in the 1400 range until one plant lost the Q1. Q1 was reinstated in 6 months and 10 days (that is a different story). This approach of reviewing and acting on metrics (with some stretch goals) drives continuous improvement.

 

[potdar]

Re: Measurables for Document Control and Internal Audits

I tend to agree with Andy's view but with a different logic.

Let me start at a basic definition. A process converts an input into an output using certain resources. it has an owner and its effectivness can be gauged with an attached measure.

Now. what should be defined as a process is left to the designer / user. For all practical purposes, the organisation's activities can be defined as a single process with customer requements as an input and customer deliveries as the output.

We split it into maller processes to make it more controllable to suit our operations and organisation structure.

Many a times while doing this, we end up defining measures for different processes that clash with each other.

In this case as well, it makes more sense to define measurables for the process of 'management activities' as a whole rather than for IA and document control and other parts separately.

Document control, when it comes to design documents may be clubbed separately with design activities.

 

[Duke Okes]

Re: Measurables for Document Control and Internal Audits

My original response was intended to describe a process that can be used to identify metrics/objectives for anything. However, I'd have to agree with AndyN that measures/objectives for document control and auditing are probably pretty useless. Almost no one in the organization would be interested.

Metrics/objectives should be focused on business processes, and only on support processes where their impact on business processes is significant.