Establishing Metrics - Where do you document metrics?

[Michael_M]

I am starting to 'officially' establish metrics that are to be reviewed on an on-going basis. As part of the metrics, I have to establish 'good' range and 'bad' range.

For example, job time entries (how many parts are produced in XX time).

I am not sure where the best place to put these numbers at. I don't want to put it in the actual Quality manual nor can I think of a specific procedure that the range of numbers should be put into (staying with the job time entries as an example).

I was thinking of writing up Work Instructions on how to review the data but that seems a bit too micro-management to me since some data would have some subjectivity to it, does the operator run 1 machine, 2 machines (slows production down a bit), or run 3 machines (slows production down a bit more, but 3 jobs are running).

So the actual question: where would I put numbers we call 'good', 'follow up', or 'bad, need to fix' so they are somewhat official?

I don't know if this helps, but we have:
QM
Procedures
Work Instructions
Forms

 

[Steve Prevette]

Re: Establishing metrics, where do you document them?

I am starting to 'officially' establish metrics that are to be reviewed on an on-going basis. As part of the metrics, I have to establish 'good' range and 'bad' range.

I suggest the first discussion is the how did you identify the "good" range and "bad" range pieces. You may find that if they are arbitrary set points, especially not based upon the subject data, you will have a lot of frustration and confusion. I would suggest you look at SPC techniques (documented elsewhere here in the Cove).

As far as where to document your metrics, you did not mention if you were working to ISO or other standards. I'd say you do need a procedure at least on how you are going to interpret the results (I have an example if you are interested in SPC). At this point in the game, I would not "carve in stone" what your specific metrics are, as very rarely does the first attempt to define metrics turn out to have 100% the "correct" metrics. You will discover some don't work so well and need to be dropped, and new ones may arise.

A more important question is how will the results be displayed to the workers and managers? A simple bulletin board may suffice. There are formal "scorecard" software out there for posting metrics to a location folks can find them. Or perhaps a monthly / quarterly report.

 

[Michael_M]

Re: Establishing metrics, where do you document them?

I suggest the first discussion is the how did you identify the "good" range and "bad" range pieces. You may find that if they are arbitrary set points, especially not based upon the subject data, you will have a lot of frustration and confusion. I would suggest you look at SPC techniques (documented elsewhere here in the Cove).

Right now we are ISO9001 compliant but moving into AS9100 certified. Also note, we are a machine shop.

In the example I am working with, we can generate a 'performance summary'. In general, we are looking for 100% performance (XX parts in YY time). In some cases, we need to alter the available time to run a part as actual run times are slower than expected. In some cases, something else is occurring.

What I am trying to find out is were to put 'anything between 95% and 115% is good, 85%-94% is suspect, and anything under 84% or over 116% we need to address somehow. Right now our numbers are 'verbal' and 'from memory'. Where would I document the base line of numbers we want to use for reference?

 

[Steve Prevette]

Re: Establishing metrics, where do you document them?

What I am trying to find out is were to put 'anything between 95% and 115% is good, 85%-94% is suspect, and anything under 84% or over 116% we need to address somehow. Right now our numbers are 'verbal' and 'from memory'. Where would I document the base line of numbers we want to use for reference?

First, I'd make sure those statements are in the chart graphic itself, so whenever someone looks at the chart they see at least the targets themselves.

Second, I'd make sure there is some form of controlled source for people to retrieve their charts from. That could be within your document control system, or just on a fileserver that is read-only for everyone except the chart maker. A document could be placed in this location that captures the logic of how the targets were set.

Third, I'd make at least a quarterly report for the record of the critical measures to line management. A technical appendix to the report could contain the logic by which the targets were established.

 

[Mikishots]

I am starting to 'officially' establish metrics that are to be reviewed on an on-going basis. As part of the metrics, I have to establish 'good' range and 'bad' range.

For example, job time entries (how many parts are produced in XX time).

I am not sure where the best place to put these numbers at. I don't want to put it in the actual Quality manual nor can I think of a specific procedure that the range of numbers should be put into (staying with the job time entries as an example).

I was thinking of writing up Work Instructions on how to review the data but that seems a bit too micro-management to me since some data would have some subjectivity to it, does the operator run 1 machine, 2 machines (slows production down a bit), or run 3 machines (slows production down a bit more, but 3 jobs are running).

So the actual question: where would I put numbers we call 'good', 'follow up', or 'bad, need to fix' so they are somewhat official?

I don't know if this helps, but we have:
QM
Procedures
Work Instructions
Forms

We use two methods. The first is metrics kept by each department /cell, displayed on a bespoke board. The board has numerous legal size plastic display slots for the KPIs identified by that area, so that the trending can be displayed and updated as necessary. Management does a morning "boardwalk" to each area, and the manager of that area explains the results for each metric.

The second is a simple Excel workbook where each department keeps their metrics, including a sheet that compiles the results in to the major process areas (engineering, production, business development etc.).

Works well, as anyone can see at a glance what's going on in any area, either by visiting physically or by using the network. I've seen (more than a few times) workers from a cell discussing the info on the boards, ribbing each other over the NCM sheets and attendance records. Staff is LOOKING at the metrics, which is always the first step.

 

[Michael_M]

I may have been a little confusion, for that I apologize.

I am trying to put together a list for those who are reviewing the numbers, not for employees (at this point--that will come later). We have allot of data right now. To me, the data is meaningless as there is no format or 'what do we do with the data'.

What I am looking for is recorded guild lines as to what we find to be acceptable at this time. Since it has to be recorded, I am looking at 'where do I record these numbers to keep them official', or do I even have to keep them officially?

 

[Steve Prevette]

I may have been a little confusion, for that I apologize.

I am trying to put together a list for those who are reviewing the numbers, not for employees (at this point--that will come later). We have allot of data right now. To me, the data is meaningless as there is no format or 'what do we do with the data'.

What I am looking for is recorded guild lines as to what we find to be acceptable at this time. Since it has to be recorded, I am looking at 'where do I record these numbers to keep them official', or do I even have to keep them officially?

Yes, we seem to be talking past each other. I am in a group of 3 statisticians publishing more than 3,000 charts per month for my employer. I see all sorts of variations on the "where do I record these numbers to keep them official". In some cases, it is a formal report to management. In other cases, there is a "scorecard" software that records and displays the results. In the simplest case, it is simply an email with the charts to the manager. So I don't see this as a critical question, unless your ISO requirements specify something more formal.

Yes, I agree that "the data is meaningless as there is no format or 'what do we do with the data'. That is why I do suggest you look at the SPC format. You may find some examples at http://www.efcog.org/wg/esh_es/docs/DOE_Charts/elect_events_charts.htm. This is a good example of storage of charts to be retrieve by anyone within the national Department of Energy complex.

Theory may be found at http://www.efcog.org/wg/esh_es/Statistical_Process_Control/

 

[dsanabria]

I am starting to 'officially' establish metrics that are to be reviewed on an on-going basis. As part of the metrics, I have to establish 'good' range and 'bad' range.

For example, job time entries (how many parts are produced in XX time).

I am not sure where the best place to put these numbers at. I don't want to put it in the actual Quality manual nor can I think of a specific procedure that the range of numbers should be put into (staying with the job time entries as an example).

I was thinking of writing up Work Instructions on how to review the data but that seems a bit too micro-management to me since some data would have some subjectivity to it, does the operator run 1 machine, 2 machines (slows production down a bit), or run 3 machines (slows production down a bit more, but 3 jobs are running).

So the actual question: where would I put numbers we call 'good', 'follow up', or 'bad, need to fix' so they are somewhat official?

I don't know if this helps, but we have:
QM
Procedures
Work Instructions
Forms

Have you looked at "Cost of Quality" concepts - management would love you. Then step into a kaizen event of documents / procedures. Lean of processes would be a step behind.