There's a movement afoot to choose some non-financial performance metrics that the QA department will track. I'm under pressure to pick something simple that is meaningless but easily trackable.

I'd like to choose something that we could actually work to improve to our benefit, but everything I'd like to improve is an issue that is outside of our direct control.

Do any of you have any metrics regarding your department's performance that you're tracking that you are content with or happy about?

Thanks
Norman Lutes

Your departmental PPM.

Well, we're not exactly a million output kind of outfit. Ruggedized computers for the military.

I already track flaws found at inspection, and flaws found at test. I have considered tracking flaws that get past inspection that COULD have been caught by inspection but got found in testing, but that would generate a blip every few months and nothing more.

Metrics should be 1) quantifiable, 2) observable and 3) meaningful.

If the metric is not meaningful, then why bother doing it?

I've used non-financial metrics to highlight processes that need improvement, for example. How many hours are being spent writing up defects--can that process be improved to reduce that burden? What is the DPMO by product--is one particular product in need of some help? What is the warranty return rate--are things that should be caught before shipment being missed, does a test that is being performed need to be changed or added?

All of the root-causes of these data may be out of your control, but if you can highlight the need for change (and something is done to address that need), then the metric is successful.

Greg

Other options for performance metrics we are using is:

Target percentage of corrective and preventive action requests per:
• Reporting Source
• Type
• Subject

There's a movement afoot to choose some non-financial performance metrics that the QA department will track. I'm under pressure to pick something simple that is meaningless but easily trackable.

I'd like to choose something that we could actually work to improve to our benefit, but everything I'd like to improve is an issue that is outside of our direct control.

Do any of you have any metrics regarding your department's performance that you're tracking that you are content with or happy about?

Thanks
Norman Lutes

In these days, there are very few efforts in a corporation that are within anyone's "direct control". If your measure is "meaningul" it is not likely under your direct control. So, go with it. If the measure reflects something important that QA has a role in, measure and learn from it. You may need to team with another department in order to deal with it, but "it" will become a win for the company.

Well, perhaps I've not expressed my original sentiment very clearly. Let me try again.

I believe that tracking quantifiable issues and seeking to improve trends is a good thing. I do it all the time.

My boss has been asked to set some criteria for his own performance that he can be measured against, and the same for his direct reports. I'm not certain if the goal of the exercise is to improve where improvements are possible, or if it's a holdover from the "merit pay will be based on performance against metrics" philosophy.

My boss is encouraging us to choose metrics within our control. My problem is, everything that is within my control is mundane and of small value to improve. For example, conformance to a three day time window to get my data entry and analysis done AFTER I receive the raw data from Production.

The exercise is academic. If I get it done in three days, it's because the rush is over and there was time to do it. If I fail to get it done in three days, it's because the rush was still on and I'd have to stop shipments to do the data entry and analysis.

There are plenty of things I can and do track that involve other departments, but I've been asked to choose goals germane to the QA department.

My boss has been asked to set some criteria for his own performance that he can be measured against, and the same for his direct reports. I'm not certain if the goal of the exercise is to improve where improvements are possible, or if it's a holdover from the "merit pay will be based on performance against metrics" philosophy.

There are plenty of things I can and do track that involve other departments, but I've been asked to choose goals germane to the QA department.


I would say that selecting metrics in a vacumn is futile. The process approach would have you define the important criteria for this Quality process first (4.1.c). When you define what is important, those cirteria will essentially define for you what should be measured (4.1.e). For some reason, we frequently get it in the wrong order, and it becomes somewhat undefined or even arbitrary.

If you can tell me the important criteria, then I can easily tell you what should be measured.

The process approach would have you define the important criteria for this Quality process first (4.1.c). That would assume there is such a thing ... I can't for the life of me think what a "Quality process" is there are plenty of business processes that members of the quality department are involved in but I thionk we are in danger of going down the "IT management" argument.

I think you all have good points, but I don't think I'm getting the answer I was looking for to the question I posed :

[Do any of you have any metrics regarding your department's performance that you're tracking that you are content with or happy about?]

Which is not necessarily a bad thing, it just means that there may not be an answer I want to hear to the question as I posed it. :(

This thread is about to fall off the bottom of the active list, and with [45 Registered Users and 101 Unregistered Guests] currently in the Forums, I'm going to have to assume that nobody here has a metric they're satisfied with that reflects their departmental performance that they currently are tracked and held to.

I find this discouraging. I could track something like service response time, but since it's either instantaneous or "as soon as I'm done with this fire I'll come put out yours" it would be a moot point.

Help me out, people, I'm dyin' up here :bigwave:

Customer complaints-number & type
Returns as a percent of sales
Reduce scrap and rework from_to_ by Dec.2006
Train dept. employees in SPC during month of _


Not sure if these are any help either, but posed anyhow in case it might

Ralph had one that we use: returns as a percent of line items shipped. Here are some others that I won't necessarily vouch for as rock solid: DMR / NC material dwell time (time to closure) \ DMR / NC material value ($$$$). For auditing we track Ontime and Ratio of Minor / Major findings per month (i.e. 6 majors over 6 audits = 100% rate of major findings). Dunno if any of these are effective or controllable but these are what we track currently in our QMS that I relate more directly to the QA function.:cfingers:

There's a movement afoot to choose some non-financial performance metrics that the QA department will track. I'm under pressure to pick something simple that is meaningless but easily trackable.

I'd like to choose something that we could actually work to improve to our benefit, but everything I'd like to improve is an issue that is outside of our direct control.

Do any of you have any metrics regarding your department's performance that you're tracking that you are content with or happy about?

Thanks
Norman LutesI found this insightful because of the line "I'm under pressure to pick something simple that is meaningless but easily trackable."

Do you have evaluations with line items listing behaviors to which employees are graded? If so, it seems fair to consider those established factors as a starting point. I have noted that some evaluation lines are indeed pretty meaningless. :rolleyes:

It's quite true that there are few functions purely under QA's control. That's because we are a service/support function. How fast, how often, how correctly we perform tasks can very often be tied to member(s) outside of our direct control.

That may be why there's increased scrutiny on effective performance metrics and repsonses to them. You say you suspect this may be lingering from an old merit-based pay system. Is there a reason why you can't/shouldn't inquire about that--what is this initiative's goal?

If it's not part of merit-based pay, QA's modern role is in helping various functions perform better via the Theory of Constraints. There may be an opportunity in spite of few areas under your department's direct controls. Can you make a metric that requires liaison efforts--that is, helping another department, process or function to perform in greater alignment with the organization's mission and strategic objectives?

To sum it up, I'd want more information on what the big idea is so as to make a value-added response or establish the entire thing is without value. It's hard to march well until given even a few bars of the tune.

I found this site for developing performance metrics.
http://www.orau.gov/pbm/documents/overview/uc.html

Good Morning;
Can you get the information to support: the number of data entry mistakes per data entry? This would give you a measure of accuracy. You could also measure the time to re-do the data entry - and compare to volume of work over available time. This would rise and fall with your workload. You could also monitor number of mistakes in proportion to number of entries within a specific timeframe.

This gives you measurables, and also can lead to continuous improvement activites and workload distribution information.

The story has a happy ending. I've been dragging my feet on choosing knee-jerk metrics, searching for an appropriate choice.

And apparently my boss has received clarification as to the nature of the mission, and an appropriate schedule for choosing, reviewing, and implementing APPROPRIATE metrics.

So, while I'm off the unpleasant hook for the moment, I'll wind up with more work in the long run, but at least it will be "the right thing for the right reason".

Thanks
Norman :bigwave:

The story has a happy ending. I've been dragging my feet on choosing knee-jerk metrics, searching for an appropriate choice.

And apparently my boss has received clarification as to the nature of the mission, and an appropriate schedule for choosing, reviewing, and implementing APPROPRIATE metrics.

So, while I'm off the unpleasant hook for the moment, I'll wind up with more work in the long run, but at least it will be "the right thing for the right reason".

Thanks
Norman :bigwave:

More work?!?! Sorry, but I disagree...it's more like "right work" (in my opinion). For me, I report metrics focusing on the state or health of our Management System. Everything from Customer complaints to overdue nonconformances to out-of-date metrics from other departments.

Define "appropriate"...appropriate for you, the department or the company? What about tracking the status of projects you're working on?

.....I'm going to have to assume that nobody here has a metric they're satisfied with that reflects their departmental performance that they currently are tracked and held to.

I find this discouraging. I could track something like service response time, but since it's either instantaneous or "as soon as I'm done with this fire I'll come put out yours" it would be a moot point.

Help me out, people, I'm dyin' up here

normzone

My department is quality assurance, but we do not have a process called "quality assurance"

Thus we have no process metrics for quality assurance.

The manufacturing process tracks defects. Shipping tracks late orders, etc.

But none for us, we are a pure staff/support function.

Surprised Deming hasn't raised his head here.....abolish numerical targets!