How do you folks meet this requirement?

There are lots of ways to 'measure' training effectiveness.

It depends upon the training. On the job training versus (for example) managers training. You have to look at the training. Some OJT is a matter of observation by the trainer. Some OJT (particularly for 'hi-tech' jobs) is verified by certification testing.

I suggest you check out the following from the 'old' forum:

A Short Relevant Thread (http://Elsmar.com/cgi-bin/board.cgi?read=2865)

and

A *Better* Relevant Thread - Do read this one! (http://Elsmar.com/cgi-bin/archiveboard.cgi?read=2243)

Then come back to this forum with more specific questions! Hope this helps!

Mr. Stevan and Mr. Mark,

I would like to share with you one of my approaches for measuring Training effectiveness. As such, measuring training effectiveness is highly subjective but some sort of approach can be followed as follows:

1. One week after the completion of the training programme, a feedback should be obtained from the students/participants on how much of knowledge they had gained during training are they retaining. We all know, that we tend to forget things if not implemented immediately. This feedback is only to check whether they have got sufficient knowledge for implementation. People generally forget 70% of things they learnt within one week.

2. After 3 months, a feedback should be obtained for the effective implementation of the training got. This is to check how they are using what they have learnt. Whether any additional training has to be given or some sort of help is required.

3. After 6 months, the training effectiveness should be monitored using Performance Appraisal by the respective Department Managers. The effectivenss of the training they have got should be linked to their appraisal system, because I feel that then only the training will be very effective.

The above is one of the many methodologies. I am expecting your comments on this.

Subject: Training Effectiveness
Date: Thu, 16 Sep 1999 15:53:25 -0400
From: "Kelly Speiser"
To: "QS List Post"

QS Listserv Members: I pass this along from a Training Listserv for which I am a member. I thought this is of specific interest to those complying with ISO.

"If you are involved in measuring the effectiveness of training, this might be of interest. The U.S. Department of Energy (DOE) and Westinghouse are making TRANS-MEASURE, a transfer of training measurement and improvement tool, available to U.S. businesses, consultants, educational institutions, government agencies, non-profit organizations, and individual citizens at no-cost to recipients through a federal sharing program. TRANS-MEASURE provides organizations with the following information for their training courses:

+ Transfer of Training Rate -- the percentage of skills successfully transferred from the learning setting to the job.
+ Prior Knowledge Rate -- the percentage of skills that trainees already knew how to perform before taking the training.
+ "Didn't Stick" Rate -- the percentage of skills presented in a training course that the trainee cannot perform six months after training.
+ Value-Added Rating -- the amount of skills critical to the success of the organization learned during the training and performed frequently on themjob.

TRANS-MEASURE is an streamlined, user-friendly offspring of the Westinghouse-developed Transfer of Training Evaluation Model. This is a complete, turnkey package that can be administered by nearly every type of organization with no modification necessary. To obtain TRANS-MEASURE at no cost, fill out an application at the DOE/Westinghouse technology transfer website

Forwarded to you by
Kelly Speiser
Jackson MI

I am interested in the Training Effectiveness Model mentioned above, i have registered and requested the notes, but being Ireland based i am not sure they will give it to me.

Can anybody tell me what it is all about?. Is it a software package, or a questionnaire that tests employees on what they have learnt from the training.?

Regards



------------------
Andy B

I don't know myself. You might want to contact the sender of the info --> "Kelly Speiser"

Just downloaded the info Kelly is talking about; passed it around and received positive feedback for content and "user friendly" format.

Does anyone have any backgorund on how to determine effectiveness of training? We were told during audit to pre and post test after training. Does anyone have any examples of these types of tests that I could take a look at? THANKS!!!!

Why not do an observe/record proficiency check?

Or take the same test before and after...BUT...be ready to do something about it when the delegates have the same score before and after!!!!

Most public offerings do an effectivity evaluation survey..... That works.

examples.....there must be millions, all task specific and probably of no value to you. take the training offered, or SOP, etc., and create a test that covers the important points/requirements to be covered..(the goals of the training?) Give it prior to the training, and again after...chart the results....you should see marked improvements to show that the training ws effective. for workmanship reinforcement/training you can monitor yeilds.....before and after training.

I do a proficiency record to verify grandfathered training/or lack of records....Also use it as a tool to verify training effectity when required as followup to CA requests that have training identified as teh root cause.

Doesn't tell you how, just that you must...what would add value and not be a useless tool?

[This message has been edited by barb butrym (edited 06 March 2000).]

A good suggestion Barb. Trending both sets of tests (before and after) is a nice thing to show auditors. Picture, picture.

Kev

One of the easiest ways I've found is to show that the operator is not responsible for any errors after training is a review of the NC system. Example:

"We trained the technician at this station how to solder. We know the training was effective as we track problems found with solder joints and a search of the NC database shows the operator in question was not responsible for any 'bad' joints."

Make sure you look at the specific situation. Sometimes - yes, a test is appropriate. Especially if the risk is high (failure will be expensive, dangerous, etc.) For example in a wafer fab one mistake can cost bundles of bucks. In addition, arsenic is used ion the process. What do the companies do? Not only train but retrain and re-certify (test) every 6 months.

In the case of the soldering - let's say it's a simple light switch - I doubt you'll have to retrain every 6 months. Whether or not you need a test is - well, I doubt I'd require a test. Now if the soldering was for a Mil-Spec item (let's say a wiring harness for one of the space shuttles or a bomb fuse) I would want a test.

Remember common sense, folks.

Marc,

I like your suggestion. It is quite simple. You rely on other aspects of the system to confirm effectiveness of training in other areas (in this case, NC system confirms Training). Organizations should sell this point to a registrar. You will win, provided the aspect you are relying on is not under-developed.

Regards,

Kevin

Date: Wed, 15 Mar 2000 12:39:09 -0500
From: Mike Tillmans
Subject: Re: TQM--- tRAINING EFFECTIVENESS MEASUREMENT

On 3/13 Anil writes:

<< I would be very thankful if you could give your views on these points:

How is training effectiveness measured?
Is it done through a questionnaire? or by the boss on the performance?
After what time frame this is to be done?
Any standard/Guideline available to control this .>>

Several years ago Kirkpatrick identified four levels of measures of training
effectiveness that are widely used in business training. Although I'm not
restating them exactly, you'll get the picture.

Level 1: student opinion--collected in end-of-course student questionnaires

Level 2: student learning--test data collected by performance tests at the end ofthe course or by activities during the course

Level 3: impact on the job--follow up survey/test/study is done to determine how well the skills are applied on the job

Level 4: impact on the business--study is done to see how the skills contributed to the business growth/profitability/customer service/etc.

Each level gives you more important information, but each level is usually harder (read: more time consuming and expensive) to collect and interpret. If this looks useful to you, I'll dig out the original reference and/or some of the recent articles written on his approach.

Mike Tillmans

From: "Anilkumar"

ISO 9001 DIS calls for measurement of Training Effectiveness.

How is it done generally?
Does it mean giving questionnaire at the end of the session
or the performance evaluation by the Boss after the course .

Is it reliable?
What is the approximate number of companies certified to ISO 9000?
Where can I get statistics on this for each country? >>

--------------snippo---------------

From: ISO Standards Discussion
Date: Wed, 15 Mar 2000 08:42:25 -0600
Subject: Re: ISO 9000- Training Effectiveness /Anilkumar/Kiely

At a recent ISO 9001:2000 Transition Seminar that I attended the issue of "effectiveness" was a hot topic of discussion. This seminar was attended by quality management professionals from all levels as well as by Auditors from our assessing body. When asked, the Auditor's stated that they would by definition be looking for some type of "quantifiable" measure to illustrate what the Organization (formerly called the supplier) considers "effective"... The Auditors stopped just short of saying that a test score would suffice... I believe the more pressing issue however will be with how our individual assessing bodies interpret the words "competent" and "competency" (hopefully these words will be dropped from the final draft).

Gerard Kiely, CQA

---------snippo-----------

From: ISO Standards Discussion
Date: Wed, 15 Mar 2000 08:49:21 -0600
Subject: Re: ISO 9000- Training Effectiveness /Anilkumar/Kozenko

A training evaluation document at the end of training is a valuable tool to collect information on what was good, and not so good, about the training conducted. It helps the training writer improve the training program itself, and it shows the differences between one trainer and another, particularly on the same subject -- so, over the course of time, you can use the trainers who do best on "this" subject to train it, and if they're not so good on "that" subject, use some other trainer who is.

Training is often overlooked in root cause analysis. The DIS Standard forces training to become one of the mandatory "look-see" areas when evaluating root cause for corrective action.

For the rest of your questions, I would recommend www.qualitydigest.com (http://www.qualitydigest.com) -- use the database of registered companies that is linked off that home page.

David M. Kozenko

--------------snippo----------------

From: ISO Standards Discussion
Date: Wed, 15 Mar 2000 08:52:32 -0600
Subject: Re: ISO 9000- Training Effectiveness /Anilkumar/Humphries

Anilkumar,

> ISO 9001 DIS calls for measurement of Training Effectiveness.
>
> How is it done generally?
> Does it mean giving questionnaire at the end of the session
> or the performance evaluation by the Boss after the course .

This is one way. However, remember that in the ISO section related to Measuring Equipment are provisions for Go/No Go gauges: a pass/fail measurement is still a measurement. So a review by supervisor followed by decision as to Competent/Not Competent is also a measurement.

Best Regards

From: ISO Standards Discussion
Date: Thu, 16 Mar 2000 09:41:22 -0600
Subject: Re: ISO 9000- Training Effectiveness /../Arbuckle

Without trying to present a seminar on training and training evaluation, let me see if I can help here. The first step in evaluation training is to establish what the outcome is supposed to be. This is done by defining objectives for the training itself, whether they are knowledge or behavioral based. With objectives defined, then the measurement and metrics can be established to determine if the training had the desired effect. All of this can sound very time consuming until you apply it realistically to real business.

Typically training is offered as a method for change. If the change happened, then the training was effective. For example, the change is to enable the employee to operate the equipment according to procedure(s) XYZ001 (and XYZ003...XYZ005). When the training is completed, the trainer, or other authorized personnel, evaluates if the employee can perform the operation as required by the document. Whether the training was classroom or OJT, the verification is the same...the evaluator reviews performance and determines if it is done correctly. If so, then the training was effective. If not, then the training was not effective and needs modification (time for a CAR?).

For those of you familiar with the Kirkpatrick Model of Evaluation of Training, we are really talking about a level four evaluation. (1=reaction; 2=learning; 3=behavior; 4=results). While Kirkpatrick goes into more detail than I would ever suggest to any of you, it is a good model to follow. For more detail please feel free to contact me.

Don

--------------snippo----------------

From: ISO Standards Discussion
Date: Thu, 16 Mar 2000 09:43:50 -0600
Subject: Re: ISO 9000- Training Effectiveness /../Kiely/Humphries

> At a recent ISO 9001:2000 Transition Seminar that I attended the issue of
> "effectiveness" was a hot topic of discussion. This seminar was attended by
> quality management professionals from all levels as well as by Auditors
> from our assessing body. When asked, the Auditor's stated that they would
> by definition be looking for some type of "quantifiable" measure to
> illustrate what the Organization (formerly called the supplier) considers
> "effective"...

Why am I generally unsurprised? ISO does NOT specify quantifiable measures ANYWHERE; but I'm sure you're right: auditors will take it on themselves to define how their clients must interpret the Standard. I guess it's up to us (still) to defend the territory of "reasonableness".

> The Auditors stopped just short of saying that a test score would suffice...

I would hope so (although what other type of quantifiable measure is there?). Can you imagine running tests in every aspect of training?

> I believe the more pressing issue however will be with how
> our individual assessing bodies interpret the words "competent" and
> "competency" (hopefully these words will be dropped from the final draft).

I have less problems with these: the world is heading towards using these concepts, and they have a value if implemented reasonably (that word again!).

Best Regards
Edwin

From: ISO Standards Discussion
Date: Thu, 16 Mar 2000 09:55:38 -0600
Subject: Re: ISO 9000- Training Effectiveness /Anilkumar/Bradley

Anilkumar writes:
< ISO 9001 DIS calls for measurement of Training Effectiveness.

How is it done generally?
Does it mean giving questionnaire at the end of the session
or the performance evaluation by the Boss after the course .

Is it reliable?
What is the approximate number of companies certified to ISO 9000?
Where can I get statistics on this for each country? >

There are four levels of training measurement:

1. Participant satisfaction. While this may not seem to indicate effectiveness, in a back-door sense, it does. If the participants are dissatisfied with the training, they will remember less, and will likely not want to attend future sessions.

2. Knowledge gain. Providing a pre and post test will determine if the participant gained any new knowledge during the class. Testing at class end, and six months later will determine if the training "stuck".

3. Job performance. After training, can the participant demonstrate the new skills. This should be important in the ISO world! Periodic performance reviews may be an excellent way to determine training effectiveness, as well as identify new training needs.

4. Increased profitability. I believe this is the hardest to quantify, because training is only one factor in profitability, and the other factors tend not to remain stable.

Now this is a real important part. I did not make this up. I read it in a book many years ago. I am going to show a bit of my age here, a "Senior Moment" is preventing me from remembering all of the details; mainly the person who wrote the book. I'm certain someone in the group will refresh my memory. I tend to remember what books are about (the lesson), but not the details about who wrote them.

David

I am looking to prove my training effectiveness, without having to write a test each time we do training. I have a Manager Responsible for Training that is Training happy, every internal audit finding sends the foreman out to re-train. Does anyone have a typical Evaluation form, or any other sugestions. My Reg Audit Jan 25th & 26th, I will be testing my lab personnel on SPC Monday to satisfy, but I would like to havesome other ideas.
Thanks to all in advance.

Are the people doing their work as they were trained to do? If so, the training provided is effective. This can be verified using:
internal audits,
suitable metrics,
and peer/supervisor review.

ANND,

Do you have any internal audit results, management review records, or employee review records that could help show training effectiveness?

Good luck with the Audit!

ASD...

Thanks that helps.

That is the way we had intended to go but my Auditors keeps bring up an eval. like the one I did on him. I know my employee evals are not clear enough yet but I can clear that up.

Thanks again.

I always point out customer returns data and results (data) of inspections and tests. If folks are not doing their job correctly (whether as a training issue or as a work instruction issue) both of these should be flagging the problem(s).

Take soldering. If you train someone and inspections / tests and customer returns investigations do not point to soldering problem(s), it is obvious that the soldering training was (is) effective.

We make mention in our Training procedures that we define peer/supervisor evaluations as the measurement of training effectiveness - and follow that with a line at the bottom of the eval forms that speaks to effectiveness, including identifying any future training needs and stating that the effectiveness of this training will be evaluated at the next scheduled performance evaluation

I have to chime into this training issue I am a staff member at UAH in alabama and am always troubled with the lack of investment most companys apply to any training. So be it ISO or lean its better to train your employees and lose them than to not train them and keep them. Remember training is the only business expense that dosent depreciate after its used. Thanx!

You might want to first try a search here in the Forums. Use 'training effectiveness' (without the 's) as your search terms. There are a number of threads on training effetiveness in the QS-9000 forums - and in the ISO forums as well, I believe. As an example, see Training Effectiveness (http://Elsmar.com/ubb/Forum1/HTML/000414.html)

[This message has been edited by Marc Smith (edited 22 July 2001).]

[i]Edited 27 December 2001

Marc,
This will help me to start my implementation and my new ISO/QS procedures

Thanks
Alejandra

As I said - that's only 1 link. A search will provide several more extensive discussions, including Measuring Training Effectiveness (http://Elsmar.com/ubb/Forum1/HTML/000004.html)
and
Training Effectiveness (http://Elsmar.com/ubb/Forum1/HTML/000259.html)
and
Training Effectiveness - 1 (http://Elsmar.com/ubb/Forum28/HTML/000075.html)

To name a few.

Edited 27 December 2001

Originally posted by Kevin Mader

I like your suggestion. It is quite simple. You rely on other aspects of the system to confirm effectiveness of training in other areas (in this case, NC system confirms Training). Organizations should sell this point to a registrar. You will win, provided the aspect you are relying on is not under-developed.I always used it as at least one aspect of proof of OJT effectiveness. I've never had a registrar argue or object to it.

Also see: How do I measure Measuring Training Effectiveness? Performance evaluation? (http://Elsmar.com/Forums/showthread.php?t=4002)

TRANS-MEASURE is an streamlined, user-friendly offspring of the Westinghouse-developed Transfer of Training Evaluation Model. This is a complete, turnkey package that can be administered by nearly every type of organization with no modification necessary. To obtain TRANS-MEASURE at no cost, fill out an application at the DOE/Westinghouse technology transfer website: ***DEAD LINK REMOVED***

This is just a note to say this link is NOW DEAD.

There are some other great training downloads also available. HazCom, BloodBorne Pathogens, Safety, etc. Great link! Thank You. (My application is still pending):bonk: :smokin:

Originally posted by energy

Great link! Thank You.That'll be 5 cents for the consultation, please.

From: ISO Standards Discussion
Date: Thu, 16 Mar 2000 09:41:22 -0600
Subject: Re: ISO 9000- Training Effectiveness /../Arbuckle

Without trying to present a seminar on training and training evaluation, let me see if I can help here. The first step in evaluation training is to establish what the outcome is supposed to be. This is done by defining objectives for the training itself, whether they are knowledge or behavioral based. With objectives defined, then the measurement and metrics can be established to determine if the training had the desired effect. All of this can sound very time consuming until you apply it realistically to real business.

Typically training is offered as a method for change. If the change happened, then the training was effective. For example, the change is to enable the employee to operate the equipment according to procedure(s) XYZ001 (and XYZ003...XYZ005). When the training is completed, the trainer, or other authorized personnel, evaluates if the employee can perform the operation as required by the document. Whether the training was classroom or OJT, the verification is the same...the evaluator reviews performance and determines if it is done correctly. If so, then the training was effective. If not, then the training was not effective and needs modification (time for a CAR?).

For those of you familiar with the Kirkpatrick Model of Evaluation of Training, we are really talking about a level four evaluation. (1=reaction; 2=learning; 3=behavior; 4=results). While Kirkpatrick goes into more detail than I would ever suggest to any of you, it is a good model to follow. For more detail please feel free to contact me.

Don

--------------snippo----------------

From: ISO Standards Discussion
Date: Thu, 16 Mar 2000 09:43:50 -0600
Subject: Re: ISO 9000- Training Effectiveness /../Kiely/Humphries

> At a recent ISO 9001:2000 Transition Seminar that I attended the issue of
> "effectiveness" was a hot topic of discussion. This seminar was attended by
> quality management professionals from all levels as well as by Auditors
> from our assessing body. When asked, the Auditor's stated that they would
> by definition be looking for some type of "quantifiable" measure to
> illustrate what the Organization (formerly called the supplier) considers
> "effective"...

Why am I generally unsurprised? ISO does NOT specify quantifiable measures ANYWHERE; but I'm sure you're right: auditors will take it on themselves to define how their clients must interpret the Standard. I guess it's up to us (still) to defend the territory of "reasonableness".

> The Auditors stopped just short of saying that a test score would suffice...

I would hope so (although what other type of quantifiable measure is there?). Can you imagine running tests in every aspect of training?

> I believe the more pressing issue however will be with how
> our individual assessing bodies interpret the words "competent" and
> "competency" (hopefully these words will be dropped from the final draft).

I have less problems with these: the world is heading towards using these concepts, and they have a value if implemented reasonably (that word again!).

Best Regards
Edwin

I agree. Very often, the objective is assumed, but not clearly defined. Without a clear objective, what do you compare "effectiveness" to? You haven't defined the expected outcome, so it is difficult to determine if you achieved "it."

Also, neither the ISO 9001 or TS-16949 standard says that training effectiveness has to be measured. They do state (in various subclauses) that it has to be determined, evaluated, ensured and qualified. This then permits the many other good methods to determine or evaluate effectiveness that others have already shared in this thread. Auditors cannot legitimately demand that every process be measured. That is a common convention, but not a requirement.

In another good thread the practice of evaluating training effectiveness based on process results (good product or output) has been explored.

Indeed, where a process output depends on training to be good, one can argue that the output alone is sufficient to qualify an operator.

What is left is the relative success in arguing that practice with an auditor, some of whom are less easily satisfied with no clear evidence of establishing or acknowleging competency.

It may be enough to organize data in such a way that training (or an incoming operator's competence) is the lone dependent variable. In other words, that there is little to no chance that the process output would be satisfactory without the competency. I expect the degree to which this would work depends on lots of things: the tasks and process mistake proofing, for example.

So perhaps it is a defensive mechanism to "prove" competency by noting its establishment through training, observation of successful task execution, and perhaps an occasional refresher depending on the variable critical nature of that task. Training records exist to show a structured method in approaching this and don't require persuasiveness to breeze through an audit.

Let us also consider the method as part of the audit process output. If I, the training manager, were to be hit by a bus while getting my mail I would arguably have been deficient if I had left my system in a manner that an underling or associate could not pick up where I left off. The outcome as evidence of training seems to me a precarious approach unless any representative could defend it as successfully as I would. Ensuring that happens is also a form of mistake proofing.

And so the outcome method could work, but I am not comfortable with it because I am concerned it would not prove durable when subject to scrutiny.

Since there are likely already some training records required for regulations like OSHA, IMHO it seems reasonable to also include tasks that are critical to output quality and make it a clearly functioning program. The trick is not to overdo it; why make your life hard by having training records for noncritical tasks? Misunderstanding this has caused some organizations to bloat their training programs and spark a rebellion against organizing training.

I agree with your thoughts, Jennifer. I think you expressed them well. I would add two comments.

1. As an auditor, I really only have two options for nonconformities in this process. If I discover certain people are obviously not knowledgeable or competent in their responsibilities or there is no "reasonable" methodolgy of evaluating competence. You are right, there are some auditors who may go to excess in this, but I would advocate that where the standard is vague, reasonable is the best fallback option. As intelligent professionals, we have to try to reach an agreement as to what is reasonable. As an auditor, I don't want to impose my opinions on your system. You have to live with it, it has to meet your needs. If an auditor cannot accept these constraints, then I think he is not performing well.

2. Regarding your comment: The outcome (approach) as evidence of training seems to me a precarious approach... I agree. If you use the "outcome approach" as evidence with which you render a judgement that that particular employee has demonstrated competence in his task, I would accept that. But I would expect to see records that you created to demonstrate that you have made that determination. A good example of this outcome approach is forklift training. There generally is some kind of written test, which is beneficial and is usually in the employee's training file. However, the acid test is always an actual demonstration that they know how to operate the forklift properly. We tell them to hop on and show us what they can do. If they fail the actual driving, they wouldn't get their "license," regardless whether they passed the written. That is "outcome" based. However, I would expect to see a record of that successful demonstration in the employees file, and would expect to see timely renewals, as required by OSHA.

PS: I suggest you evaluate "proficiency" rather than competency. It is a much better goal than basic competence.

proficiency

I suggest you evaluate "proficiency" rather than competency. It is a much better goal than basic competence.

By so doing you're going beyond the requirements of the standard (not necessarily a bad thing, of course) and creating a requirement for measurement that might not be necessary. In the forklift example where life and limb are at stake, proficiency should be the target and could be measured by testing individuals on something more than moving forward and backing up. Whether or not to create a measurement requirement is part of design of the system, and things that don't need to be measured shouldn't be.

I completely agree. I believe I stated the same thing in my last post. "Measuring" training is not a requirement of the standard. I am certainly not trying to add a requirmeent the standard does not require.

The forkilift discussion was intended as an example of "outcome based evaluation of training" that Jennifer described in the post before mine. If the fellow could properly operate the forklift, then the training was deemed effective. If he could not, then the training would be deemed not effective. The outcome demonstrated whether the training was effective, not a measurement. It is just an example of how most companies use some form of outcome based results to determine training effectiveness.

The discussion of proficiency was not a requirement. I merely suggested it is a better goal than basic competency. If a company is trying to achieve zero defects, competency won't get them there. There would need to be a high degree of proficiency in skills across the board to achieve that. Competent teams don't make it to the Super Bowl.

The discussion of proficiency was not a requirement. I merely suggested it is a better goal than basic competency. If a company is trying to achieve zero defects, competency won't get them there. There would need to be a high degree of proficiency in skills across the board to achieve that. Competent teams don't make it to the Super Bowl.
Sometimes it's a better goal than basic competency. Sometimes basic competency is all that's needed. And goals that extend beyond what's needed, and call for measurement of things that don't need to be measured, are wasteful.

A team can make it to the Super Bowl with an outstanding defense and basically competent offense and special teams (witness the '85 Bears, a team near and dear to my heart.) And none of the teams who have ever won the Super Bowl have ever achieved zero defects. No one ever has, and no one ever will, unless "defect" is redefined to exclude all defects.

I do very much agree with the idea to avoid overkill, summed up nicely as "Sometimes basic competency is all that's needed. And goals that extend beyond what's needed, and call for measurement of things that don't need to be measured, are wasteful."

I do very much agree with the written test, as appropriate, and a demonstration of skill within defined expectations. Record these events and ensure they happen in a timely fashion, and it should be plenty. The method matters less than that the recording acknowlegement of necessary skill occurs.

The standard is intentionally vague on recordkeeping requirements, since organizations' needs will radically differ according to their tasks, the industry, end product/service needs, safety, etc. It is not necessary, indeed it's worse than wasteful to overdo records for tasks that don't impact output quality because the records that are necessary may then not get the needed care.

The organization is required to define the needs, and according to them make records, or whatever approach appropriate to those needs. The expressions of concern about "What ISO/auditor(s) wants" conceptually places the cart before the horse.

We're not supposed to be satisfying a standard, we are supposed to be using the standard as a guide to build a system that satisfies our needs. This misunderstanding creates a sense of need to do a lot of extra work in records establishment and maintenance.

To add to this:

I would like to see organizations apply some flexibility in how they acknowlege skill. A written test given to a dyslexic person (as one of many good reasons why some people are poor readers) may unnecessarily "prove" the person's incompetence when in fact he/she may be perfectly capable. I hope organizations allow tests to be taken verbally, which would require the organization to recognize that learning disabilities may oftentimes have a non-issue effect on capability. Keeping a record of competency through verbal testing is not necessarily more complicated than if they had taken a written test.

Another way to measure training effectiveness is by looking at the findings you receive from both internal audits and external audits. If the root cause of the finding(s) is because people fail to follow procedures, then training may not have been effective (or the procedures may not have been correct in the first place).

Another way to measure training effectiveness is by looking at the findings you receive from both internal audits and external audits. If the root cause of the finding(s) is because people fail to follow procedures, then training may not have been effective (or the procedures may not have been correct in the first place).Very true, both points. I covered this in my paper in The Reading Room. There are lots of reasons why training can fail to satisfy an effort to improve the process.

I cringe when training is called out as a root cause because I almost never feel sure the troubleshooting was carried out thoroughly enough.

If training is indeed needed, what kind of training would be the right kind? I submit that developing truly effective training is possibly as great an engineering feat (human performance engineering, don't laugh) as achieving process capability.

I am often asked, "Why did you get a degree in Education?" I had thought to work within public education, but I now see I would better serve society as a human performance specialist. The more I learn about it, the more I see there is an aching need to do it well within the organization. Naturally this requires knowing when you have hit the mark in training, and knowing why you haven't. The latter is the really tricky part.

I can't seem to access the website for the TRANS-MEASURE program.

Has something changed?

Thanks!

I cringe when training is called out as a root cause Similar to the B3 not accepting Operator Error.
Jennifer I agree for the most part and I have seen training used as a catch all.

I submit that developing truly effective training is possibly as great an engineering feat (human performance engineering, don't laugh) as achieving process capability.

I would say that it is a greater engineering feat. Parts going through a process have less variability than the learning capability across a diverse population! It would be like trying to stamp a part while both the raw material and the die are jumping around.

I cringe when training is called out as a root cause because I almost never feel sure the troubleshooting was carried out thoroughly enough.

Me too!

In my limited experience, I have found that many training applications follow the following format;

Do this, this, this, that, put those together, and do this, this, and that. Now, get you butt in gear, we have to crank these parts out.

In my opinion, "if the trainee hasn't learned, the trainer hasn't taught". (Although many trainers probably disagree with this.)

I believe the biggest problem with training is that we don't train. We just simply bark a lot of instructions to be followed, and let quality take care of the fall-out.

The term "training" should never be used as a reason for root cause, unless of course our objective is to introduce poor training.

So, back to the basic question? How do we measure effective training?

My question is, why would you want too?

Of course in cases of critical processes, welding, soldering, etc.. or any process that requires certification, measured feed-back is given in the form of a grade... pass...fail.

If you pass, this means you are capable of performing this task at the required skill level, else, why did you pass?

Everything we do can be related to either an input, process, or out-put. (I guess I can't get much more generic that that).

If any measured failure occurs at any one of these points, the root cause would typically be "We failed to do what we were supposed to do, at the time it needed to be done". Is this a training error, or did we not follow the process?

If we are to produce a goods or service to a quantifiable level, and we say "training" is the root cause for not meeting this expectation..... Then I say, there is nothing wrong with the "training", rather, there is something wrong with the trainer.

I agree with everything Randy and asutherland just said.

Like many clauses in these standards, if you're not sure what is being asked for, stand back and ask "why is this clause in the standard?".

The purpose of all of the training and competency stuff is obviously to ensure that the people charged with making the product or providing the service are capable of doing so.

As a first pass, why not look at your customer feedback (rejects, complaints, satisfaction surveys etc.) plus your internal rejections, and establish what proportion of your problems were caused by mistakes by people? Then ask yourself "is there any concern within our company that we employ people who are incompetent?"

If the answers are "not many" and "no", then I would say you have competent people and that your training is effective.

I agree with everything Randy and asutherland just said.

Like many clauses in these standards, if you're not sure what is being asked for, stand back and ask "why is this clause in the standard?".

The purpose of all of the training and competency stuff is obviously to ensure that the people charged with making the product or providing the service are capable of doing so.

As a first pass, why not look at your customer feedback (rejects, complaints, satisfaction surveys etc.) plus your internal rejections, and establish what proportion of your problems were caused by mistakes by people? Then ask yourself "is there any concern within our company that we employ people who are incompetent?"

If the answers are "not many" and "no", then I would say you have competent people and that your training is effective.If your definition of a competent employee is one who doesn't make mistakes, then there are no competent employees. I do agree, however, that patterns and trends usually don't lie. As far as training as a root cause of errors is concened, it depends on whether the issue is lack of training or poor training. The former can usually be traced to time constraints, so unreasonable scheduling is a better root cause than lack of training. On the other hand, if plenty of time has been allowed but the training is inadequate somehow, then there could be justifcation for citing training--or operator competence--as a root cause. Root cause should relate to the process that needs to be fixed in order to prevent the problem from happening, and if training needs to be fixed (and there is no "parent" process that controls it that needs fixing) then training may be logically cited as a root cause.

Yeah, thanks JSW05 - you've clarified what I was trying to say.

I didn't mean to imply that you could measure the competence of individuals by the number of mistakes they make. Rather that in your business as a whole, if you are seeing that there aren't many mistakes being made, you've probably got competent people.

I see no problem using this to demonstrate that you understand the intent of the standard. Auditors I've dealt with like the approach, although usually want to see more to back it up.

hhhmmm

I tried to look at the web site and it led to a card game site?:mad:

I can't seem to access the website for the TRANS-MEASURE program.

Has something changed? That link is from around 2001 I think. It's long gone as often happens to things on the internet as the years pass by.

Does this Trans-Measure (from DOE/Westinghouse) still exist? I looked on the web and couldn't find it.

Can somebody help out here?

If it does not exist, is there are replacment or another similar product which is free?

Thanks!

hhhmmm

I tried to look at the web site and it led to a card game site?:mad:Yeah - The link is long dead...

guess that the DOE/Westinghouse technology transfer website is no longer up and running..Any idea where else can i access the tool from?

guess that the DOE/Westinghouse technology transfer website is no longer up and running..Any idea where else can i access the tool from? This site has contact information: http://www.learning-org.com/99.09/0118.html

I tried the e-mail and it was returned as undeliverable. I hope we can get this source back. It looks really neat.