Hi everyone, :bigwave:
Can anybody from the group on how I will measure the training effectiveness? Does a follow-up audit would help to guide me on measuring such training? :confused:
Any feedback would be appreciated.
Thanks in advance,
Raffy

Follow-up audit would be one way.

A "final examination" test could be another.

There are most likely a number of ways to get it documented as Effective.

Raffy,

I read a book about 10 years ago or so by a guy named Kilpatrick (spelling). In this book, he outlined 4 levels of measuring training effectiveness. Quickly;

Level 1: Smile sheets, did the trainee enjoy the training. The worst method, no real effectiveness measured.

Level 2: Pre/Post tests. Better, it measures knowledge gain, but doesn’t answer if the trainee can perform.

Level 3: On-the-job evaluation. This is the best level (IMHO). It asks if the trainee can now perform what was trained.

Level 4: Economic impact: This measures how much the organization gained as a result of the training. Excellent idea, but there are so many competing factors in financial impact, it would be almost impossible to determine (once again, IMHO)

I checked Amazon.com. I couldn’t find any reference to the book, which makes me think I have the name wrong. Can anyone else fill in the gaps?

Dave B (the other Dave)

Hi Raffy,

I have to agree with Dave and say performance evaluation is the way to measure effectiveness. Really gets to the issue...did the employee learn from the training?

CarolX

Yes, It's known as Kirkpatrick's model. Just do a web search on say, Google and you'll find loads of info.

Atul,

Thanks for setting me straight. I was looking for Kilpatrick, not KirkpatricK. That is why I couldn't find anything. It helps to get the name right. But when you get to be as old as I there are somethings you forget.

Dave B (the other Dave):D

1. There's a whole pile of info at the egroup on ROI. This group has been going for about 2 years and covers numerous facets of ROI, including the problmes with Kirkpatrick and his model.

worth a look before drowning in searches on the subject.

http://groups.yahoo.com/group/roinet


2. If you've done any courses in training you wouldn't be asking this question or even going down the ROI path.

Get on a "train the trainer" type course. It'll teach you about training plans, outcomes, key competencies, assessing/confirming competency, etc. This will provide objective evidence of the success, or otherwise, of the training provided.

3. The other dave certainly put it neatly into levels - which really makes it simple and clear (the way I like it). Just watch out when you go for level 4, as measures have to be meaningful and economically sound to collect and process.

Cheers.

First of all, I would like to thank everybody who answers on my questions.
Thank you.

Our specifications says "that all in-house trainer must undergo the training the trainer training program before conducting the training." However, with this on the specification, it limits us to provide a technical training because we must undergo first training the trainers program. That's why I'm asking questions regarding measuring training effectiveness.

I myself did not undergo the training the trainers program. My problem started when our boss transferred the responsibility with regards to training in our department. I insert a grandfathering clause in our specs.
Since I don't undergo such training, measuring training effectiveness is not clear to me.

I look the Kirkpatrick model, its great to google search. There's a lot of information on the net. It helps. I think I have to document the Kirkpatrick Model. Hmmm....I think I should have to incorporate training effectiveness in our document.

Hi CarolX, in our case, superiors perfomed "performance evaluation" annually / semi-annually.

Do we need to immediately perform performance evaluation after the training?

Originally quoted by CarolX:
"Really gets to the issue...did the employee learn from the training? "
Yes, because our productivity increased and less downtime. It is documented through our daily monitoring status and not using a form for measuring training effectiveness.

I have another thing that bothers me, "Do i need to document it using our form in measuring training effectiveness?"

Best regards,
Raffy

Also see: http://Elsmar.com/Forums/showthread.php?s=&threadid=232

Hi Marc,
Thanks for the immediate reply. The thread is very informative, for sure it would answer all my queries. Thank you very much. and Happy New Year in advance.
Best regards,
Raffy

Here is a link to an old discussion that might interest the group.

http://Elsmar.com/Forums/showthread.php?s=&threadid=3394&perpage=10&pagenumber=2

Respectfully, it is no mystery to most of you that I detest the performance review. The whole process is littered with false assumptions.

Raffy's post above describes the "daily monitoring status" that to me is perhaps the best type of measure/record generation one could have to illustrate 'effectiveness'(I use the term here loosely as I will explain later). For example, testing in itself generally only captures how well one memorizes a fact, not knowledge achieved. And, knowing something and possessing the know-how to do it are also different (achieving Level 3 as provided by Dave). This is where Don Winton's explanation between education and training come to bear. Training is experience. Additionally, Raffy's example bears witness to the System, not the individual.

Peter Senge in his book The Fifth Discipline describes a learning organization. In it, he clearly describes that the the knowledge an organization possesses is NOT the sum of the individual knowledge each person possesses (he uses an example where the managers in an organization each possess an IQ of 120 or better but cumulatively only have a 63). If you believe this, as I do, then you will see that the proof is in the System and that individual measures are meaningless. Dave's level 4 is close, but I would not limit it to financial gains. In fact, I would have to say that financial gain should not be the measure behind it at all. Instead, the measure should be around things the employees themselves can measure where the feedback they need is generated by the process/system and not from arbitrary and subjective means (scoring on a test or the performance review for examples) and available to them at the point of need (not in a quarterly statement).

Now my change-up: What is the standard after anyway? Effective training or training efficiency? If we are to use Peter Druckers definitions, which is it that the Standard is after? Peter says that 'effectiveness' is doing the right things where as 'efficiency' is doing things right. Raffy's example is an example of efficiency. So, is training effectiveness more a measure of ensuring that the right things are being taught to our employees? For example, are we training the people doing the work with the right documentation? Afterall, aren't we after both: people doing the right things right (effective and efficient)? So I ask again, what is the standard after?

Anyone want to take a swing at my change-up?? :biglaugh:

Regards,

Kevin

p.s. I want to make a special mention here to a dear contributor here at the Cove, Don Winton. A few years back, Don did some extensive posting here on 'effectiveness' and 'training' that got my mind working (I recommend doing a search using these words and Don Winton's name). I began to read up on this topic looking for information to confirm what he had presented here at the Cove and on his own site. I found a good deal of support and as such, I have changed my focus on how I pursue training and what training is (to me anyway). Thanks, Don!!

What "change up" Kevin?

We don't use Drucker's definitions when we use ISO9K2K. We use ISO9000's definitions.Which are spelled out for both effectiveness and efficiency.

Training effectiveness is what the standard calls for,not efficiency. Effectiveness as defined by the standard is achieving planned results. Obviously what is called for is some thought into what you hope to achieve by the training and how you will measure that achievement. The Planning. Then you measure effectiveness= how well did you meet your planned goal. :bonk:

No need to confuse the issue with Drucker's definitions. They aren't applicable.

Originally posted by Lucinda
What "change up" Kevin?
Training effectiveness is what the standard calls for,not efficiency. Effectiveness as defined by the standard is achieving planned results. Obviously what is called for is some thought into what you hope to achieve by the training and how you will measure that achievement. The Planning.

When I first read this, I thought " You know, Lucinda is correct and is interpreting the standard". Then, after digesting the suggested "curve" that Kevin offered up, because Luicinda asked "what curve?", my thoughts changed a bit. IMLO, the standard was created to keep Customers satisfied. This means doing things right. While the standard pushed "effectiveness", the Customer wants "efficiency" as defined by Drucker..."doing things right". How you do it is not their concern, just do it. Of course, in order to "do things right", you have to "do the right things".:bonk: Huh?:ko: :smokin:

The introduction of Drucker’s definitions was not done to create confusion, so I hope I didn't create it. Rather, they were introduced to create thought and discussion. Despite the semantics involved, your comment that ‘ISO auditors’ will be looking for ‘ISO definitions’ is appropriate. However, it should be noted that Drucker’s definitions are consistent with Webster’s definitions and are, in my opinion, better and more accurate. On a side note, I don't much care what the ISO folks say to that point.

Along Drucker's definitions, it should be kept in mind that anyone can be effective without being efficient (I can use a spoon instead of a shovel to clear snow from my porch if I want to). Being effective is not about doing things with proficiency or efficient use of resources. Instead, it is about getting things done by most any means. By this definition, this suits most organizations. Efficiency, on the other hand, is about optimization, and for me, an essential requirement for system development and continuous improvement.

No matter how much thought one puts into what they hope to get out of training, there is no way possible to determine the true ROI of education or training. About all that can be said quantitatively about either are about how much time and at what cost they both come. This asks the question: why try to quantify it in the first place? We invest in education and training because in theory, it will have a handsome payoff.

Apparently by ISO’s definition, effectiveness is achieving planned results. If clearing the porch of snow is the goal, either the shovel or the spoon will do the trick!

Regards,

Kevin

Kevin,

It didn't confuse. It allowed me to step outside the proverbial box and think about why the standard was created in the first place. It's all about the standard writer's idea about satisfying the Customer. In that respect, I liked it!:p Those who live inside the standard fall prey to what it was intended for. Druckdude's interpretation of what effectiveness and efficiency is just that: How he sees it It's not the gospel. Just another viewpoint. Have a Happy New year. Incidentally, your town is now one of the prime Atlantic Salmon Breeding grounds. I intend to exploit that. I will "do the right thing" and hopefully "do everything right":vfunny: :smokin:

Some random thoughts ...

This discussion is turning out to be a discussion on effectiveness and efficiency and not on measuring training effectiveness.

"Doing the right thing / Doing things right" are IMHO catchy phrase

The definitions I would like to go with are :

Effectiveness - Achieving the desired results
Efficiency - Minimising resources (optimising as Kevin has mentioned) to be effective.

Now, if resource optimisation is part of "desired results" then is their any need to split hair with the above definitions?

As far as measurement of training is concerned, I think Kirkpatric Model does provide the necessary framwork. However, increase in competency level cannot be attributed to training only, hence correlating improvement in competency with training becomes a major "statistical" exercise. I am not convinced of the cost / benefit of such an exercise.

The organisational goal should be performance improvement, and training is one of the contributing factor. The impact of different factors also depends to a large extent on the profile of the employees. Training should not become an end in itself.

The key to a succesful training program lies in identifying training needs and finding competent trainers. Thus there should be some way of measuring how well the course contents match with the identified training needs and how "competent" the trainer is/(was?).

**********

Originally posted by Neelanshu Varma
Some random thoughts ...
This discussion is turning out to be a discussion on effectiveness and efficiency and not on measuring training effectiveness.


Neelanshu,

We tend to do that sometimes!:rolleyes: It usually happens when the standard poses more problems than it solves. The whole topic regarding training effectiveness is about what we "think" it means. The varying opinions so far, are just that. Opinions. There is no clear cut answer to send anyone on their way with an understanding of training effectiveness. So, we dabble in the interpretation and the intent of the standard's effectiveness.:vfunny: Thanks for snapping me out of my pre-occupation with definitions. I also wish I had a nickel each time a topic strayed from from the original post.:p It probably has something to do with a disdain for regimentation.:ko: :smokin:

Measuring Training Effectiveness
---------------------------------------------
Train an operator on how to run a part.

Are they running at the stated quality/production rates? Then they are effective.

--------------------------------------------

Train a person to inspect a part.

Is there a higher than expected fallout rate at the next operation? Then they are are not effective.

-------------------------------------------


These are very basic, but real, examples that can be expanded upon. People try to make all types of effectiveness too complex when they need to stand back and realize the the measureabls are alrerady in place and need to be collected and analyzed.

MHO
:)

Originally posted by Al Dyer
Measuring Training Effectiveness
---------------------------------------------
Train an operator on how to run a part.
Are they running at the stated quality/production rates? Then they are effective.
--------------------------------------------
Train a person to inspect a part.
Is there a higher than expected fallout rate at the next operation? Then they are are not effective.
-------------------------------------------


That's too easy. Try Designers, Engineers, All types of Managers, Trainers and you!:biglaugh:
Not everybody produces something that can readily measured. That's where the real issues of training effectiveness are. Not to mention competency!:ko: :smokin:

Like any measurement, training effectiveness is measured by looking at the output and comparing it to the desired result, i.e. competence.

This can be established through written examinations, witnessing work being carried out, reviewing work carried out, to name but a few examples that spring to mind.

Energy sums up the difference between discussion and dialogue pretty well. From my perspective, the dialogue here is equally, if not more so, valuable. Digression in an open format should be encouraged, but as any of us is capable of noticing, when both the discussion and dialogue continue, it is probably better to break out the dialogue aspect (for example, I think that I might have confused Lucinda by introducing Drucker’s definitions into the discussion, hence the changeup).

As such, I will return to the discussion of training effectiveness. If we think that we should break out the ‘definition discussion’, I will post a new thread to further explore this.

I would like to use Neelanshu’s closing comments here:

“The key to a successful training program lies in identifying training needs and finding competent trainers. Thus there should be some way of measuring how well the course contents match with the identified training needs and how "competent" the trainer is/(was?).”

I like them and I identify well with them. In days of old, I was prone to give a test at the end of a program to determine comprehension. Heck, it was what I knew since this was done to me at the end of training exercises and done to me all through school. In reality however, I only measured how well folks memorized things. Memorizing information is not knowledge and classroom exercises are more education than training. The education and training are linked through knowledge: knowledge gained and knowledge applied.

The path is: information > knowledge (education/theory) > know-how (applied theory/knowledge)

(My projection to follow): Let’s look at Neelanshu’s comment “The key to a successful training program lies in identifying training needs and finding competent trainers.” How are needs determined? Generally, they are determined through detected deficiencies. Sometimes, they are determined through projected deficiencies. Either way, needs are established and provide the frame work to determine what level of mastery is required to drive improvement (corrective or preventive). This gives us a starting point in asking key questions to the training candidate. Course content should be developed to address the needs and a trainer, with the level of mastery needed to provide such training, needs to be selected (this probably should precede content development). Course content should include education and training to be most effective in developing understanding and skills required. Exit feedback is required, the measurement of training effectiveness. This can be accomplished a number of ways.

From my perspective, I do not like to use grading/ranking systems. I use a sampling method shown to me by a close colleague that serves me well. I use a pre-sample of questions (mixed open-ended and multiple choice questions) to determine an initial starting point. The course content is built around noted deficiencies and any new information contained on the pre-sample (the pre-sample questions are developed by the individual with mastery of the skills and knowledge required to perform as expected). The course is conducted to provide knowledge base and the candidate returns to work to further develop skills. At some point after, the post-sample is done to determine how effective the education/training were based on the initial starting point. Without grades, one can determine if progress was made. Sometime follow-up education/training is required to help the associate achieve the required proficiency. In addition, the trainee is allowed to give feedback in order to improve the training process itself.

Returning to Raffy’s example of daily monitoring of status of a given process, this serves to better illustrate the effectiveness of planning for that process and the efficient execution of that plan. It only serves as a possible indicator of the success of the training. For example, if the process returns to a stable process after being found out of control where the remedial action was to bolster the training program, then one could say that the training was effective by this means (training was “doing the right thing”). However, as correctly pointed out in other posts above, there are many contributing variables to any given process. One cannot assume that training and ongoing training are the underlying reasons for improvement. You can only do so through isolation. Daily monitoring here serves as a potential source for Training and I like it because it is system driven. Through cause and effect, you may determine that the likely cause was training and you present yourself with an opportunity.

Regards,

Kevin

Quote

"For example, if the process returns to a stable process after being found out of control where the remedial action was to bolster the training program, then one could say that the training was effective by this means (training was “doing the right thing”). However, as correctly pointed out in other posts above, there are many contributing variables to any given process. One cannot assume that training and ongoing training are the underlying reasons for improvement."

If you run an intelligent corrective action program you will firstly analyse the root cause of a problem. You would not assign an arbitrary corrective action to an out of control process and then assume that your action had an effect. If you found the root cause to be lack of knowledge then your corrective action would be some form of training, for which you should see an effect on the process. Thus you could conclude with some certainty the effectiveness of the training.

Well put. Treat the cause, not the symptom.

Originally posted by M Greenaway

This can be established through written examinations.
Who and on what?
witnessing work being carried out,
by Who?
reviewing work carried out,
by Who?
to name but a few examples that spring to mind.


Sounds like performance reviews to me! And we beat that one to death!:bonk:

Keep them coming!:ko: :smokin:

OK energy - sounds like we should all give up and go home then. It is very easy to sit back and pick any proposal to pieces, there are pro's and con's with everything. What would you suggest ?

Originally posted by M Greenaway
OK energy - sounds like we should all give up and go home then. It is very easy to sit back and pick any proposal to pieces, there are pro's and con's with everything. What would you suggest ?

I'm here to learn. It's always put forth as this is how you do it, with no clear cut method of how. You get text book answers with no "experience" factor. That's all. Like you, I hope, I'm still searching for solutions, not pat answers. :bonk: Relax, it's just another day.:ko: :smokin:

skullsike,

Be sure to include M Greenway's comments when considering if improvement is attributable to training activities conducted.

Regards,

Kevin