Evidence of Effectiveness of OJT (On the Job Training) - ISO 9001 Clause 6.2.2

[Shawn Johnson - 2009]

Here's the deal... maybe someone out there can give me some ideas!

We have been ISO registered since 1997. We successfully transitioned to ISO 9001:2000 in May 2003. We just had our recert audit, and I have a finding specifically about 6.2.2, and documenting the effectiveness of training... even though we've had the same system in place ad infinitum.

We have used the same process since day 1 of our ISO journey. Our job descriptions discuss specific requirements with regard to education, training, and general competance for the position.

Our performance review is both generic in some sections, but also is specific to a job - based on the requirements for the position.

When someone is hired, there is a checklist of specific activities that someone has to learn during their course of "training" period. As subjects/tasks are addressed, they are signed off by the supervisor and the person being trained.

I understand some of the concepts here... I know that some people do pre/post testing and such. But that kind of approach only works for specific kinds of training. Safety, for instance. But how do you use this if most of the training is on-the-job? Sure, I can add a new form that has the supervisor sign off that they feel someone is performing the job as they should (e.g. effective) but two things: one, I'd hate to put another "form" in there where all a supervisor has to do is sign off and two, isn't that what a performance review is for... talking about one's ability to perform a job?

In addition to standard performance, any specific training that is identified is added as a "goal" for the following year - which will be followed up on.

I guess I am looking for advice on how to do "effectiveness" of training, when most of the tasks are learned on-the-job. HELP!

 

[AndyN]

You've got a number of options available.......

including that, from analysis, if there's an overall performance trend improvement after a training initiative (like everyone getting SPC training) then you could infer that shows effectiveness. Another option is to derive the effectiveness from a round of internal audits, since IA is one of the methods of measurement and monitoring in ISO 9001:2000. Perhaps, if when analyzing defects, the root cause isn't training, competency or skills etc, that might show effective training - although that's a toughy if you subscribe to the Deming principles. I prefer pre & post evaluations of deployment of the individuals' training, because it's just that, individual and IMHO, performance reviews are only a very broad gauge of competencies and training.

Andy

 

[RoxaneB]

Job Observations or Task Analysis or whatever you wish to call it is the way we go about this...with the results supported by Nonconformances and Customer Complaints (or rather the lack thereof).

Yes, it's another form...but it's also a feedback tool for the operator. These Job Observations are safety and environment and process based. Right now, our methodology is being revised and with close to 800 documents in the system, we've decided that Job Observations will be conducted only on the identified key standards. If it has not yet been identified as being key to safety/environment/process, it does not require a Job Observation.

All crews will be observed performing the key task/function and feedback provided immediately and posted. There is also time for operator feedback (i.e., if they have a way to improve safety/environmental/process aspects).

There is an action log to record any "To Do's" generated from the Job Observation which is to be reviewed on a monthly basis at all levels.

From a due diligence viewpoint, the Job Observation has been our means to address several potential hotspots, while verifying training effectiveness and promoting consistency amongst the crews.

 

[db]

Don't get too wrapped up over this. Remember that ISO has three components in this area. First, you identify your competency needs, second you meet those needs, and third you evaluate how well you did against those two. Of course, you are also required to maintain "appropriate" records. If certain forms and sign-offs help you, then good, but they are not required.

With that in mind, how does the supervisor know if an operator "gets" the OJT? My guess is that the supervisor observes the behavior of the operator. If the operator's work is satisfactory, then apparently the training was effective. If not, then the lack of performance needs to be addressed. Assuming the performance is satisfactory, the next question is does the evaluation have to be formalized with a record? This is something the organization must decide on a case-by-case basis. If so, then develop a method to capture that information. If it does not have to be formalized, then the evidence of the evaluation would be the satisfactory performance. If someone asked how the supervisor knows the operator was trained, the response should be it is irrelevant. You know the person is competent due to the output of work. Someone who is competent will have consistent satisfactory performance.

 

[Justin]

Here's the deal... maybe someone out there can give me some ideas!

We have been ISO registered since 1997. We successfully transitioned to ISO 9001:2000 in May 2003. We just had our recert audit, and I have a finding specifically about 6.2.2, and documenting the effectiveness of training... even though we've had the same system in place ad infinitum.

We have used the same process since day 1 of our ISO journey. Our job descriptions discuss specific requirements with regard to education, training, and general competance for the position.

Our performance review is both generic in some sections, but also is specific to a job - based on the requirements for the position.

When someone is hired, there is a checklist of specific activities that someone has to learn during their course of "training" period. As subjects/tasks are addressed, they are signed off by the supervisor and the person being trained.

I understand some of the concepts here... I know that some people do pre/post testing and such. But that kind of approach only works for specific kinds of training. Safety, for instance. But how do you use this if most of the training is on-the-job? Sure, I can add a new form that has the supervisor sign off that they feel someone is performing the job as they should (e.g. effective) but two things: one, I'd hate to put another "form" in there where all a supervisor has to do is sign off and two, isn't that what a performance review is for... talking about one's ability to perform a job?

In addition to standard performance, any specific training that is identified is added as a "goal" for the following year - which will be followed up on.

I guess I am looking for advice on how to do "effectiveness" of training, when most of the tasks are learned on-the-job. HELP!

Looks to me like you have all of 6.2.2 covered. The effectiveness of training is measured during your performance review. You will also find out during your internal audits whether or not your training has been effective.

Sounds to me like your auditor was fishing.

 

[RoxaneB]

Looks to me like you have all of 6.2.2 covered. The effectiveness of training is measured during your performance review. You will also find out during your internal audits whether or not your training has been effective.

Sounds to me like your auditor was fishing.

Ooh...let me changes hats....I'm now the Devil's Advocate. How does a performance review which more than likely occurs once or twice a year adequately assess training which was done 6 months prior to the review?

Example : Off-site internal auditor training. I will admit, I've left some of these courses scratching my head in total confusion....only through my own need to have answers (i.e., the Cove and other contacts) have I learned much of what I know (or what I think I know). If I was assessed 6 months after this training (but had not done my own reseach), I will admit that the effectiveness of the training could have been called into question as the calibre of my audits would have been sorely lacking.

I don't know if the auditor was fishing for a finding...but perhaps fishing for an opportunity for improvement?

 

[db]

Ooh...let me changes hats....I'm now the Devil's Advocate. How does a performance review which more than likely occurs once or twice a year adequately assess training which was done 6 months prior to the review?

That has never made since to me. Performance reviews do not, in my opinion, address 6.2.2 c). The evaluation in c) is evaluating what was done in b). Hence, my question about how do you know the person trained (or hired) is competent? By watching their performance. Normal supervision and normal production records will indicate whether the person is competent or not.

 

[Jen Kirley]

I can only guess what the auditor was hoping to see, but you may very well succeed if you stand your ground, pointing to:

1. The operator gets OJT, which is kinetic (hands on) learning with perhaps verbal and visual lesson delivery. Since these approach the top three learning style needs, we shouldn't knock OJT.

2. The "trainer" signs or initials the training's conclusion and vouches for demonstrated task competence. These records are available, show dates and identify the personnel.

3. There is an absence of flaws due to training errors or ignorance, showing the operator is in fact performing within parameters.

Refresher training may be a good idea if you think there may be some freelance "improvements" not allowable or for other good reasons like regulation and safety. Otherwise it sounds to me like you're covered.

 

[Randy]

Nice subject competence and OTJ is.. (do I sound like Yoda?)

As the victim, or recipient as the case may be, of numerous types of OJT in a diverse number of fields I'd like to add my

Many times in my life competence (or luck) and effectiveness of OJT was proven by your breathing and walking about at the end of the "task".

Essentially our goal should be for the receiver of OJT, or any other training for that matter, to be able to translate "theory" into reality. Competence is nothing more than be able to perform to expectation within prescribed parameters and achieve the desired outcome. As defined in ISO 19011 competence is "demonstrated personal attributes and demonstrated ability to apply knowledge and skills"

In reality it's not the process that's important, it's the outcome...kinda like making sausage. I believe that too much emphasis is focused on process than outcome. The process has to be a variable that changes as trends require change.

The difficult problem presented to us is "how to measure competence?". What is a good KPI or do we need to use a mixture to validate?

As a firearms instructor I had to take many a young man (no girls allowed where we were) and be able to get them to put rounds on target. The training (which was essentially classroom and OJT mix) had to reflect the individual ability of each shooter....everyone got the same "class" (theory) but the hands on or OJT element, though following an established format became customized for each shooter. We knew competence as a basic shooter (Marksman) was accomplished when each shooter could score a 175 out of a possible 250 points (70%) unassisted. There were those that excelled beyond the minimum D ) but our competence goal (KPI) was 175. Below 175 was unacceptable and many times required the individual to go back to the classroom. Sometimes competence was never achieved consequently causing career problems for the individual in question...minimum competence was an absolute.

Where I'm going with this is that you've got to figure it out yourself based upon your individual organization values and needs, measure your level of success and provide justification and validation for the process and results.

 

[Justin]

Ooh...let me changes hats....I'm now the Devil's Advocate. How does a performance review which more than likely occurs once or twice a year adequately assess training which was done 6 months prior to the review?

Your answer is in two parts.

1. Training Records

2. You still have your job.