ISO 9001:2015 7.1.5 Monitoring and measuring resources: Application to training providers

[Cats Clause]

Hi all

I asked a similar question of the forum a while ago, but am looking for some more input. I am currently working with a training provider who designs courses and then delivers them to the Learners. I am happy that they have good process controls in place for both the content design and delivery. It is the following elements of 7.1.5 of which I would like some feedback please:

a) (resources) are suitable for the specific type of monitoring and measurement activities being undertaken;
b) are maintained to ensure their continuing fitness for their purpose.

The monitoring and measuring is conducted via an independent process audit process (QA personnel drop into courses and monitor the quality of the course including the tutors performance) and also post course surveys.

The documented evidence of the selection and monitoring of appropriately experienced and qualified QA auditors would seem to be a sensible step. But what of the post course questionnaires? How to ensure they are gathering accurate performance feedback? (We have all come across poorly worded surveys which can give a skewed assessment!)

Thanks all.

 

[Jen Kirley]

Hello Cats Clause,

Surveys are always questionable tools for accurate information gathering. Can you contact the recipients and go through the survey together?

 

[Cats Clause]

Hello Cats Clause,

Surveys are always questionable tools for accurate information gathering. Can you contact the recipients and go through the survey together?

Hi Jen
Thanks for the reply.
These are embedded processes within quite a large organisation which are working well for them at the moment, so they will be reluctant to change. I am looking at the as-is systems to see what gaps need to be addressed to meet the requirements of 9001. Hence the specific question on how to demonstrate that both the Process Auditors and Surveys are suitable and maintained.

 

[Randy]

Wow, sounds like something I've been doing with a training provider on a 3rd party certification basis for about 11-12 years now. Here's the deal ( I might be more colorful than Jen). Surveys, unless the results can be validated in some fashion, can result in erroneous information. A good majority of survey's I review show Bull Fritter responses where the individual just checks a block and moves on (you can tell by looking), if they even take the time to do the survey (out of 60000+ surveys over a 3 year cycle a return of about 35% was experienced, basically 2 out of 3 went unanswered). Not knowing your survey design makes critique tough but here's what I know as someone who has a bit of experience in adult education & training (ask around) so without feedback beyond, yes-no, satisfied-not satisfied, 1-5 or whatever you really don't get much.

 

[Tagin]

The monitoring and measuring is conducted via an independent process audit process (QA personnel drop into courses and monitor the quality of the course including the tutors performance) and also post course surveys.

What specifically are the surveys intended to monitor/measure? If the QA auditors are monitoring "the quality of the course including the tutors performance", then what other specific attributes are the surveys intended to monitor/measure?

 

[John Broomfield]

You instructors are competent.

So they’ll be testing for understanding as they teach.

They’ll also be discussing other concepts, ideas and interpretations with their class.

And every lecture is followed by a quiz to further engage the students in verifying their learning.

Lectures and quizzes are interspersed with workshops that engage the students in role play from which they learn from each other.

All of this is monitored and corrected by the instructor with and without the help of fellow students before the final exam.

Quite difficult to replicate with e-learning.

 

[Jen Kirley]

Hi Jen
Thanks for the reply.
These are embedded processes within quite a large organisation which are working well for them at the moment, so they will be reluctant to change. I am looking at the as-is systems to see what gaps need to be addressed to meet the requirements of 9001. Hence the specific question on how to demonstrate that both the Process Auditors and Surveys are suitable and maintained.

It is hard to be sure what to advise without seeing your process in depth, but here goes:

If the organization is using ineffective survey methods why would they be reluctant to change? If something is not working (which I gather or else you would not have made your initial post) do they really want to keep marching down the same path that maybe leads to a cliff?

According to Kirkpatrick, surveys are one of the lower levels for evaluating training. Still, they have their place. Survey design might make a difference with response rates and accuracy.

About meeting the requirements of ISO 9001: you are putting the cart before the horse. The needs of your customers and stakeholders (interested parties) are the most important. Think of what could go wrong (risk based thinking) and your stakeholders' needs and wants, develop your programs as you have (in-person auditing of classes is excellent), capture nonconforming output (understand the causes of failure and address them where possible) and understand value through measurement (since the purpose of training is to improve ourselves in some way) and the result should be conformance with ISO 9001.

I haven't seen ISO 21001 yet - I wonder if it would be a better fit.

 

[BradM]

It's a good question you posed. I certainly wish more folks would pose such a question.

Suppose Jennifer and Randy here are letting me sit in on their internal audit training classes (Jennifer/ Randy... I would love that!!!). Afterwards..... I get a survey to complete for both. He and I got along really well. I like his no nonsense teaching style and his stories on some lessons he learned. So I give him high marks.... ,if I decide to complete the survey. I mean... I'm busy... if the survey is too long, forget it.

Jennifer.... has forgotten more than I will ever learn. She really had a great way to cover the standard forwards and backwards. I learned a lot about the standard!! So I will give her high marks.... if I decide to complete the survey for her. I mean... I'm busy... if the survey is too long, forget it.

The surveys may (note may...) be adequately valid to assess the perceived quality of the instructors and their instruction method. But at no time was I assessed whether I really learned anything to consider being a better internal auditor.

Most surveys (forgive the bluntness...) are garbage. And the ones that are decently constructed, most responses rates are very low. Most folks complete them only if they are really happy (like I would be with Randy and Jennifer) or if they are really disappointed and angry (like... most folks would be if I tried to teach that class ).

The first thing I would start with is asking the right questions: what exactly am I interested in measuring? What data would be useful to us to assess effectiveness? What data is reasonably available to me? If I decide to analyze data using statistical methods, do I have access to folks to do that and help explain the results?

So to your question as to accurate performance feedback, I would start with asking about what information you really are interested in getting and how you can go about getting it (maybe using other methods than surveys).

 

[malasuerte]

Hi all

I asked a similar question of the forum a while ago, but am looking for some more input. I am currently working with a training provider who designs courses and then delivers them to the Learners. I am happy that they have good process controls in place for both the content design and delivery. It is the following elements of 7.1.5 of which I would like some feedback please:

a) (resources) are suitable for the specific type of monitoring and measurement activities being undertaken;
b) are maintained to ensure their continuing fitness for their purpose.

The monitoring and measuring is conducted via an independent process audit process (QA personnel drop into courses and monitor the quality of the course including the tutors performance) and also post course surveys.

The documented evidence of the selection and monitoring of appropriately experienced and qualified QA auditors would seem to be a sensible step. But what of the post course questionnaires? How to ensure they are gathering accurate performance feedback? (We have all come across poorly worded surveys which can give a skewed assessment!)

Thanks all.

Apologies _ I think you are misunderstanding clause 7.1.5. Or I am misunderstanding what you are doing?!

Clause 7.1.5 is about the measurement of products or services to requirements.

Let me use an example: You make basketballs.

• To meet clause 7.1.5, you need to determine and provide the resources needed in order to guarantee you get correct/valid results from any 'monitoring and measuring' you perform to verify the conformity of the product.
• The requirements that will need to be measured are size (dimensions), thickness, ability to hold air
• My company in turn buys calipers, a tool to spin the ball to make sure it is round, an an air pressure tester
• in 7.1.5 a - I now need to determine if these tools are suitable (capable, tolerances, etc.) to be able to perform the needed measurements
• in 7.1.5 b - I need to maintain the tools - calibration, maintenance, etc

Yes, I know - "resources" can be people. So we can apply the same process.

Apologies again - am I missing something here? Context maybe?

 

[Cats Clause]

Hi all and thanks for some great feedback. Early days in the project - I have not seen a sample of the surveys, the data extracted from them or actions taken as a result as yet, so cant really comment. My brief is to get the organisation in shape for 9001 certification rather than process improvement at this point (although of course as opportunities arise, I will be pointing these out - hence I will be taking on board the interesting comments on the validity of surveys.) Another measure of the accuracy of the measuring (surveys / audits or other) I am thinking would be the correlation between the survey / auditor feedback and the learning outcomes from the course? IE: High survey scores on content and delivery but low pass rate would indicate the measuring systems are not accurately reflecting the effectiveness of the course? A poor (very lagging) indicator for sure, but a valid one all the same?