ISO 9001:2015 7.1.5 Monitoring and measuring resources: Application to training providers

Cats Clause

Starting to get Involved
#1
Hi all

I asked a similar question of the forum a while ago, but am looking for some more input. I am currently working with a training provider who designs courses and then delivers them to the Learners. I am happy that they have good process controls in place for both the content design and delivery. It is the following elements of 7.1.5 of which I would like some feedback please:

a) (resources) are suitable for the specific type of monitoring and measurement activities being undertaken;
b) are maintained to ensure their continuing fitness for their purpose.

The monitoring and measuring is conducted via an independent process audit process (QA personnel drop into courses and monitor the quality of the course including the tutors performance) and also post course surveys.

The documented evidence of the selection and monitoring of appropriately experienced and qualified QA auditors would seem to be a sensible step. But what of the post course questionnaires? How to ensure they are gathering accurate performance feedback? (We have all come across poorly worded surveys which can give a skewed assessment!)

Thanks all.
 
Elsmar Forum Sponsor

Jen Kirley

Quality and Auditing Expert
Leader
Admin
#2
Hello Cats Clause,

Surveys are always questionable tools for accurate information gathering. Can you contact the recipients and go through the survey together?
 

Cats Clause

Starting to get Involved
#3
Hello Cats Clause,

Surveys are always questionable tools for accurate information gathering. Can you contact the recipients and go through the survey together?
Hi Jen
Thanks for the reply.
These are embedded processes within quite a large organisation which are working well for them at the moment, so they will be reluctant to change. I am looking at the as-is systems to see what gaps need to be addressed to meet the requirements of 9001. Hence the specific question on how to demonstrate that both the Process Auditors and Surveys are suitable and maintained.
 

Randy

Super Moderator
#4
Wow, sounds like something I've been doing with a training provider on a 3rd party certification basis for about 11-12 years now. Here's the deal ( I might be more colorful than Jen). Surveys, unless the results can be validated in some fashion, can result in erroneous information. A good majority of survey's I review show Bull Fritter responses where the individual just checks a block and moves on (you can tell by looking), if they even take the time to do the survey (out of 60000+ surveys over a 3 year cycle a return of about 35% was experienced, basically 2 out of 3 went unanswered). Not knowing your survey design makes critique tough but here's what I know as someone who has a bit of experience in adult education & training (ask around) so without feedback beyond, yes-no, satisfied-not satisfied, 1-5 or whatever you really don't get much.
 

Tagin

Trusted Information Resource
#5
The monitoring and measuring is conducted via an independent process audit process (QA personnel drop into courses and monitor the quality of the course including the tutors performance) and also post course surveys.
What specifically are the surveys intended to monitor/measure? If the QA auditors are monitoring "the quality of the course including the tutors performance", then what other specific attributes are the surveys intended to monitor/measure?
 

John Broomfield

Leader
Super Moderator
#6
You instructors are competent.

So they’ll be testing for understanding as they teach.

They’ll also be discussing other concepts, ideas and interpretations with their class.

And every lecture is followed by a quiz to further engage the students in verifying their learning.

Lectures and quizzes are interspersed with workshops that engage the students in role play from which they learn from each other.

All of this is monitored and corrected by the instructor with and without the help of fellow students before the final exam.

Quite difficult to replicate with e-learning.
 

Jen Kirley

Quality and Auditing Expert
Leader
Admin
#7
Hi Jen
Thanks for the reply.
These are embedded processes within quite a large organisation which are working well for them at the moment, so they will be reluctant to change. I am looking at the as-is systems to see what gaps need to be addressed to meet the requirements of 9001. Hence the specific question on how to demonstrate that both the Process Auditors and Surveys are suitable and maintained.
It is hard to be sure what to advise without seeing your process in depth, but here goes:

If the organization is using ineffective survey methods why would they be reluctant to change? If something is not working (which I gather or else you would not have made your initial post) do they really want to keep marching down the same path that maybe leads to a cliff?

According to Kirkpatrick, surveys are one of the lower levels for evaluating training. Still, they have their place. Survey design might make a difference with response rates and accuracy.

About meeting the requirements of ISO 9001: you are putting the cart before the horse. The needs of your customers and stakeholders (interested parties) are the most important. Think of what could go wrong (risk based thinking) and your stakeholders' needs and wants, develop your programs as you have (in-person auditing of classes is excellent), capture nonconforming output (understand the causes of failure and address them where possible) and understand value through measurement (since the purpose of training is to improve ourselves in some way) and the result should be conformance with ISO 9001.

I haven't seen ISO 21001 yet - I wonder if it would be a better fit.
 
Last edited:

BradM

Leader
Admin
#8
It's a good question you posed. I certainly wish more folks would pose such a question.

Suppose Jennifer and Randy here are letting me sit in on their internal audit training classes (Jennifer/ Randy... I would love that!!!). Afterwards..... I get a survey to complete for both. He and I got along really well. I like his no nonsense teaching style and his stories on some lessons he learned. So I give him high marks.... ,if I decide to complete the survey. I mean... I'm busy... if the survey is too long, forget it.

Jennifer.... has forgotten more than I will ever learn. She really had a great way to cover the standard forwards and backwards. I learned a lot about the standard!! So I will give her high marks.... if I decide to complete the survey for her. I mean... I'm busy... if the survey is too long, forget it.

The surveys may (note may...) be adequately valid to assess the perceived quality of the instructors and their instruction method. But at no time was I assessed whether I really learned anything to consider being a better internal auditor.

Most surveys (forgive the bluntness...) are garbage. And the ones that are decently constructed, most responses rates are very low. Most folks complete them only if they are really happy (like I would be with Randy and Jennifer) or if they are really disappointed and angry (like... most folks would be if I tried to teach that class :) ).

The first thing I would start with is asking the right questions: what exactly am I interested in measuring? What data would be useful to us to assess effectiveness? What data is reasonably available to me? If I decide to analyze data using statistical methods, do I have access to folks to do that and help explain the results?

So to your question as to accurate performance feedback, I would start with asking about what information you really are interested in getting and how you can go about getting it (maybe using other methods than surveys).
 

malasuerte

Quite Involved in Discussions
#9
Hi all

I asked a similar question of the forum a while ago, but am looking for some more input. I am currently working with a training provider who designs courses and then delivers them to the Learners. I am happy that they have good process controls in place for both the content design and delivery. It is the following elements of 7.1.5 of which I would like some feedback please:

a) (resources) are suitable for the specific type of monitoring and measurement activities being undertaken;
b) are maintained to ensure their continuing fitness for their purpose.

The monitoring and measuring is conducted via an independent process audit process (QA personnel drop into courses and monitor the quality of the course including the tutors performance) and also post course surveys.

The documented evidence of the selection and monitoring of appropriately experienced and qualified QA auditors would seem to be a sensible step. But what of the post course questionnaires? How to ensure they are gathering accurate performance feedback? (We have all come across poorly worded surveys which can give a skewed assessment!)

Thanks all.

Apologies _ I think you are misunderstanding clause 7.1.5. Or I am misunderstanding what you are doing?!

Clause 7.1.5 is about the measurement of products or services to requirements.

Let me use an example: You make basketballs.

  • To meet clause 7.1.5, you need to determine and provide the resources needed in order to guarantee you get correct/valid results from any 'monitoring and measuring' you perform to verify the conformity of the product.
  • The requirements that will need to be measured are size (dimensions), thickness, ability to hold air
  • My company in turn buys calipers, a tool to spin the ball to make sure it is round, an an air pressure tester
  • in 7.1.5 a - I now need to determine if these tools are suitable (capable, tolerances, etc.) to be able to perform the needed measurements
  • in 7.1.5 b - I need to maintain the tools - calibration, maintenance, etc
Yes, I know - "resources" can be people. So we can apply the same process.

Apologies again - am I missing something here? Context maybe?
 

Cats Clause

Starting to get Involved
#10
Hi all and thanks for some great feedback. Early days in the project - I have not seen a sample of the surveys, the data extracted from them or actions taken as a result as yet, so cant really comment. My brief is to get the organisation in shape for 9001 certification rather than process improvement at this point (although of course as opportunities arise, I will be pointing these out - hence I will be taking on board the interesting comments on the validity of surveys.) Another measure of the accuracy of the measuring (surveys / audits or other) I am thinking would be the correlation between the survey / auditor feedback and the learning outcomes from the course? IE: High survey scores on content and delivery but low pass rate would indicate the measuring systems are not accurately reflecting the effectiveness of the course? A poor (very lagging) indicator for sure, but a valid one all the same?
 
Thread starter Similar threads Forum Replies Date
M Preservation of Raw Materials in ISO 9001:2015 Standard ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 12
L Assignment on the link between Quality Gurus & ISO 9001:2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
D ISO 9001:2015 Internal Audit Check Sheet ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
A ISO 9001:2015 8.5.6 and 7.5.3 Document Control Questions ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
J ISO 9001:2015 Clause 8.2.3 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 10
B ISO 9001:2015 Registered Companies ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
D ISO 9001:2015 Calibration Procedures Manufacturing and Related Processes 16
S ISO 9001:2015 Project Plan ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 4
P How to keep a ISO 9001:2015 certification when a company is sold ? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 7
D ISO 9001:2015 Recertification Audit Timing ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 11
J ISO 9001:2015 Is a Project Evaluation form necessary? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
Sidney Vianna ISO 9001 News ISO/TC 176/SC 2 Listing of Approved Interpretations against ISO 9001:2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 6
Q Management Review ISO 9001-2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 36
J Process capability analysis in ISO 9001:2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
B Use an iso 9001/2015 form for api q1 systeme Oil and Gas Industry Standards and Regulations 5
Q Which Kapt standard assignment ISO 9001:2015? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 1
Q Quality Awareness 7.3 ISO 9001-2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 8
M Quality Management-ISO 9001:2015 General Auditing Discussions 1
G ISO 9001-2015 How to co-relate Clause 4.2 & Clause 6.1 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 22
B ISO 9001:2015 vs ISO 13485:2016 Gap analysis ISO 13485:2016 - Medical Device Quality Management Systems 9
G Compliance with ISO 9001-2015 for ISO 17025 Accredited Labs? ISO 17025 related Discussions 8
L Change Log in (controlled) Forms? (ISO 9001:2015) Document Control Systems, Procedures, Forms and Templates 6
S Transition training for Internal Auditor from ISO 9001: 2008 to the ISO 9001:2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
J ISO 9001:2015 Clause 8.5.1 Control of Production and service provision - Help with Work Instruction Access ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 13
M Does the scope of ISO 9001:2015 applies to tenders, pricing and sales department of a medical devices distributor? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
C ISO 9001:2015 8.3.2. h) Design and Development Planning - What is required? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 4
B Employee Handbook in ISO 9001:2015 Section 7 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 19
Q ISO 9001-2015 Internal audit finding Internal Auditing 14
R AS9100D internal audit checklist or ISO 9001 2015 to AS9100 D AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 2
J Opportunity in ISO 9001:2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 27
D Reports under change management | ISO 13485:2016 & ISO 9001:2015 ISO 13485:2016 - Medical Device Quality Management Systems 3
K Integrating ISO 9001:2015 with ISO 17025:2017 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
S Sequence of ISO 9001:2015 Implementation Steps ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 6
Brizilla Employee Data Privacy Policy - ISO 9001:2015 requirement(s)? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 6
S ISO 9001:2015 Internal Auditing Internal Auditing 8
Q Process: Knowledge Section 7.1.6 of ISO 9001:2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 10
A What must be recorded? (ISO 9001:2015, subclause 10.2) ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
S ISO 9001:2015 vs 21 CFR Part 211 matrix Pharmaceuticals (21 CFR Part 210, 21 CFR Part 211 and related Regulations) 0
M Does the ISO 9001:2015 standard require a disaster recovery plan or emergency response plan ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 16
M ISO 9001:2015 and AS6081:2012 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
C Implementation ISO 9001: 2015 ? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
eule del ayre Documented Information - Periodic Review of Documents? IATF 16949:2016 / ISO 9001:2015 IATF 16949 - Automotive Quality Systems Standard 34
J Audit Checklist for Integrated Management System for ISO 9001:2015, ISO 14001 & OHSAS18001 (IMS) ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 7
lanley liao How to understand the clause 6 Planning of ISO 9001:2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 11
A Does ISO 9001:2015 cover all the requirements of ISO 10012:2003? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 6
T ISO 9001:2015 - Small Shop ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 7
M ISO 9001:2015 case study sample ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 8
M Sample of Nonconformity report for ISO 9001:2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
M Case study help as per ISO 9001: 2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 18
M Case study solution help required as per ISO 9001 : 2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 1

Similar threads

Top Bottom