ISO 9001 and Public Schools - Under continuous fire for low student achievement

[JW9000]

Re: ISO 9001 and Public Schools

Unless school systems are depoliticized and the people in power start looking at what the kids really need, there is no hope.

Great discussion by the way.

Stijloor.

Ultimately, the focus on state test data at the expense of input-process data is killing our educational system more than anything else, and it is getting worse. We struggle to identify effective strategies because we have not established measurements for gauging the effectiveness of teaching and curriculum in real time. (Well, my company has, but the educational community has not.)

With no way of knowing what works and what doesn't, where can you go?

 

[JaneB]

Re: ISO 9001 and Public Schools

The trouble is that schools are relying almost solely on output processes, which is problematic. Measuring output tells me little on how to improve input. Our math scores are low, so now what? Is it the instruction? The curriculum? Test scores cannot tell you.

As Jim already said - there are too many variables, not all of which can be controlled. What you're insisting on appears to be simply taking the 'manufacturing approach' (which I said won't work). For example, are the students all 'uniform inputs to be processed'? I don't think so. Until/unless you are willing to seriously confront and deal with the multiplicity of variables, arguing the point is relatively pointless.

I'm a little confused. Is ISO 9001 applicable to learning institutions? In the first sentence, you seem to indicate it is, but not in the second sentence.

Au contraire. I said it was (first sentence) - clause 1.2 of the Standard clearly indicates this. The second sentence referred to adopting a particular approach that won't work - the mechanical/manufacturing one. They're two different things: applicability of Standard vs method of implementing.

For my info - do you have any experience in service organisations? And in implementing 9001 in same?

More importantly, how are you measuring success when you say that it has been successfully applied?

The best way possible, using the organisation's chosen KPIs for success. Their organisation, their system, their 'quality objectives', and their measures for success.

It's a tricky point in a public system (more of those variables again) - who defines success. Parents? Kids? Board? Government responsible? Teachers? Onlookers? Public? etc

 

[JW9000]

Re: ISO 9001 and Public Schools

As Jim already said - there are too many variables, not all of which can be controlled. What you're insisting on appears to be simply taking the 'manufacturing approach' (which I said won't work). For example, are the students all 'uniform inputs to be processed'? I don't think so. Until/unless you are willing to seriously confront and deal with the multiplicity of variables, arguing the point is relatively pointless.

If I own a body shop, every car that comes in is unique. Some are in great shape, some are not. Some cars are inherently better built than others. I see a Yugo on one day, a Lamborghini the next. Regardless, there are practices of good bodywork that uniformly apply, and others that do not.

So, if ISO 9001 can only be applied in cases where the system is identical in every situation, then the body shop cannot be ISO 9001 certified in principle.

In such a case, can any standard system be employed?

Au contraire. I said it was (first sentence) - clause 1.2 of the Standard clearly indicates this. The second sentence referred to adopting a particular approach that won't work - the mechanical/manufacturing one. They're two different things: applicability of Standard vs method of implementing.

For my info - do you have any experience in service organisations? And in implementing 9001 in same?

I have plenty of experience in service organizations, but ISO 9001 is new to me, which is kinda' why I'm here. I'm not arguing from a standpoint of someone who knows better and is trying to convince everyone else that they are wrong. I am seriously trying to understand how a district can call itself ISO 9001 certified when it cannot answer even basic questions about its product.

The best way possible, using the organisation's chosen KPIs for success. Their organisation, their system, their 'quality objectives', and their measures for success.

It's a tricky point in a public system (more of those variables again) - who defines success. Parents? Kids? Board? Government responsible? Teachers? Onlookers? Public? etc

We can answer some of those questions now.

For example, state content standards are (for the most part) uniformly accepted as the pre-eminent standard to follow when assigning curriculum. Successful curricula align to state content standards.

We also have well-established taxonomies for rigor, and it is uniformly accepted that a quality curricula samples multiple levels of rigor, which we can measure.

We know that effective teaching uses time in the classroom wisely. We can measure that.

I can go on and on. But I think all of this misses the important point: The data is not what is important; it's the process. I don't care so much if a school has 90% of its classroom time devoted to academic engagement. I care much more that (1) this classroom time was measured, (2) faculty consulted with each other and possibly outside consultants to gauge for themselves whether the results were adequate, (3) that targets were set for future measurements, and (4) the faculty implemented that which they agreed to do to reach their targets.

In other words, is the faculty following a Deming cycle toward improving student achievement? 85%, 90%, 95%... who cares? To me, it's all about process, and I want to know if ISO 9001 concerns itself more with process or results?

Here is what I don't want: "We can't measure it to aircraft-grade precision, so let's not measure it at all." That is what schools are doing, and I see it really hurting us educationally. So what can we do about it?

 

[JaneB]

Re: ISO 9001 and Public Schools

Thanks JW for the background - that gives me a little better idea, because I'm somewhat confused about what the actual question (or questions) under debate are.

You say:

I am seriously trying to understand how a district can call itself ISO 9001 certified when it cannot answer even basic questions about its product.

If this is a single instance of a single district/school etc, I can' thelp - know nothing about it and my knowledge of the US education system is very basic.

Do certain individual organisations who are certified not have very good systems? Unfortunately, yes. Why is this so - many answers.

So, if ISO 9001 can only be applied in cases where the system is identical in every situation, then the body shop cannot be ISO 9001 certified in principle.

ISO 9001 doesn't say this. By the way, have you read the ISO 9001 Standard itself? Forgive me if so, because some of your questions appear to suggest you may not have, or may not know it very well?

I don't care so much if a school has 90% of its classroom time devoted to academic engagement. I care much more that (1) this classroom time was measured, (2) faculty consulted with each other and possibly outside consultants to gauge for themselves whether the results were adequate, (3) that targets were set for future measurements, and (4) the faculty implemented that which they agreed to do to reach their targets.

In other words, is the faculty following a Deming cycle toward improving student achievement? 85%, 90%, 95%... who cares?

Yes, I agree and I won't argue there.

To me, it's all about process, and I want to know if ISO 9001 concerns itself more with process or results?

Well, see here's where we're into vague terminology and questions.

"ISO" is the organisation that concerns itself with producing the Standards, so its use in this context is inaccurate, and renders the question vague and open to (mis)interpretation.

 

[JW9000]

Re: ISO 9001 and Public Schools

Well, see here's where we're into vague terminology and questions.

"ISO" is the organisation that concerns itself with producing the Standards, so its use in this context is inaccurate, and renders the question vague and open to (mis)interpretation.

Let me ask a question that may go a long way toward clearing up the ambiguity.

An ISO auditor visits a school and asks the school principal if the staff has examined the school's product. Yes, the principal pulls out charts that show the school staff has examined its curriculum (both adopted and enacted) quite thoroughly, found deficiencies, performed professional development, and set new targets for the school year.

Is the auditor concerned more with the process in place (monitoring, training, target setting) or the actual results of the curriculum monitoring?

The IWA says this about processes:

Process approach: educational organizations should adopt a process approach when developing and implementing a quality management system. The organization should identify the degree to which each operational process creates learner value. For this reason it should include the processes related to the aim of the organization. Understanding interactions among processes is important for the educational organization to improve processes while balancing the system at large.

In Sec. 4 it states:

The educational organization should define and manage the processes for the quality management system. Processes related to the aim of the organization should be included during and following the provision of the
educational service:

a) education design;
b) curriculum development;
c) education delivery;
d) assessment of learning.

So to me it appears that the ISO 9001 standards are process-driven, at least when applied to school systems. So, if a school does not have such processes defined in place, should it be ISO 9001 certified?

 

[bobdoering]

Do schools institute behaviors to measure the quality of their product? Well, let's consider a few questions:

1. Does the school directly monitor the delivered curriculum for its alignment to its own standards?
2. Does the school directly monitor the rigor of their delivered curriculum?
3. Does the school directly monitor the manner in which time is availed in classrooms?
4. Does the school directly monitor the use of the teaching strategies learned in recent professional development workshops?

Can a school system pass ISO certification requirements without directly monitoring such fundamental aspects of their product?

As has been stated ad nauseam, ISO is a system that is implemented to provide an atmosphere for success, not ensure it. Can you pass ISO without directly monitoring your specific list of aspects of their product? Sure, the system can be designed to measure any measures the process owner's think are important - not just the ones you think are important.

This goes back to the issue I brought up before, and that is the issue of measures and measurements. In industry, we invoke MSA to ensure that our measurement systems are statistically correct. You claim that your measurement systems can not be evaluated in that manner, and it is not true. There are doctoral students forced to verify that all of the time. But even so, you do not identify how "delivered curriculum for its alignment to its own standards" is to be measured accurately. Is word for word recitation of a standard curriculum 100% compliance, and anything less a defect? How about "teaching strategies learned in recent professional development workshops"? What if they do not fit a specific group of students? Is it necessary to employ these strategies lock-step, or it is a defect?

Measures of teaching effectiveness have been an ongoing debate because the process is nebulous. State test score standards are an attempt at this. And they are well and good IF they are really testing on the life skills the student really needs as a graduate. From the students I am getting as a college instructor, writing is not one of them, apparently. Is it the curriculum? Is it the instructor? I do not get them. I get the student.

 

[Jen Kirley]

What an interesting discussion.

Having been an incognito QA Professional working as a teacher's aide for a few years in my district's middle and high schools, I can submit my viewpoint that the disciplined manufacturing model is singularly inappropriate for modern education...unless you want to adopt the custom manufacturing model, as in custom made jeans when making an order by giving body measurements.

And I think focusing on outcomes is critical. After all, if the point of schooling is to make young people ready to functionally contribute to society, why not recognize that? After all, the essential definition of quality is fitness for use. What good is schooling if the people cannot obtain and sustain gainful employment, or at least succeed in a postsecondary school to prepare for educated/skilled employment?

I think that family involvement is very important, and it's no good for a school system to say that's outside their sphere or control. That comes dangerously close to a copout, in my view.

Successful schools set their goals, take the various stakeholders' interests and make them into the actual learning plan, and leverage their position as the hub - the facilitator of the learning process, only part of which happens at school but more responsibility for which is being thrust on schools. When looking at how to make schools successful, I would open up the study to include the abstract methods along with the disciplined structures and process approaches. When I say abstract I am thinking of examples like these.

 

[JW9000]

As has been stated ad nauseam, ISO is a system that is implemented to provide an atmosphere for success, not ensure it. Can you pass ISO without directly monitoring your specific list of aspects of their product? Sure, the system can be designed to measure any measures the process owner's think are important - not just the ones you think are important.

Okay, fair enough.

This goes back to the issue I brought up before, and that is the issue of measures and measurements. In industry, we invoke MSA to ensure that our measurement systems are statistically correct. You claim that your measurement systems can not be evaluated in that manner, and it is not true. There are doctoral students forced to verify that all of the time.

For the measures I am discussing, statistical significance is tough to generate even on a massive scale. A single district?

We collected 200,000 sheets of student work from 200 schools and barely made statistical significance on the measures we are performing. A doctoral student doesn't stand a chance. (The work stacks 80 feet high.)

Statistical significance is tough to pull off in education research, which is why they resort to meta-analysis to generate statistically significant results. (And don't even get me started on meta-analysis.)

But even so, you do not identify how "delivered curriculum for its alignment to its own standards" is to be measured accurately. Is word for word recitation of a standard curriculum 100% compliance, and anything less a defect? How about "teaching strategies learned in recent professional development workshops"? What if they do not fit a specific group of students? Is it necessary to employ these strategies lock-step, or it is a defect?

Again, you are focusing on the data and not the process. The district measured X and got 5. I don't care if they got 5 or 6. What's important is that the district evaluates the 5 they obtained in terms of their own needs and institutes a procedure in response. That is the Deming cycle.

Measures of teaching effectiveness have been an ongoing debate because the process is nebulous. State test score standards are an attempt at this. And they are well and good IF they are really testing on the life skills the student really needs as a graduate.

I fully disagree. Even if the state test was PERFECT in analyzing students' true proficiency, it does not satisfy the needs of "check" demanded of the Deming cycle because it is not formative. State tests do not check the process used to generate the results, they only test the outcomes.

So my math scores are low. Even if the state tests is perfect, that still doesn't point to a solution. Is it the instruction? Is it the curriculum? Is it the alignment of the curriculum? Is it the rigor of in-class questioning? State tests won't tell you.

In the Deming cycle, the check phase does not mean, "Sell the cars and see if you generate any customer complaints," which is directly analogous to the method of using state test scores to measure teaching effectiveness.

From the students I am getting as a college instructor, writing is not one of them, apparently. Is it the curriculum? Is it the instructor? I do not get them. I get the student.

Are you teaching them to write? I taught college physics and I spent a great deal of time teaching my students to write a proper introduction, conclusion, body. I even taught them how to a write a description of the force table we used in lab. One of the assignments covered figure and table captions.

An observer would look at me teaching this and see if I was presenting the concepts of writing, providing examples, using communication strategies, presenting a procedural algorithm, discussing the importance of learning how to write, and so on.

These students don't know how to write because no one has ever taught them. Why? Because it was always someone else's responsibilty.

Besides, what is it you are teaching and are subskill proficiencies getting in the way of assessing their proficiency on the content? That is one of the things I look for when observing classrooms. Can I measure that to national specifications? No. But it still needs measuring.

 

[JW9000]

What an interesting discussion.

Having been an incognito QA Professional working as a teacher's aide for a few years in my district's middle and high schools, I can submit my viewpoint that the disciplined manufacturing model is singularly inappropriate for modern education...unless you want to adopt the custom manufacturing model, as in custom made jeans when making an order by giving body measurements.

I don't disagree.

And I think focusing on outcomes is critical. After all, if the point of schooling is to make young people ready to functionally contribute to society, why not recognize that? After all, the essential definition of quality is fitness for use. What good is schooling if the people cannot obtain and sustain gainful employment, or at least succeed in a postsecondary school to prepare for educated/skilled employment?

Testing ensures none of that.

We have tons of tests. We have tons of test data. We know to high precision how students are performing in math and English language arts (not so much in science or history).

So what? What does that data do for us? Testing students hasn't raised student proficiency.

I'm not against state testing in principle. I think at some point you need a customer-complaint box. But focusing on outcomes is hurting our schools because it trains attention away from the source of the problems.

I think that family involvement is very important, and it's no good for a school system to say that's outside their sphere or control. That comes dangerously close to a copout, in my view.

No, focusing on family involvement is the cop-out. By focusing attention on that which takes place outside the confines of the school, it provides an excuse for failing. I have a lot of experience working with schools and I can tell you that the staff in poorly performing schools LOVES to talk about the parents and how important the family is to learning.

And they're right. But if we focus on that which we cannot directly control, it provides an excuse for failure.

By the way, your point illustrates precisely why we cannot focus on outcomes. When we do, the school staff looks at all the factors that go into the outcomes, decides that they cannot control all of them, and quits. "These students will never pass their state tests, so why do we even bother with all of this professional development?" I have heard it many times.

But if we focus on input processes, things change. Regardless of whether the families are rotten, I want to see teachers using good questioning strategies and allocating classroom time wisely. I want to see a high-quality curriculum being delivered.

If we don't measure it, it won't happen.

 

[JW9000]

When I say abstract I am thinking of examples like these.

I see the Baldridge-type processes splashed on district web sites all over the country.

For example, they will list "Data collection" as one of their processes. What data?

Here is one item listed by the Chugach district:
"Determine assessments aligned to standards."

In other words, they are measuring the alignment to standards of their district assessments. That is precisely one of the measures we perform.

The schools there are mostly Baldridge Award winners, so these school districts probably take the process of school improvement more seriously than others. But what exactly are they measuring? They're not saying.

On the other hand, if by data analysis these districts mean "analysis of state test scores," then they are doing nothing more than the vast majority of schools in the country. Just about all schools coalesce their staffs to analyze state test data. It doesn't help much because they are relying on the wrong data.