posted 07 May 2000 01:32 PM               

From: Pat Dey
Subject: RE: Tick-IT & ISO 9001 /Chen/Kirk/Peter/Dey
>
> Pat,
>
> Since you've made some general statements about the CMM and ISO, I'd
> thought I'd play a little devil's advocate with you. My point throughout
> this response is to offer up a theme. The theme is that models really
> don't amount to a hill of beans as much as a desire to change. I make
> this statement in support of the fellow who said that all models are
> wrong....some are useful. I would say that all models have warts...and
> that the key to software process improvement (or any improvement) is
> wanting to improve....to deliver products better and faster...

I agree.

> > The strength of the CMM is its model for continuous improvement. The SEI's
> > method includes, not only the process framework (the CMM itself) but the
> > methods for managing improvement by involving everyone, eg, through a
> > Software Engineering Process Group.
>
> But the reality is that the CMM is a staged model that doesn't really
> focus on defect prevention until level 5.

Hmm ... well as one devil's advocate to another:

1. Reviews are at level 3 in the CMM and, I hope, can be used for defect detection & removal (yes, not proactive defect prevention, but better than nothing).

2. I'm inclined to pull reviews down to level 2 because of the civilising behaviour they can be used to introduce to organisations. (They're at level 3, I believe, because they need the time management disciplines that level 2 introduces.)

3. More advanced processes like defect prevention are introduced at higher levels in the CMM after making the system stable, especially with level 2 (ie, Deming's idea of bringing the system into statistical control, then improving it - and ignoring what exactly "statistical control" means in software development). My experience of trying to introduce such stuff earlier (in ISO9k efforts) has been that the statistics merely measure noise & instability in the system, telling you little you don't know already. In other words, to follow an old software engineering adage, first design (the process) so that it works, then optimise it.

To put it another way: the staged model offers a strategy for what to do now, what to do later. Its philosophy is stabilise, then improve continuously - which logically puts proactive defect prevention last. What other strategies might there be for ordering the processes to put in place now, or later? (All together is too hard.)

> In addition, the SEPG can have
> the same problems you've referenced below with ISO auditors, in that they
> can end up driving an incredible bureaucracy that doesn't serve the
> developers or the rest of the product team.
>

A truth I had forgotten.

My experience is limited to organisations using the CMM willingly, mostly in Europe; not of organisations using it in the States unwillingly where I can imagine this happens (

> In addition, the SEPG can't address a lack of management commitment.
> This is no different than a distributed model for ISO implementation that
> lacks true top-level support....you can have the buy-in from the troops
> and middle management and still fizzle.
>
Nothing and nobody can address lack of management commitment except the CEO, agreed.

> > A TickIT certificate is somewhere around level 2/3 of the CMM - it skews
> > across. the CMM has more software detail, ISO has more general business
> > stuff, both useful and overlapping.
>
> Actually, there is no true correlation between maturity levels and ISO
> implementation. However, it is true that there is strong support for ISO
> at all CMM levels, including Defect Prevention at Level 5. If one were to
> take a true organizational approach to an ISO implementation, then it
> would very much represent a level 3 organization. Both models pretty much
> say the same thing, whats not hows.....but one takes about 479 pages to do
> it.

I think ultimately they say more or less the same thing. But I maintain that the CMM, with its concept of levels, gives a strategy for getting there.

Specifically, I think for a software organisation to concentrate as the CMM suggests on level 2 processes such as planning, tracking, configuration management etc first, is a high leverage focus which pays immediate dividends in customer satisfaction. I think ISO9k lacks such a strategic sense of how to build the QMS.

Further, I think some form of the SEI's self assessment methodology is vital. It directly implements Deming's "involve everyone". Where is that, specifically, in ISO9k or TickIT? ISO 9k should insist upon involvement of engineers in continuous improvement, as it insists upon other necessary practices. Why doesn't it?

That's not to say the CMM can't lose it, when driven by unreasoning management hunger for a "CMM level Certificate"; but at least it's there in the SEI method, and in Watts Humphrey's book "Managing the Software Process" (seminal CMM reference for those who want the reference).

> > If an organisation is immature, the CMM offers a better strategy for
> > building a QMS because it offers a sense of priority . TickIT and ISO
> > require everything and can be overwhelming.
>
> If you remove the models and look at software engineering fundamentals,
> you have the same problem of trying to bite off more than you can chew. A
> phased approach in any implementation is necessary. And I think alot has
> to do whether you agree with the construction of the CMM which pretty much
> focuses strictly on the management side at level 2 and doesn't have an
> engineering focus until level 3.

That's because engineers know what they're doing and managers know neither what they nor their engineers are doing ;o)

> The simple framework that ISO offers can be phased-in on a project by
> project basis, with the areas that offer the biggest bang for the buck
> being addressed first.
>
> The biggest problem I see in SPI is that folks don't have good
> implementation planning skills. This is the same for the CMM and
> ISO...and when you look at how large the CMM model is and how little has
> been written about how to successfully implement it.....well...the job can
> be daunting.
>
> Although more commercial organizations are looking to the CMM for process
> improvement, it pales in comparison to the organizations that must
> implement or else....DoD contractors in bidding wars.

In telecommunications too process improvement is vital - customers typically ask for ISO9k but increasingly they understand and respect the CMM ideas - without insisting on achievement of a magic level (yet).

> > Further, the continuous references to clauses and how auditors might
> > interpret them takes ownership away from the people and gives it to
> > auditors. The SEI's approach leaves ownership with those who operate the
> > process, so it's better balanced, less inclined to be bureaucratic. Compare
> > the discussion traffic in this List with, eg, comp.software-engineering.
>
> The same darn problem exists in the CMM world. Don't kid yourself.
> Organizations face SCEs (software capability evaluations) and CBA
> IPIs....CMM based assessments for Internal Process Improvement. Many
> times the "Level Rating" is all that matters....even with the CBA IPI
> approach which is supposed to be a collaborative exercise for improvement.
> You would be surprised how many organizations coach their employees to get
> ready for a CBA IPI, when that is not the intent.....it's not supposed to
> be about the "score"....it's supposed to be about improvement.
>
> Again....this is a management issue similar to the ISO implementation that
> says....let's get the certificate.....and we're done.....
>
Yes. If management are strong then the certificate is real, else it's just paper and grief.

> > One way to approach this is to build the QMS using the SEI's CMM guidance,
> > document it soundly, include a reconciliation with TickIT clauses - and add
> > in the bits that the CMM does not explicitly require (eg, contract review,
> > security & backups, etc).
>
> Yes...there is a lot of information in the CMM that can assist one with
> an ISO implementation. I would also say that folks can also turn to the
> IEEE standards or ISO 12207 or other sources of information.
>
Quite so. Also, for software management, www.spmn.com has some good stuff.

> It's all fundamental stuff.
>
Indeed. Fred Brooks said a whole bunch of it 25 years ago, and yet we still have to teach managers that months are mythical. Why so?

> > Under the CMM, you can be a level 2, 3, 4 or 5 organisation (or, sadly,
> > level 1).
>
> Yes...but you could be a level 3 organization and that would mean little
> in many circumstances. For example, when was the organization's last
> assessment....3 years ago? I've been in shops that tout their level 3
> profile but were behaving as level 1 (chaotic). Remember CMM ratings are
> not a certification scheme of any sort. There is no requirement other
> than individual customer or market requirements that would require you to
> reassess your organization.
>
> > Under TickIT, you can be TickIT Certified. There's no measurement scale.
> >
> > Regards,
> > Pat
>
> Yes and TickIT is just an ISO 9001 registration, pure and simple. But the
> true measurement in implementing either model is whether you have a return
> on investment and whether it translates into better product and staying
> abreast of your competitors.
>
> The companies that succeed with the CMM and ISO succeed because they
> aren't driven by what is in the model and they overlook the shortcomings
> of the models. They embrace what is good for their business and question
> what is unnecessary. They go beyond the models to crush their
> competition.

I agree with that whole heartedly. I think several questions in this list are from organisations who did just that and are having trouble convincing their auditors of it. It's important to pick an auditor who truly understands your business - a valuable element of TickIT - so that you get firm but fair treatment.

> I've seen both models work effectively...and...I've seen them both fail
> miserably.
>
> The same can be said about Deming, Juran, Crosby, TQM.....
>
> For the most part, the model doesn't matter...change does....

Yes.

> For some organizations models can be handy, because they can hang their
> failure on choosing one particular model over another.....but that's
> another story.....
>

posted 07 May 2000 01:35 PM               

From: Brian Charles Kohn [email protected]
Subject: RE: Tick-IT & ISO 9001 /.../Peter/Dey/Duong/Kohn

> I believe that CMM is better than ISO because CMM is dedicated to
> software. Whereas ISO was created for manufacturing first, adapted to
> software later.

Interesting argument, but I'm not terribly convinced. CMM doesn't provide much assurance to your customer beyond what your own word of honor could have provided.

If you look at the job market these days, especially in the IT industry, employers are beginning to really focus on whether job candidates are certified. Looking around this mailing list there are enough CQAs and QSLAs to fill 100 cans of alphabet soup.

Suppliers are not much different than people. To be sure that they are competent in meeting your needs as a purchaser, you need some assurances. Often, someone's word is enough; sometimes you need contractual protections; sometimes accredited certification to ISO 9001 or TickIT will give you the assurance you need. CMM just can't satisfy *that* need (though I feel it is surely better at satisfying the needs *it* was structured to meet.)

Brian

---------------snippo-----------------

From: ISO Standards Discussion
Date: Wed, 27 Jan 1999 09:18:30 -0600
Subject: Re: Tick-IT & ISO 9001 /../Dey/Deibler/Kohn

From: Brian Charles Kohn
Subject: RE: Tick-IT & ISO 9001 /.../Dey/Deibler/Kohn

> Since you've made some general statements about the CMM and ISO, I'd
> thought I'd play a little devil's advocate with you. My point throughout
> this response is to offer up a theme. The theme is that models really
> don't amount to a hill of beans as much as a desire to change.

I think the biggest mistake folks pursuing ISO 9001 and TickIT could make is mistaking them for models for improvement. These two standards are tools for demonstrating to customers that you meet minimally acceptable standards for addressing quality. Use them however you wish; get out of them whatever you can from the standpoint of improving your business; but never forget that the point of the standards is to protect the customer.

Brian

----------snippo-----------

From: ISO Standards Discussion
Date: Wed, 27 Jan 1999 09:20:08 -0600
Subject: Re: Tick-IT & ISO 9001 /Chen/Hale/Kohn

From: Brian Charles Kohn
Subject: RE: Tick-IT & ISO 9001 /Chen/Hale/Kohn

> The main thing you get with TickIT is an auditor with software
> qualifications.

Actually, you get an auditor with software qualifications when you get ISO 9001 registration services from any RAB- or RvA-accredited registrar, operating in compliance with procedures.

What you *do* get extra is assessment to a set of requirements that are either over-and-above ISO 9001, or simply more stringent or prescriptive than the corresponding requirement in ISO 9001.

> With Lloyd's Register Quality Assurance, the only extra you pay
> extra for is the $60 or so for the additional certification mark.

This is somewhat misleading. While this is perhaps true with Lloyd's, my experience is that the minimum number of assessor-days required to conduct a valid TickIT assessment are a bit more than the RvA requirements for the minimum number of assessor-days for an ISO 9001 assessment. That will make the costs proportionally more.