SBS - The best value in QMS software

Tick-IT vs. ISO 9001 vs. CMM - Software Quality Assurance

Marc

Hunkered Down for the Duration with a Mask on...
Staff member
Admin
#1
Tick-IT & ISO 9001

From: ISO Standards Discussion
Date: Wed, 27 Jan 1999 08:54:28 -0600
Subject: Re: Tick-IT & ISO 9001 /Chen/Kirk/Peter/Dey

> Pat,
>
> Since you've made some general statements about the CMM and ISO, I'd
> thought I'd play a little devil's advocate with you. My point throughout
> this response is to offer up a theme. The theme is that models really
> don't amount to a hill of beans as much as a desire to change. I make
> this statement in support of the fellow who said that all models are
> wrong....some are useful. I would say that all models have warts...and
> that the key to software process improvement (or any improvement) is
> wanting to improve....to deliver products better and faster...

I agree.

> > The strength of the CMM is its model for continuous improvement. The SEI's
> > method includes, not only the process framework (the CMM itself) but the
> > methods for managing improvement by involving everyone, eg, through a
> > Software Engineering Process Group.
>
> But the reality is that the CMM is a staged model that doesn't really
> focus on defect prevention until level 5.

Hmm ... well as one devil's advocate to another:

1. Reviews are at level 3 in the CMM and, I hope, can be used for defect detection & removal (yes, not proactive defect prevention, but better than nothing).

2. I'm inclined to pull reviews down to level 2 because of the civilising behaviour they can be used to introduce to organisations. (They're at level 3, I believe, because they need the time management disciplines that level 2 introduces.)

3. More advanced processes like defect prevention are introduced at higher levels in the CMM after making the system stable, especially with level 2 (ie, Deming's idea of bringing the system into statistical control, then improving it - and ignoring what exactly "statistical control" means in software development). My experience of trying to introduce such stuff earlier (in ISO9k efforts) has been that the statistics merely measure noise & instability in the system, telling you little you don't know already. In other words, to follow an old software engineering adage, first design (the process) so that it works, then optimise it.

To put it another way: the staged model offers a strategy for what to do now, what to do later. Its philosophy is stabilise, then improve continuously - which logically puts proactive defect prevention last. What other strategies might there be for ordering the processes to put in place now, or later? (All together is too hard.)

> In addition, the SEPG can have
> the same problems you've referenced below with ISO auditors, in that they
> can end up driving an incredible bureaucracy that doesn't serve the
> developers or the rest of the product team.
>

A truth I had forgotten.

My experience is limited to organisations using the CMM willingly, mostly in Europe; not of organisations using it in the States unwillingly where I can imagine this happens

> In addition, the SEPG can't address a lack of management commitment.
> This is no different than a distributed model for ISO implementation that
> lacks true top-level support....you can have the buy-in from the troops
> and middle management and still fizzle.
>
Nothing and nobody can address lack of management commitment except the CEO, agreed.

> > A TickIT certificate is somewhere around level 2/3 of the CMM - it skews
> > across. the CMM has more software detail, ISO has more general business
> > stuff, both useful and overlapping.
>
> Actually, there is no true correlation between maturity levels and ISO
> implementation. However, it is true that there is strong support for ISO
> at all CMM levels, including Defect Prevention at Level 5. If one were to
> take a true organizational approach to an ISO implementation, then it
> would very much represent a level 3 organization. Both models pretty much
> say the same thing, whats not hows.....but one takes about 479 pages to do
> it.

I think ultimately they say more or less the same thing. But I maintain that the CMM, with its concept of levels, gives a strategy for getting there.

Specifically, I think for a software organisation to concentrate as the CMM suggests on level 2 processes such as planning, tracking, configuration management etc first, is a high leverage focus which pays immediate dividends in customer satisfaction. I think ISO9k lacks such a strategic sense of how to build the QMS.

Further, I think some form of the SEI's self assessment methodology is vital. It directly implements Deming's "involve everyone". Where is that, specifically, in ISO9k or TickIT? ISO 9k should insist upon involvement of engineers in continuous improvement, as it insists upon other necessary practices. Why doesn't it?

That's not to say the CMM can't lose it, when driven by unreasoning management hunger for a "CMM level Certificate"; but at least it's there in the SEI method, and in Watts Humphrey's book "Managing the Software Process" (seminal CMM reference for those who want the reference).

> > If an organisation is immature, the CMM offers a better strategy for
> > building a QMS because it offers a sense of priority . TickIT and ISO
> > require everything and can be overwhelming.
>
> If you remove the models and look at software engineering fundamentals,
> you have the same problem of trying to bite off more than you can chew. A
> phased approach in any implementation is necessary. And I think alot has
> to do whether you agree with the construction of the CMM which pretty much
> focuses strictly on the management side at level 2 and doesn't have an
> engineering focus until level 3.

That's because engineers know what they're doing and managers know neither what they nor their engineers are doing ;o)

> The simple framework that ISO offers can be phased-in on a project by
> project basis, with the areas that offer the biggest bang for the buck
> being addressed first.
>
> The biggest problem I see in SPI is that folks don't have good
> implementation planning skills. This is the same for the CMM and
> ISO...and when you look at how large the CMM model is and how little has
> been written about how to successfully implement it.....well...the job can
> be daunting.
>
> Although more commercial organizations are looking to the CMM for process
> improvement, it pales in comparison to the organizations that must
> implement or else....DoD contractors in bidding wars.

In telecommunications too process improvement is vital - customers typically ask for ISO9k but increasingly they understand and respect the CMM ideas - without insisting on achievement of a magic level (yet).

> > Further, the continuous references to clauses and how auditors might
> > interpret them takes ownership away from the people and gives it to
> > auditors. The SEI's approach leaves ownership with those who operate the
> > process, so it's better balanced, less inclined to be bureaucratic. Compare
> > the discussion traffic in this List with, eg, comp.software-engineering.
>
> The same darn problem exists in the CMM world. Don't kid yourself.
> Organizations face SCEs (software capability evaluations) and CBA
> IPIs....CMM based assessments for Internal Process Improvement. Many
> times the "Level Rating" is all that matters....even with the CBA IPI
> approach which is supposed to be a collaborative exercise for improvement.
> You would be surprised how many organizations coach their employees to get
> ready for a CBA IPI, when that is not the intent.....it's not supposed to
> be about the "score"....it's supposed to be about improvement.
>
> Again....this is a management issue similar to the ISO implementation that
> says....let's get the certificate.....and we're done.....
>
Yes. If management are strong then the certificate is real, else it's just paper and grief.

> > One way to approach this is to build the QMS using the SEI's CMM guidance,
> > document it soundly, include a reconciliation with TickIT clauses - and add
> > in the bits that the CMM does not explicitly require (eg, contract review,
> > security & backups, etc).
>
> Yes...there is a lot of information in the CMM that can assist one with
> an ISO implementation. I would also say that folks can also turn to the
> IEEE standards or ISO 12207 or other sources of information.
>
Quite so. Also, for software management, Software Program Managers Network has some good stuff.

> It's all fundamental stuff.
>
Indeed. Fred Brooks said a whole bunch of it 25 years ago, and yet we still have to teach managers that months are mythical. Why so?

> > Under the CMM, you can be a level 2, 3, 4 or 5 organisation (or, sadly,
> > level 1).
>
> Yes...but you could be a level 3 organization and that would mean little
> in many circumstances. For example, when was the organization's last
> assessment....3 years ago? I've been in shops that tout their level 3
> profile but were behaving as level 1 (chaotic). Remember CMM ratings are
> not a certification scheme of any sort. There is no requirement other
> than individual customer or market requirements that would require you to
> reassess your organization.
>
> > Under TickIT, you can be TickIT Certified. There's no measurement scale.
> >
> > Regards,
> > Pat
>
> Yes and TickIT is just an ISO 9001 registration, pure and simple. But the
> true measurement in implementing either model is whether you have a return
> on investment and whether it translates into better product and staying
> abreast of your competitors.
>
> The companies that succeed with the CMM and ISO succeed because they
> aren't driven by what is in the model and they overlook the shortcomings
> of the models. They embrace what is good for their business and question
> what is unnecessary. They go beyond the models to crush their
> competition.

I agree with that whole heartedly. I think several questions in this list are from organisations who did just that and are having trouble convincing their auditors of it. It's important to pick an auditor who truly understands your business - a valuable element of TickIT - so that you get firm but fair treatment.

> I've seen both models work effectively...and...I've seen them both fail
> miserably.
>
> The same can be said about Deming, Juran, Crosby, TQM.....
>
> For the most part, the model doesn't matter...change does....

Yes.

> For some organizations models can be handy, because they can hang their
> failure on choosing one particular model over another.....but that's
> another story.....
>

Regards,
Pat
 
Elsmar Forum Sponsor

Marc

Hunkered Down for the Duration with a Mask on...
Staff member
Admin
#2
From: ISO Standards Discussion
Date: Wed, 27 Jan 1999 09:17:08 -0600
Subject: Re: Tick-IT & ISO 9001 /.../Peter/Dey/Duong/Kohn

> I believe that CMM is better than ISO because CMM is dedicated to
> software. Whereas ISO was created for manufacturing first, adapted to
> software later.

Interesting argument, but I'm not terribly convinced. CMM doesn't provide much assurance to your customer beyond what your own word of honor could have provided.

If you look at the job market these days, especially in the IT industry, employers are beginning to really focus on whether job candidates are certified. Looking around this mailing list there are enough CQAs and QSLAs to fill 100 cans of alphabet soup.

Suppliers are not much different than people. To be sure that they are competent in meeting your needs as a purchaser, you need some assurances. Often, someone's word is enough; sometimes you need contractual protections; sometimes accredited certification to ISO 9001 or TickIT will give you the assurance you need. CMM just can't satisfy *that* need (though I feel it is surely better at satisfying the needs *it* was structured to meet.)

Brian

---------------snippo-----------------

From: ISO Standards Discussion
Date: Wed, 27 Jan 1999 09:18:30 -0600
Subject: Re: Tick-IT & ISO 9001 /../Dey/Deibler/Kohn

> Since you've made some general statements about the CMM and ISO, I'd
> thought I'd play a little devil's advocate with you. My point throughout
> this response is to offer up a theme. The theme is that models really
> don't amount to a hill of beans as much as a desire to change.

I think the biggest mistake folks pursuing ISO 9001 and TickIT could make is mistaking them for models for improvement. These two standards are tools for demonstrating to customers that you meet minimally acceptable standards for addressing quality. Use them however you wish; get out of them whatever you can from the standpoint of improving your business; but never forget that the point of the standards is to protect the customer.

Brian

----------snippo-----------

From: ISO Standards Discussion <[email protected]>
Date: Wed, 27 Jan 1999 09:20:08 -0600
Subject: Re: Tick-IT & ISO 9001 /Chen/Hale/Kohn

> The main thing you get with TickIT is an auditor with software
> qualifications.

Actually, you get an auditor with software qualifications when you get ISO 9001 registration services from any RAB- or RvA-accredited registrar, operating in compliance with procedures.

What you *do* get extra is assessment to a set of requirements that are either over-and-above ISO 9001, or simply more stringent or prescriptive than the corresponding requirement in ISO 9001.

> With Lloyd's Register Quality Assurance, the only extra you pay
> extra for is the $60 or so for the additional certification mark.

This is somewhat misleading. While this is perhaps true with Lloyd's, my experience is that the minimum number of assessor-days required to conduct a valid TickIT assessment are a bit more than the RvA requirements for the minimum number of assessor-days for an ISO 9001 assessment. That will make the costs proportionally more.

Brian
 
Thread starter Similar threads Forum Replies Date
T Directors View - Standards like ISO 9001 are "Just a TICK in the BOX" ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 4
C Measurement Uncertainty fluctuates half a tick mark (20 millionths of an inch) Measurement Uncertainty (MU) 1
V Tick IT - Regional representative for TICKIT at Chennai Software Quality Assurance 5
Le Chiffre Online training available for ISO/IEC 17021-1: Requirements for bodies providing audit and certification of management systems Training - Internal, External, Online and Distance Learning 2
B ISO 6508 and portable hardness measurement instruments General Measurement Device and Calibration Topics 0
M Scope for ISO 13485 Certification of a Translation Service Provider ISO 13485:2016 - Medical Device Quality Management Systems 2
S Knee Implant (Femoral -Cobalt chrome)-Sub chronic toxicity test (ISO 10993-11)choice of root Medical Device and FDA Regulations and Standards News 2
Sidney Vianna Release of ISO 10013:2021, Quality management systems – Guidance for documented information Other ISO and International Standards and European Regulations 0
K Integrating ISO 9001:2015 with ISO 17025:2017 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
Q ISO 13485 7.5.6 Validation - Off the shelf Software ISO 13485:2016 - Medical Device Quality Management Systems 3
A ISO 13485 Certification for Resin Manufacturer ISO 13485:2016 - Medical Device Quality Management Systems 4
R Remote Audits for ISO 9001 (or any other standard) General Auditing Discussions 29
A ISO 13485 Sterilization Clause Applicability ISO 13485:2016 - Medical Device Quality Management Systems 7
K ISO 13485 and compliance of electronic signature ISO 13485:2016 - Medical Device Quality Management Systems 5
T ISO 13485 - Assembly instructions written vs. online ISO 13485:2016 - Medical Device Quality Management Systems 5
M ISO 13485:2016 internal audit checklist Medical Device and FDA Regulations and Standards News 5
D ISO 17021 Certificate Registrars and Notified Bodies 1
X ISO 17025 certification for Laboratory for online gambling products ISO 17025 related Discussions 3
C ISO 19227 Validation Cost Other Medical Device Related Standards 2
N 93/42/EEC certification without ISO 13485 EU Medical Device Regulations 3
T Relationship between ISO 9001 and ISO – IEC BS EN 870079- 34 2020 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 5
M How Specific in an ISO 13485:2016 Scope for a Contract Manufacturer ISO 13485:2016 - Medical Device Quality Management Systems 9
A ISO 13485 for Class 1 Medical Device ISO 13485:2016 - Medical Device Quality Management Systems 7
S Sequence of ISO 9001:2015 Implementation Steps ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 6
qualprod Business Continuity Planning in ISO 9001? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
T Help with BS EN ISO - IEC 80079-34 2020 (Explosive atmospheres QMS) Other ISO and International Standards and European Regulations 0
C Sampling - ISO 16269-6 vs ISO 2859 Other ISO and International Standards and European Regulations 0
Brizilla Employee Data Privacy Policy - ISO 9001:2015 requirement(s)? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 6
R ISO 17025 - ISO Guides 33 & 80 ISO 17025 related Discussions 1
0 ISO 13485:2016 Chapter 8 Integration of the subsections ISO 13485:2016 - Medical Device Quality Management Systems 1
S ISO 9001:2015 Internal Auditing Internal Auditing 8
Q Process: Knowledge Section 7.1.6 of ISO 9001:2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 10
N ISO 27001 for Jumb Burger - Risk Assessment sheet IEC 27001 - Information Security Management Systems (ISMS) 10
H What ISO certification is for an IT department? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 13
M Change in Constitution / Ownership of firm -------ISO 13485 ISO 13485:2016 - Medical Device Quality Management Systems 1
P ISO 9001 certification with zero customers? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 11
M ISO 14971 Determination of Competent Persons ISO 14971 - Medical Device Risk Management 4
R Clinical accuracy and repeatability of IR(infrared) thermometer, no maximum error criteria is recommended in ISO Other Medical Device Related Standards 11
Q The scope of ISO 21534 Other Medical Device Related Standards 0
A What must be recorded? (ISO 9001:2015, subclause 10.2) ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 9
M ISO 14971:2019: Criteria for overall residual risk ISO 14971 - Medical Device Risk Management 4
B Updated IATF 16949 - Will IATF 16949 get revised when ISO 9001:202X is released? IATF 16949 - Automotive Quality Systems Standard 4
S ISO 9001:2015 vs 21 CFR Part 211 matrix Pharmaceuticals (21 CFR Part 210, 21 CFR Part 211 and related Regulations) 0
N Quality Compliance Officer - ISO 13485, London Job Openings, Consulting and Employment Opportunities 1
T ISO 17025: Lockout/Tagout Requirement ISO 17025 related Discussions 1
S ISO 9001 implementation in a Gold exporting business ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
M Does the ISO 9001:2015 standard require a disaster recovery plan or emergency response plan ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
E ISO 13485 QMS certification as a Supplier ISO 13485:2016 - Medical Device Quality Management Systems 8
A Environmental Compliance obligations and risks (ISO 14001:2015 6.1.3) ISO 14001:2015 Specific Discussions 3
R ISO 17025 vertical audit checklist wanted Document Control Systems, Procedures, Forms and Templates 2

Similar threads

Top Bottom