Tick-IT vs. ISO 9001 vs. CMM - Software Quality Assurance

Marc

Hunkered Down for the Duration
Staff member
Admin
#1
Tick-IT & ISO 9001

From: ISO Standards Discussion
Date: Wed, 27 Jan 1999 08:54:28 -0600
Subject: Re: Tick-IT & ISO 9001 /Chen/Kirk/Peter/Dey

> Pat,
>
> Since you've made some general statements about the CMM and ISO, I'd
> thought I'd play a little devil's advocate with you. My point throughout
> this response is to offer up a theme. The theme is that models really
> don't amount to a hill of beans as much as a desire to change. I make
> this statement in support of the fellow who said that all models are
> wrong....some are useful. I would say that all models have warts...and
> that the key to software process improvement (or any improvement) is
> wanting to improve....to deliver products better and faster...

I agree.

> > The strength of the CMM is its model for continuous improvement. The SEI's
> > method includes, not only the process framework (the CMM itself) but the
> > methods for managing improvement by involving everyone, eg, through a
> > Software Engineering Process Group.
>
> But the reality is that the CMM is a staged model that doesn't really
> focus on defect prevention until level 5.

Hmm ... well as one devil's advocate to another:

1. Reviews are at level 3 in the CMM and, I hope, can be used for defect detection & removal (yes, not proactive defect prevention, but better than nothing).

2. I'm inclined to pull reviews down to level 2 because of the civilising behaviour they can be used to introduce to organisations. (They're at level 3, I believe, because they need the time management disciplines that level 2 introduces.)

3. More advanced processes like defect prevention are introduced at higher levels in the CMM after making the system stable, especially with level 2 (ie, Deming's idea of bringing the system into statistical control, then improving it - and ignoring what exactly "statistical control" means in software development). My experience of trying to introduce such stuff earlier (in ISO9k efforts) has been that the statistics merely measure noise & instability in the system, telling you little you don't know already. In other words, to follow an old software engineering adage, first design (the process) so that it works, then optimise it.

To put it another way: the staged model offers a strategy for what to do now, what to do later. Its philosophy is stabilise, then improve continuously - which logically puts proactive defect prevention last. What other strategies might there be for ordering the processes to put in place now, or later? (All together is too hard.)

> In addition, the SEPG can have
> the same problems you've referenced below with ISO auditors, in that they
> can end up driving an incredible bureaucracy that doesn't serve the
> developers or the rest of the product team.
>

A truth I had forgotten.

My experience is limited to organisations using the CMM willingly, mostly in Europe; not of organisations using it in the States unwillingly where I can imagine this happens

> In addition, the SEPG can't address a lack of management commitment.
> This is no different than a distributed model for ISO implementation that
> lacks true top-level support....you can have the buy-in from the troops
> and middle management and still fizzle.
>
Nothing and nobody can address lack of management commitment except the CEO, agreed.

> > A TickIT certificate is somewhere around level 2/3 of the CMM - it skews
> > across. the CMM has more software detail, ISO has more general business
> > stuff, both useful and overlapping.
>
> Actually, there is no true correlation between maturity levels and ISO
> implementation. However, it is true that there is strong support for ISO
> at all CMM levels, including Defect Prevention at Level 5. If one were to
> take a true organizational approach to an ISO implementation, then it
> would very much represent a level 3 organization. Both models pretty much
> say the same thing, whats not hows.....but one takes about 479 pages to do
> it.

I think ultimately they say more or less the same thing. But I maintain that the CMM, with its concept of levels, gives a strategy for getting there.

Specifically, I think for a software organisation to concentrate as the CMM suggests on level 2 processes such as planning, tracking, configuration management etc first, is a high leverage focus which pays immediate dividends in customer satisfaction. I think ISO9k lacks such a strategic sense of how to build the QMS.

Further, I think some form of the SEI's self assessment methodology is vital. It directly implements Deming's "involve everyone". Where is that, specifically, in ISO9k or TickIT? ISO 9k should insist upon involvement of engineers in continuous improvement, as it insists upon other necessary practices. Why doesn't it?

That's not to say the CMM can't lose it, when driven by unreasoning management hunger for a "CMM level Certificate"; but at least it's there in the SEI method, and in Watts Humphrey's book "Managing the Software Process" (seminal CMM reference for those who want the reference).

> > If an organisation is immature, the CMM offers a better strategy for
> > building a QMS because it offers a sense of priority . TickIT and ISO
> > require everything and can be overwhelming.
>
> If you remove the models and look at software engineering fundamentals,
> you have the same problem of trying to bite off more than you can chew. A
> phased approach in any implementation is necessary. And I think alot has
> to do whether you agree with the construction of the CMM which pretty much
> focuses strictly on the management side at level 2 and doesn't have an
> engineering focus until level 3.

That's because engineers know what they're doing and managers know neither what they nor their engineers are doing ;o)

> The simple framework that ISO offers can be phased-in on a project by
> project basis, with the areas that offer the biggest bang for the buck
> being addressed first.
>
> The biggest problem I see in SPI is that folks don't have good
> implementation planning skills. This is the same for the CMM and
> ISO...and when you look at how large the CMM model is and how little has
> been written about how to successfully implement it.....well...the job can
> be daunting.
>
> Although more commercial organizations are looking to the CMM for process
> improvement, it pales in comparison to the organizations that must
> implement or else....DoD contractors in bidding wars.

In telecommunications too process improvement is vital - customers typically ask for ISO9k but increasingly they understand and respect the CMM ideas - without insisting on achievement of a magic level (yet).

> > Further, the continuous references to clauses and how auditors might
> > interpret them takes ownership away from the people and gives it to
> > auditors. The SEI's approach leaves ownership with those who operate the
> > process, so it's better balanced, less inclined to be bureaucratic. Compare
> > the discussion traffic in this List with, eg, comp.software-engineering.
>
> The same darn problem exists in the CMM world. Don't kid yourself.
> Organizations face SCEs (software capability evaluations) and CBA
> IPIs....CMM based assessments for Internal Process Improvement. Many
> times the "Level Rating" is all that matters....even with the CBA IPI
> approach which is supposed to be a collaborative exercise for improvement.
> You would be surprised how many organizations coach their employees to get
> ready for a CBA IPI, when that is not the intent.....it's not supposed to
> be about the "score"....it's supposed to be about improvement.
>
> Again....this is a management issue similar to the ISO implementation that
> says....let's get the certificate.....and we're done.....
>
Yes. If management are strong then the certificate is real, else it's just paper and grief.

> > One way to approach this is to build the QMS using the SEI's CMM guidance,
> > document it soundly, include a reconciliation with TickIT clauses - and add
> > in the bits that the CMM does not explicitly require (eg, contract review,
> > security & backups, etc).
>
> Yes...there is a lot of information in the CMM that can assist one with
> an ISO implementation. I would also say that folks can also turn to the
> IEEE standards or ISO 12207 or other sources of information.
>
Quite so. Also, for software management, Software Program Managers Network has some good stuff.

> It's all fundamental stuff.
>
Indeed. Fred Brooks said a whole bunch of it 25 years ago, and yet we still have to teach managers that months are mythical. Why so?

> > Under the CMM, you can be a level 2, 3, 4 or 5 organisation (or, sadly,
> > level 1).
>
> Yes...but you could be a level 3 organization and that would mean little
> in many circumstances. For example, when was the organization's last
> assessment....3 years ago? I've been in shops that tout their level 3
> profile but were behaving as level 1 (chaotic). Remember CMM ratings are
> not a certification scheme of any sort. There is no requirement other
> than individual customer or market requirements that would require you to
> reassess your organization.
>
> > Under TickIT, you can be TickIT Certified. There's no measurement scale.
> >
> > Regards,
> > Pat
>
> Yes and TickIT is just an ISO 9001 registration, pure and simple. But the
> true measurement in implementing either model is whether you have a return
> on investment and whether it translates into better product and staying
> abreast of your competitors.
>
> The companies that succeed with the CMM and ISO succeed because they
> aren't driven by what is in the model and they overlook the shortcomings
> of the models. They embrace what is good for their business and question
> what is unnecessary. They go beyond the models to crush their
> competition.

I agree with that whole heartedly. I think several questions in this list are from organisations who did just that and are having trouble convincing their auditors of it. It's important to pick an auditor who truly understands your business - a valuable element of TickIT - so that you get firm but fair treatment.

> I've seen both models work effectively...and...I've seen them both fail
> miserably.
>
> The same can be said about Deming, Juran, Crosby, TQM.....
>
> For the most part, the model doesn't matter...change does....

Yes.

> For some organizations models can be handy, because they can hang their
> failure on choosing one particular model over another.....but that's
> another story.....
>

Regards,
Pat
 
Elsmar Forum Sponsor

Marc

Hunkered Down for the Duration
Staff member
Admin
#2
From: ISO Standards Discussion
Date: Wed, 27 Jan 1999 09:17:08 -0600
Subject: Re: Tick-IT & ISO 9001 /.../Peter/Dey/Duong/Kohn

> I believe that CMM is better than ISO because CMM is dedicated to
> software. Whereas ISO was created for manufacturing first, adapted to
> software later.

Interesting argument, but I'm not terribly convinced. CMM doesn't provide much assurance to your customer beyond what your own word of honor could have provided.

If you look at the job market these days, especially in the IT industry, employers are beginning to really focus on whether job candidates are certified. Looking around this mailing list there are enough CQAs and QSLAs to fill 100 cans of alphabet soup.

Suppliers are not much different than people. To be sure that they are competent in meeting your needs as a purchaser, you need some assurances. Often, someone's word is enough; sometimes you need contractual protections; sometimes accredited certification to ISO 9001 or TickIT will give you the assurance you need. CMM just can't satisfy *that* need (though I feel it is surely better at satisfying the needs *it* was structured to meet.)

Brian

---------------snippo-----------------

From: ISO Standards Discussion
Date: Wed, 27 Jan 1999 09:18:30 -0600
Subject: Re: Tick-IT & ISO 9001 /../Dey/Deibler/Kohn

> Since you've made some general statements about the CMM and ISO, I'd
> thought I'd play a little devil's advocate with you. My point throughout
> this response is to offer up a theme. The theme is that models really
> don't amount to a hill of beans as much as a desire to change.

I think the biggest mistake folks pursuing ISO 9001 and TickIT could make is mistaking them for models for improvement. These two standards are tools for demonstrating to customers that you meet minimally acceptable standards for addressing quality. Use them however you wish; get out of them whatever you can from the standpoint of improving your business; but never forget that the point of the standards is to protect the customer.

Brian

----------snippo-----------

From: ISO Standards Discussion <[email protected]>
Date: Wed, 27 Jan 1999 09:20:08 -0600
Subject: Re: Tick-IT & ISO 9001 /Chen/Hale/Kohn

> The main thing you get with TickIT is an auditor with software
> qualifications.

Actually, you get an auditor with software qualifications when you get ISO 9001 registration services from any RAB- or RvA-accredited registrar, operating in compliance with procedures.

What you *do* get extra is assessment to a set of requirements that are either over-and-above ISO 9001, or simply more stringent or prescriptive than the corresponding requirement in ISO 9001.

> With Lloyd's Register Quality Assurance, the only extra you pay
> extra for is the $60 or so for the additional certification mark.

This is somewhat misleading. While this is perhaps true with Lloyd's, my experience is that the minimum number of assessor-days required to conduct a valid TickIT assessment are a bit more than the RvA requirements for the minimum number of assessor-days for an ISO 9001 assessment. That will make the costs proportionally more.

Brian
 
Thread starter Similar threads Forum Replies Date
T Directors View - Standards like ISO 9001 are "Just a TICK in the BOX" ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 4
C Measurement Uncertainty fluctuates half a tick mark (20 millionths of an inch) Measurement Uncertainty (MU) 1
V Tick IT - Regional representative for TICKIT at Chennai Software Quality Assurance 5
A Scope of ISO 13485 certificate ISO 13485:2016 - Medical Device Quality Management Systems 1
A ASL requirement when the supplier is certified for ISO 13485 ISO 13485:2016 - Medical Device Quality Management Systems 6
D ISO 9001 certificate issued by QMS International for 10 years - legit? Registrars and Notified Bodies 12
Z Does anyone have experience with EN ISO 17664 ? IEC 62366 - Medical Device Usability Engineering 6
K Medical Device Repairs and ISO Scope ISO 13485:2016 - Medical Device Quality Management Systems 3
K Software Updates in the Field and ISO scope ISO 13485:2016 - Medical Device Quality Management Systems 0
M ISO 13485-2016 online certification ISO 13485:2016 - Medical Device Quality Management Systems 3
Z Auditor Findings ISO 14001:2015 vs. 45001:2015 ISO 14001:2015 Specific Discussions 5
S Thoughts on managing ISO 9001, 13485, IATF 16949 and 17025 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 26
S Supplier Management ISO 13485: 2016- Which supplier needs to fill in a self assessment form? ISO 13485:2016 - Medical Device Quality Management Systems 6
C ISO/IEC 17021-1 clause 7.1.2 - Determination of competence criteria Document Control Systems, Procedures, Forms and Templates 2
G ISO 17023 2017-11 - Suggestions for good books ISO 17025 related Discussions 0
B Timeframe for updating QMS / transitioning from ISO 14971:2012 to ISO 14971:2019 ISO 14971 - Medical Device Risk Management 5
M ISO 9001:2015 and AS6081:2012 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2
B Gage calibration frequency, ISO and IATF - What are the requirements Calibration Frequency (Interval) 3
M FDA News FDA Releases Draft Guidance Clarifying Application of ISO 10993-1 Biocompatibility Standard Medical Device and FDA Regulations and Standards News 0
C Implementation ISO 9001: 2015 ? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
C ISO/ IEC 17021 Resource requirement (need help) Document Control Systems, Procedures, Forms and Templates 5
J Possible to get ISO 13485 certified with only OEM Product? ISO 13485:2016 - Medical Device Quality Management Systems 4
D ISO 14971:2019 vs MDR Annex 1, Requirement #4 - "Manufacturers shall inform users of any residual risks" ISO 14971 - Medical Device Risk Management 2
D Definition of equipment for ISO 13485:2016 ISO 13485:2016 - Medical Device Quality Management Systems 1
L Biological Assessment (ISO 10993-1) Other Medical Device Related Standards 1
M ISO 13485:2016 Complaint Definition Clarity Customer Complaints 2
eule del ayre Documented Information - Periodic Review of Documents? IATF 16949:2016 / ISO 9001:2015 IATF 16949 - Automotive Quality Systems Standard 34
J ISO 9001:2015, ISO 14001 & OHSAS18000 (IMS) ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 1
G ISO 14001 - 6.1.3 Compliance Obligations ISO 14001:2015 Specific Discussions 1
D Rules for Paper Forms outside of an eQMS - 3 Questions (ISO 13485) Document Control Systems, Procedures, Forms and Templates 9
S Qualification question - ISO 13485 Reliability Analysis - Predictions, Testing and Standards 1
K ISO 13485 clause 8.5.2 'Any necessary CA shall be taken without undue delay' ISO 13485:2016 - Medical Device Quality Management Systems 11
Aymaneh ISO 11607-1: 2019 main changes Other Medical Device Related Standards 2
G National Structural Steel Specification 7th Edition - Do I now have to be audited against ISO 3843-3 as well as ISO 9001? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 1
J How much to charge for helping a startup company with initial ISO 13485 certification? Consultants and Consulting 3
J ISO 13485 System 'soft start' - How to best reflect this in initial audits, management review minutes and other records? ISO 13485:2016 - Medical Device Quality Management Systems 3
L How to understand the clause 6 Planning of ISO 9001:2015 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 11
D ISO 13485 - 7.3.6 Design and development verification - Do most folks create a separate SOP? ISO 13485:2016 - Medical Device Quality Management Systems 4
B ISO 8536-4 Contamination Index ISO 13485:2016 - Medical Device Quality Management Systems 0
S Practical Implementation of ISO 14971 ISO 14971 - Medical Device Risk Management 6
C ISO 13485 :2016 - CAPA - Does every CAPA need to be checked by regulations? ISO 13485:2016 - Medical Device Quality Management Systems 9
L Process changes and biocompatibility (ISO 10993-1) Other Medical Device Related Standards 1
J Recommendations for online ISO 19011 training? Training - Internal, External, Online and Distance Learning 6
D ISO 13485 8.2.1 and 8.2.2 - Customer Feedback and Customer Complaints ISO 13485:2016 - Medical Device Quality Management Systems 5
S Requirements to obtain ISO 50001 Certification ISO 14001:2015 Specific Discussions 2
A ISO 11135:2014, B.1.4, BI resistance x product bioburden ISO 13485:2016 - Medical Device Quality Management Systems 6
Sravan Manchikanti How to interpret '8.3 Control of nonconforming product' for SaMD device while implementing ISO 13485 & MDSAP ISO 13485:2016 - Medical Device Quality Management Systems 4
J Sister-company providing parts is only ISO 9001 registered IATF 16949 - Automotive Quality Systems Standard 7
M Getting started in ISO 13485 ISO 13485:2016 - Medical Device Quality Management Systems 21
G Copy of withdrawn ISO 9001:1994 Quality Management Standard ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 2

Similar threads

Top Bottom