Design Verification vs. Validation - What is YOUR definition?

[Marc]

I went thru this argument (discussion) some years ago so when I saw this I thought I'd post it.

-----snippo-----

Subject: Re: Design Validation -REQ: Clarification /Meron
Date: Thu, 22 Oct 1998 11:38:15 -0600
From: ISO Standards Discussion

From: Emanuel Meron
Subject: RE: Design Validation -REQ: Clarification /Meron

> From: Norm Ennis
> Subject: Q: Design Validation - Re: Clarification /Ennis
>
> I am seeking some clarification on a good defination of the term
> "Design Validation".
>
> I am working in the telecommunications field with a company that has an
> R+D dept. We are working on getting R+D ready for an ISO 9001 review in
> December and I am stumbling over this "Design Validation" section.
> The more I look at it, it seems to be a Marketing area. (ex. "Does the
> customer really want Pink Flamingos (sp) or would they prefer blue swans??)
>
> However, this would put the issue before Design Input / Output not after.
> Yet ISO indicates that Design Validation comes after Design Verification.
> Ref: "Design validation follows successfull design verification."
>
> Para. 4.4.7 states "..design verification shall be performed to ensure
> that design-stage output meets the design-stage input requirements."
>
> At our facility, Marketing determines customer need and feeds that info to
> R+D thru a specific document that defines specifications, delivery dates,
> etc. R+Ds only customer is Marketing. Once Marketing defines their
> requirements, R+D creates the device that meets those requirements. R+D
> verifies that the design functions to the design output specs.
>
> So where does that leave Validation?
>
> Any help would be appreciated.
>
> Yours,
>
> Norm Ennis, ECI Telecom, USA
>

These are FDA's (an agency notorious for their strict definitions...) of design verification and design validation:

Design verification means confirmation by examination and provision of objective evidence that specified requirements have been fulfilled.

Design validation means establishing by objective evidence that device (product) specifications conform with user needs and intended use(s).

In other words:

You verify a design by checking drawings and specs, running simulations, checking that all design requirements have been addressed, that calculations are correct, etc. It's mainly a "paper" exercise and when you are through with it you should be quite confident that the design is complete and that a product that will eventually be built according to those drawings and specs stands a good chance of conforming to the requirements in the real world.

You validate a design by trying out actual products (an initial run or batch of products), built as above by real workers, installed and operated in the real environment of use by real operators, etc. It's the proverbial proof of the pudding where you can catch errors and other problems that escaped all former verification efforts ... and others bugs that might have crept in later on.

Hope this helps. Comments anyone?

Emanuel Meron

[Marc]

And this came in as well:

-------snippo-----

Subject: Re: Q: Design 4.4/ Ennis/Eastick
Date: Thu, 22 Oct 1998 14:50:33 -0600
From: ISO Standards Discussion

From: Doug Eastick
Subject: Re: Q: Design 4.4/ Ennis/Eastick

ISO 8402:1994(E/F/R) in my ISO Compendium has the following definitions:

verification: confirmation by examination and provision of OBJECTIVE EVIDENCE (2.19) that specified requirements have been fulfilled.

validation: confirmation by examination and provision of OBJECTIVE EVIDENCE (2.19) that the particular requirements for a specific intended use are fulfilled.

In my own words, I would say that verification is like "inspect as per drawing" -- comparing the finished part/product to the original specifications (does it have 4 wheels as specified?). Whereas validation is "does it do that the intended use is" (does it roll down the road?).

...glad I had to look this up since I'm upgrading from 9002 to 9001 this winter :-).

Doug Eastick, P.Eng
QA Engineer
Bristol Machine Works Ltd.

[Roger Eastin]

This discussion is very helpful to me. Thanks for posting it. This is one of those "common sense" areas for design that the verbage used in ISO9000 can make too complicated (for simple minds like mine, anyway).

[Marc]

And it rears its ugly head again:

From: ISO 9000 Standards Discussion
Date: Wed, 21 Mar 2001 15:11:32 -0600
Subject: Re: Validation and Verification /.../Pfrang/Mackenzie/Isackson/

From: FIsackson

Mr. McKenzie wrote:

>Let's keep it simple:
>Verification: Does it work
>Validation: is it what we wanted

Although I'm as much as aficionado of the KISS axiom as the next, I'm much fonder of the the FDA's take on the V & V issue, to wit:

S 820.3(z) Validation means confirmation by examination and provision of objective evidence that the particular requirements for a specific intended use can be consistently fulfilled. (1) Process Validation means establishing by objective evidence that a process consistently produces a result or product meeting its predetermined specifications. (2) Design Validation means establishing by objective evidence that device specifications conform with user needs and intended use(s).

S820.3(aa) Verification means confirmation by examination and provision of objective evidence that specified requirements have been fulfilled.Whereas verification is a detailed examination of aspects of a design at various stages in the development, design validation is a cumulative summation of all efforts to assure that the design will conform with user needs and intended use(s), given expected variations in components, materials, manufacturing processes, and the use environment.

Gentlefolk, man your retorts.

Frank Isackson
From: ISO 9000 Standards Discussion
Date: Wed, 21 Mar 2001 15:12:06 -0600
Subject: Re: Validation and Verification /.../Pfrang/Mackenzie/Paten/

From: "Mike Paten"

Jim,

I have to disagree - keeping it simple:

Verification - has it been built (or provided) per spec
Validation - does it work (or accomplish the intended result)

Mike Paten

From: ISO 9000 Standards Discussion
Date: Wed, 21 Mar 2001 15:12:34 -0600
Subject: Re: Validation and Verification /.../Pfrang/Mackenzie/Myhrberg/

From: "Erik V. Myhrberg"

Jim:

Almost there -

Design Verification - does the output information match the input requirements (check it).
Design Validation - does the product and/or service perform to the requirements (try it).

Erik Myhrberg

From: ISO 9000 Standards Discussion
Date: Wed, 21 Mar 2001 15:13:39 -0600
Subject: Re: Validation and Verification /.../Pfrang/Mackenzie/Griver/

From: "Jon Griver"

I would like to bring this discussion to the core practical point of how to design validation tests of the basis of the definition of validation.

My discomfort with the term 'intended use' has been strengthened by this discussion, as in the following quote from Doug Pfrang

>
In the real world, product designers sometimes don't correctly anticipate all of the things the customer will want the product to do, and they discover that the product exhibits certain unanticipated or undesirable characteristics, from the user's perspective.
>

The implication is that the designer cannot define particular validation tests and all he can do is give it to the user and see what happens. If validation testing goes beyond testing against requirements which are in design input, then the designer is not in a position to define measurable validation tests.

Maybe I am taking too rigorous an approach in looking for a definition of validation that will make it clear how to define validation tests, and this subject is by its nature, a bit woolly.

Your comments please,

Jon Griver

[Marc]

From: ISO 9000 Standards Discussion
Date: Mon, 19 Mar 2001 16:08:13 -0600
Subject: Re: Validation and Verification /.../Pfrang/

From: Doug Pfrang

Hello All,

After reading many of the responses to the verification/validation problem, I keep wondering why, after all these years, this is still debated. I am also surprised at how unhelpful most of the examples posted on this list appear to be, because they often do not accurately differentiate between the two concepts. I hope the description below will help.

Verification means examining all of the things you have anticipated the customer will want the product to do, and checking to see if the product does those things the way you have anticipated the customer will want. Verification is about checking the things you have intentionally designed into the product and making sure the product meets those design requirements. It is about whether you were smart enough to _meet_ your specified design goals.

Validation means checking to see if the product does what the customer (or user) actually wants the product to do under real-world conditions, or as-close-to-real-world conditions as you can possibly simulate. Validation encompasses all of the things that can be verified, as well as all of the things that cannot -- i.e., all of the things that the product designers might never have anticipated the customer might want or expect the product to do. It is about whether you were smart enough to specify the _right_ design goals.

Of course, in the ideal world, product designers would correctly anticipate all of the things the customer would ever want the product to do, they would specify all the right design goals, and they would meet them. As a result, the validation would reveal nothing beyond what the verification covered.

In the real world, product designers sometimes don't correctly anticipate all of the things the customer will want the product to do, and they discover that the product exhibits certain unanticipated or undesirable characteristics, from the user's perspective.

Thus, the difference between verification and validation is that verification checks the adequacy of the design with respect to _identified_ design goals, whereas validation checks the adequacy of the design with respect to both identified and _unidentified_ customer expectataions.

If you want an example of the difference between verification and validation, consider the difference between the foot-operated controls and the hand-operated controls on most cars. If you rent a car, even a model you have never driven before, you always know which foot pedal makes the car go and which foot pedal makes the car stop. In fact, virtually any driver can easily get into any rental car -- at night or in the rain -- and operate the foot pedals to make the car go and stop. This is an example of a design that passes both verification and validation, because the pedals work as the designers intended (verification) and they also work as the user expects, even under foreseeable adverse use conditions (validation) such as darkness or rain.

Now consider the hand-operated controls. I don't know about you, but there have been times when I have rented a car and simply not known how to operate some of the critical controls. For example, the stalk to the left of the steering wheel might operate the headlights, it might operate the windshield wipers, it might operate both, or it might operate neither. If the controls differ from the car I own and drive daily, I might reach for the stalk in an effort to turn on the windshield wipers, and instead turn on the headlights. This is an example of a design that passes verification, because the controls perform their stated function; but it fails validation, because the controls do not work as the user expects under foreseeable conditions of use. Indeed, in this example, a user's inability to correctly operate the vehicle controls could, at a critical moment, cause the user to be unable to see oncoming hazards on the road, possibly with fatal consequences.

This illustrates one more thing about verification and validation: instead of "intended use," which many people think is relevant to a design validation, I find it far more helpful to think in terms of "reasonably foreseeable use," "reasonably foreseeable user" and "reasonably foreseeable use environment." The important distinction is that what is "intended" is never, ever enough: one must also consider things that are "unintended." The phrase "reasonably foreseeable" includes both, and is, in fact, the standard that U.S. tort law applies in cases of product liability. It should therefore be the minimum standard that should be used by product designers.

Another very simple example is the common screwdriver. Intended use: turn screws. Intended use environment: home and shop. Intended user: Joe Sixpack. Simple, right? Now consider what is "reasonably foreseeable." Reasonably foreseeable uses might include: turn screws, pry open paint cans, chisel holes in wood, remove dried spackling compound, chisel off rusted bolts.... Reasonably foreseeable use environments: home and shop, chemistry lab, salt-water pier, high tension electrical tower, off-shore oil rig.... Reasonably foreseeable user: Joe Sixpack, his neighbor Martha with arthritis in her hands, her 10-year-old grandson Johnny with small hands, his sister Sue who is left handed, their mother Jan who works in that chemical lab with all sorts of corrosive materials, her husband Jim who works on those high tension electrical towers wearing heavily insulated gloves, Jim's brother Dave who works on that off-shore oil rig and who's hands are always covered with oil....

Get the idea? If you only think in terms of what is "intended," you might not think hard enough about all the ways your product might be used. If so, your product might actually be hazardous to use or, at the very least, you might miss an important market segment that your competitor will be very happy to fill at your expense -- and, by doing so, reduce your market share, eat your lunch, and take away the income you were planning to save for your kid's college education or your early retirement.

And that, as I see it, is the important difference between verification and validation.

Doug Pfrang

[Andy Bassett]

I distilled from the thread the following summaries posted by various helpful souls;

Number 1
In my own words, I would say that verification is like "inspect as per drawing" -- comparing the finished part/product to the original specifications (does it have 4 wheels as specified?). Whereas validation is "does it do what the intended use is" (does it roll down the road?).

I could agree with that, but then i no longer understand the difference between a development output (which i thought was the checking of a drawing) and design verification.

Number 2
Verification - has it been built (or provided) per spec
Validation - does it work (or accomplish the intended result)

I think this is saying more or less the same as above, (check the drawing and then physically test the product) so i am left with the same problem of understanding the difference between design output and verification.

Number 3
Design Verification - does the output information match the input requirements (check it).
Design Validation - does the product and/or service perform to the requirements (try it).

And that is the one i will use. ie
I will encourage my customer to go back and compare the product with the original specification (design input) ie Weight limit reached, Torque reached, Target price within 10% etc etc This i hope will satisfy Design Verification.

For Design validation i will present the actual physcial Test Reports relating to the physical testing of the product.

Any comments, what do you think, is this realistic?

BTW Having spent a couple of hours agonising over the meaning of ISO 9000:2000 in this area, i firmly beleive that the Standard deserves every criticism that it gets.

Regards


------------------
Andy B

[This message has been edited by Andy Bassett (edited 03 August 2001).]

Andy, I completely understand your confusion at this greyest of grey areas of the standard. My understanding is as follows:

Verification - Check in put meets output, in my mind the output of design and development reviews

Validation - Check design is capable of doing the job for which it is intended - trials etc

Hope this helps but I to am stuggling through this area as we speak.

[Marc]

The two words - Design and Development - are to some degree synonymous. Ehat I mean is we're reaching into the world of symantics. Many companies call their design function just that - design. However, we think of a 'Design and Development' process in which Design is a part of the larger system.

-> For Design validation i will present the actual physcial
-> Test Reports relating to the physical testing of the
-> product.

Every company and industry is different. What does an equipment manufacturer do as 'validation' as opposed to an electric motor manufacturer?

For an equipment manufacturer validation is typically performed at the customer's facility. Sometimes it is done at a 'turn-key' company. And yes - some do test their systems prior to shipping, but seldom is reliability an issue in such a 'validation'.

On the other hand, if you're manufacturing electric motors and you make and sell 1M a year, you probably have a number of accellerated life tests you do and other validations.

My point is to look at your design (and Development) process (read system) and determine where the appropriate 'key' points are and how you define each within your system / company.

As per your statement you have done that. I'd want some more info on verification ("...please explain some more..." an auditor may ask). I think you have validation defined OK.