My third and final post of the day stemming from my pre-assessment this week was regarding Design and Development.

The auditor indicated that I had to have proof of verification and validation. The way I explained it to him is:

1. Our verification process is by performing a first article test on prototypes proving that the outputs do meet the inputs of the design.

2. Our validation is doing production functional testing to validate that all deliverables meet customer specifications/ inputs.

He mentioned that I was lacking on the validation part. He said the validation should be how the customer uses the product and that we should gather customer feedback indicating the part conforms to their expectations.

I don't get it? The customer inputs are verified (i.e. customer requirements) which is what the customer wants (by prototype testing). The validation is that the inputs are met by functional test for all deliverable goods (part meets customer required specs).

Why is the feedback needed for validation? I understand that customer satisfaction and customer feedback is a requirement and my company is performing that. How does that fall under Design and Development validation?

Again, I am confused... what am I not understanding?

Thanks again folks!

Hello! We're glad you dropped by!:bigwave:

If you haven't looked, there is an existing thread where this very subject has been discussed. If you are so inclined, you might want to start there and read it. I would pay particular attention to the definitions provided by Randy in that thread.

verification vs. validation (http://elsmar.com/Forums/showthread.php?t=27737&highlight=validation)

See if that helps.:)

I may be wrong on this, in fact I would love to hear more on the subject. I think your FAI can benefit from some of your V&V activities. Not having gone through either one, but having studied the requirements of both I think, in part, some of the FAI requirements could be met if you can produce your V&V records.

Thoughts?

Thank you all for your feedback.

I think I have a better idea on how to address this. The product is our off the shelf design where custom changes can be made suitable to the customer's inputs.

When it comes to validation, in this case, I would be gathering customer feedback that the product does indeed work to their expectations.

For some background, my company takes the inputs from a potential customer and actually performs the Engineering work for them, meaning sometimes the customer can't put or doesn't understand the design themselves.

To close the loop I am thinking that the inputs is the spec sheet or parameters a customer wants, the verification is our internal R&D results (i.e. FAI), and the validation is gathering customer feedback on how it is working for them.

The validation is not a measurable item, meaning either it works or doesn't in the customer's eyes, but it can be measured via survey or feedback.

JkelleyCDS,

Using post market surveillance data might constitute a retrospective validation. This is not generally the preferred approach, but it can be effective in many industries. Aerospace, Medical Device and Pharma companies might end up in large lawsuits if they did, but widget makers might be alright in using this approach.

Your auditor might be concerned that your approach might be too late and less effective (and efficient). Waiting to hear that your customer would like you to change your widget to red rather than blue might mean you have a warehouse of slow moving/no moving products. Also, validation is from the customer's perspective. My feeling here is that you might confirm that you are meeting an intended use or requirement of the customer, but you don't really know unless you are the customer (and only a small part of the greater population).

Read the link that Brad provided. Lot's of discussion there regarding what the differences are. But to a degree, your argument should have been fairly successful. The auditor might have been looking for prospective data.

Regards,

Kevin

I thought this debate was over ....just looked in and found it is still kicking.
My thoughts are that you are both wrong.
Both verification and validation are activities to ensure that the design outputs meet design inputs. The main diferrence is that verification focuses on the primary or key features and is performed earlier in the development cycle, thus could use models, engineering analysis, prototypes and so on while validation completes the verification by carrying out tests on first article(s) representing not only the design but also the manufacturing process and checking that the spec limits are met (or changing the specs limits accordingly). Say for a car that is designed among many other things to have a top speed of 138 miles/hr, this is a design input that is verified to some extent in the framework of verification but cannot completely be validated until a few actual production items are ready and road tested under controlled conditions (highway, city driving, etc)
The involving of customers requirements in validation in my opinion is a misunderstanding. Whenever there are clear customer requirements or even just expectations , these are likely already in the design inputs. Nobody in his right mind is going to design a product that does not take into account customer's requirements when these are known. Of course there are cases of products where the manufacturer can only guess (rightly or wrongly) what the market would like. But the organization should still validate the perceived requirements before placing the product on the market. Otherwise , after the fact how can the organization know in case of negative market feedback wether these are due to having guessed wrongly what the market wants or perhaps the guess was correct but the product was never fully validated to ascertain that it really performs as intended?
Customer satisfaction is on another plane and has nothing to do with the design and development cycle.

I thought this debate was over ....just looked in and found it is still kicking.

Willy,

This is one of of the Cove's hot topics that will always spur debate:argue:, and will never go away, :D no matter what.:yes:

Stijloor.

Willy,

Please keep in mind that newcomers often ask repeat questions. This is to be expected, so in essence, the debate is never over.

If the customer requirement is for a car to go 138 mph, then the manufacturer will have to develop a test to validate that its new design meets the need. Validation is done from the customer’s perspective as it’s done to ensure that the product meets user needs and intended uses.

Regards,

Kevin

My third and final post of the day stemming from my pre-assessment this week was regarding Design and Development.

The auditor indicated that I had to have proof of verification and validation. The way I explained it to him is:

1. Our verification process is by performing a first article test on prototypes proving that the outputs do meet the inputs of the design.

2. Our validation is doing production functional testing to validate that all deliverables meet customer specifications/ inputs.

He mentioned that I was lacking on the validation part. He said the validation should be how the customer uses the product and that we should gather customer feedback indicating the part conforms to their expectations.

I don't get it? The customer inputs are verified (i.e. customer requirements) which is what the customer wants (by prototype testing). The validation is that the inputs are met by functional test for all deliverable goods (part meets customer required specs).

Why is the feedback needed for validation? I understand that customer satisfaction and customer feedback is a requirement and my company is performing that. How does that fall under Design and Development validation?

Again, I am confused... what am I not understanding?

Thanks again folks!

I don't see enough here to call this a nonconformance. Perhaps there is more to it.

I does sound like the auditor is basing this on what he would like to see or what he usually sees instead of the requirements of the standard.

Lets look at the definitions from ISO 9000:2005. Please note my added emphasis to validation. Also note that NOTE 1 in both do not seem pertinent so I left them out.

verification
conformation, through the provision of objective evidence that specified requirements have been fulfilled.
NOTE 2 Confirmation can comprise activities such as
-- performing alternative calculations,
-- comparing new design specifications with a similar proven design specification,
-- undertaking test and demonstrations, and
-- reviewing documents prior to issue.

validation
confirmation, through the provision of objective evidence, that the requirements for a specific intended use or application have been fulfilled
NOTE 2 The conditions for validation can be real or simulated.

Now lets look at ISO 9001:2000. Please note my emphasis in validation.

7.3.5 Design and development verification

Verification shall be performed in accordance with planned arrangements (see 7.3.1) to ensure that the design and development outputs have met the design and development input requirements. Records of the results of the verification and any necessary actions shall be maintained (see 4.2.4)

7.3.6 Design and development validation

Design and development validation shall be performed in accordance with planned arrangements (see 7.3.1) to ensure that the resulting product is capable of meeting the requirements for the specified application or intended use, where known. Whenever practicable, validation shall be completed prior to the delivery or implementation of the product. Records of the results of validation and any necessary actions shall be maintained (see 4.2.4).

Now lets look at one more from ISO 9001:2000.

4.1c The organization shall determine criteria and methods needed to ensure that both the operation and control of these processes are effective.

Now for my comments.

The main role of validation is to ensure that the resulting product is suitable for the intended use. Have you done what you can to make that determination? Is there any evidence that it is not effective?

Answer those questions for yourself and see where you stand.

Kevin,Stijloor,

I agree 100%.

Big Jim, I don't think you could be clearer than that.
Thx

Big Jim is probably right. I think that all of us at one time or another had an auditor who probably relied too heavily on fulfillment of a personal paradigm rather than looking for the objective evidence in the format presented. Maybe there is more to the story, but as presented, I agree that it sniffs more of an observation than a finding.

Back to the group...

Kevin

The main role of validation is to ensure that the resulting product is suitable for the intended use. Have you done what you can to make that determination? Is there any evidence that it is not effective?

Answer those questions for yourself and see where you stand.

I like Big Jim's explanation, but perhaps it can be made simpler. Just because a product meets design specifications doesn't mean that it meets design intent.

When I was a teenager I worked at McDonald's, in a company-owned store. AMF had developed an automatic fry-bagging machine, and brought it into our store for a little tryout. While I'm sure that all of the parts of the machine met their individual specifications (confirmed through verification), and the machine itself performed as intended (for the most part--it had some bugs), it didn't work in the application because it was too slow to keep up with demand and required too much attention. It wound up on a junk heap somewhere, and today McDonald's handles French fries pretty much the same way they did forty years ago.

Look again at Big Jim's quote of the ISO definition of validation: "...requirements for a specific intended use or application have been fulfilled." The fry-bagging machine worked as designed, but the design didn't meet the requirements of the application.

Just because a product meets design specifications doesn't mean that it meets design intent.


Precisely. I like both the simplification and the nifty example to illustrate!

Jim,

It could also be that AMF did not properly establish the requirements. This is common when an entity tries to innovate but lacks the intimacy of what the customer really needs (a fast bag machine rather than a bag machine). Further customer testing and extended research might have been in order. In the end, it could be that to make it faster and more practical was too expensive and more economical to use human labor.

Regards,

Kevin

Jim,

It could also be that AMF did not properly establish the requirements. This is common when an entity tries to innovate but lacks the intimacy of what the customer really needs (a fast bag machine rather than a bag machine). Further customer testing and extended research might have been in order. In the end, it could be that to make it faster and more practical was too expensive and more economical to use human labor.

Regards,

Kevin

I think that even in 1968 AMF was probably aware that McDonald's (especially in one of their busiest stores) sold a lot of French fries, and prided themselves on quick service.

Dear all,
I think with these simple examples you are blurring the general picture.
Meet design specification but not meet design "intent"? Hmmm.. To me this only means that the design inputs (specs) weren't comprehensive enough to begin with.
In high-tech products there is usually no well known specific customer. There is a marketing department that tries to figure out what the market might want in 12-24 months and comes up with an MRD-Marketing Requirements Document which in turn is turned into a specification.
The MRD may wrongly guess what the market wants, or the way the market is going to go or how fast, so you may have perfect design verification and validation and yet the product turn out to be a flop.
I worked for a company that had a great patent to triple coax cable bamdwith (capacity) on the existing infrastructure ten years ago. A real technological breakthrough, it was verified validated and met design specifications, design intent , everything. The cable operators looked at it and said "great" but right now we have all the capacity we need....Ten years later , they were all suddenly interested
with the triple play video, telephony and data choking their capacity....but my company burnt their last $ just a few months before...and went belly up.
So success in the market place/ customer acceptance is not the way to validate a product . As I said before, in many markets (automotive, military,enterprise softwsre, etc) customers won't even touch a product that hasn't been verified and validated before
So with all due respect to the AMF, I would have asked him if he would board a plane that was partially validated by the manufacturer and the validation is now being completed by actual use or would he bank with a bank that has just installed this great account management software that was verified and partially validated by the developer and its validation is still ongoing through actual use....his paycheck may end up in someone else's account but the softwre is being validated just as he wanted..

Dear all,
I think with these simple examples you are blurring the general picture.
Meet design specification but not meet design "intent"? Hmmm.. To me this only means that the design inputs (specs) weren't comprehensive enough to begin with.

And that, I believe, is one of the main reasons for validation. It is an opportunity for course correction. If the design team got off course for any of several reasons this is an opportunity for a "reality check".

Your other comments are spot on.

It is my personal belief that inadequate validation is the main cause of recalls.

And that, I believe, is one of the main reasons for validation. It is an opportunity for course correction. If the design team got off course for any of several reasons this is an opportunity for a "reality check".

Your other comments are spot on.

It is my personal belief that inadequate validation is the main cause of recalls.

I guess this is my point when it comes to our customer base. They rely on our company to Engineer an item based on their performance criteria and lean on us to do the design work.

The products we develop based on customer specs always get great feedback.

I guess when it comes to validation, customer feedback may be my only way to validate the item meets their intent.

For example, if we make a head mounted display for a simulation (video) application and meet all the specifications, where would the validation be? The unit is tested for all parameters as the intent/specifications require.

I guess I understand the definition and why validation is important (believe me), but wonder in a situation like this, how do I validate?

I know that customer feedback and satisfaction are other requirements of the ISO standard. but can it be used for the validation?

I just don't know any other way. It may just be that it is not black and white in our industry. Again, does this button work, do all the colors meet, is it bright enough? All these are measured at my company's facility based on a customer's specification. The intent is that the unit turn on and you can adjust the brightness.

It's not like it needs to meet a shake (vibration) or environmental test. I could see how that would need validation.

I guess I'm still a little confused on how it applies to our industry. I appreciate everyone's feedback by the way. It makes sense, but like I mentioned before, does it makes sense for my company's technology?

Meet design specification but not meet design "intent"? Hmmm.. To me this only means that the design inputs (specs) weren't comprehensive enough to begin with.

Yes, just so. Hence, no doubt, the inclusion of validation (not just verification) in the Standard, and its importance. Was it actually the 'right' design? Did we get all the specs right, or miss some? etc. The majority of things in software go wrong at the specs stage.

I just don't know any other way. It may just be that it is not black and white in our industry. ...

I guess I'm still a little confused on how it applies to our industry. I appreciate everyone's feedback by the way. It makes sense, but like I mentioned before, does it makes sense for my company's technology

And that, of course, is the big question. How does this work in my company or organisation? How can we (can we?) validate? You may well be right that it isn't B&W - does it perhaps vary for different products? Jobs?

(Sorry, I'm not trying to be unhelpful at all!) There is, of course, that 'as appropriate' in the relevant clause (7.3.6) because it can and does vary so much.

I guess when it comes to validation, customer feedback may be my only way to validate the item meets their intent.

For example, if we make a head mounted display for a simulation (video) application and meet all the specifications, where would the validation be? The unit is tested for all parameters as the intent/specifications require.


I guess I understand the definition and why validation is important (believe me), but wonder in a situation like this, how do I validate?

I know that customer feedback and satisfaction are other requirements of the ISO standard. but can it be used for the validation?

In software, broadly speaking of course, testing that the product meets all performance requirements would fall under verification.
Validation would be all tests whose purpose would be to test that the application does NOT DO what it is not expected to do: e.g . doesn't crash/stall, doesn't time out, etc. due to an illegal command or selection sequence by the user, due to overheating, scalability /stress tests under multiple users...etc. In some situations, customers may use your product under different OS, hardware or in interaction with other existing applications. Even the latter could be simulated to a degree in your own labs but if not or the parts that cannot, must be validated by customers.
In fact , Beta versions , given to selected customers to play around with and find bugs that may have sneaked through all previous test is exactly validation.
This has nothing to do with customer satisfaction, customers who receive beta versions are fully aware that they may contain bugs, glitches and so on.
Hope this helps

Big Jim,
It is a reality check albeit a very late one. I don't think the intent of ISO is to put on the market "half-baked" or partially tested products and push the responsibility to validate on the customers.
If the design team got off course, the oportunity to catch this is DESIGN REVIEWS! Marketing, Production, Quality plus project management as well as all design disciplines need to be active participants in design reviews, so chances of the design going off course not being noticed until validation are very slim in a well run company.
I think that the main misunderstanding in this thread is that there are simple products for which there is no real distinction between verification and validation, and a simple inspection is all that is needed. On the other hand for complex /critical products validation is a critical phase , it is definitekly the developer's responsibility even when he must use customer facilities/environments to do it. For the CATV products I mentioned in my earlier post, my company had labs with miles of cables and simulators for hundreds of subscribers, routers, switchers and so on to simulate as closely as possible the real world environment.
Only when all this was done succesfully, did the prospective users (CATV operators) do their own tests in a real neighborhood with thousands of subscribers using the real infrastructure.
An aeroplane on the other hand is put into service by the airliner the next day it lands at its facility. Let's all hope that Boeing and Airbus do a thorough validation and don't rely on customers to tell them their design went off course:)
Luckilly ,the FAA wouldn't issue them and Airworthiness Certification if they operated IAW that auditor's advice:)

Willy,

I'm not the one that suggested putting off validation on the customer. I believe that the standard requires the organization to do what it can. I agree that sometimes it is limited, but I also know that it is often nearly ignored.

One of the most famous cases of validation performed too late is the instance of Rolls Royce developing jet engines for the Lockheed L1011. They didn't perform the chicken test to confirm that the engine could withstand bird injestion until they were nearly done with the project. It failed the test, and bankrupt Rolls Royce. The British government had to step in to rescue them as they were a prime military contractor.

A large number of posts imply that validation is something done at the end. Validation, like verification needs to be done throughout the development lifecycle.
Everytime a person answers the question "does this meet the intended use?" they are performing validation. (note: satisfying the intended use may not satisfy the customer - this is why there is a customer satisfaction requirement). For example: requirements are verified to be accurate, properly written, useful in the next phase, etc. Requirements are also validatated that they fulfill the intended use and often the customer need.
The two (V&V) play together throughout the whole lifecycle.

my :2cents:

danpa,

You raise an interesting observation and you are correct that this is an iterative process especially in the case of customer preference testing. Typically speaking though, many organizations label early developmental verification and validation as a type of engineering test, meaning that while V & V activities in execution, they are mainly informal as the organization dials in on a final design. Once the design is roughly frozen, formal design V & V activities are generally executed. Of course there are no firm rules on how an organization establishes their design models or what they consider informal or formal testing. In the end, they must establish for outside examination what their plans and records are so that established input may be verified in ouputs.

It was a good two cents to throw into this discussion!

Regards,

Kevin

hi ,

i am doing ISO 9001:2000 implementation for a construction company mainly dealt with soil investigation & Piling works. I just wanna to know weather it is necessary to do the validation process for the soil investigation works or i can exclude the clause 7.3.6. pls any one clarrify.

hi ,

i am doing ISO 9001:2000 implementation for a construction company mainly dealt with soil investigation & Piling works. I just wanna to know weather it is necessary to do the validation process for the soil investigation works or i can exclude the clause 7.3.6. pls any one clarrify.

Hello Rajes,

Welcome to The Cove Forums! :bigwave: :bigwave:

Does your Client actually engage in design and development activities?

Design of soil investigation services/activities?

Design of piling systems (http://en.wikipedia.org/wiki/Deep_foundation)/materials?

Can you clarify?

In my opinion, verification (evaluation of design documentation) and validation (evaluation of the actual product/service) can not be separated.

Stijloor.

Actually they r not involved in any design and development in both soil investigation & piling works.

Actually they are not involved in any design and development in both soil investigation & piling works.

If this is indeed the case, then Design and Development can be excluded from the scope of certificaton. This exclusion must be justified and included in the Quality Manual. You (or your Client) write a paragraph in the Quality Manual stating why D&D does not apply. The Certification Body will verify if the exclusion is justified.

Stijloor.

thanks stijloor