Process to Validate the Accuracy of Test Reports (AS9100)

[alspread]

AS9100 has re-instated the requirement to "implement a process to validate the accuracy of test reports" for material identified as a significant operational risk. AS9100:B had a similar requirement, but included the term "periodically" and most places set up some kind of independent test on some kind of frequency.
I am looking for any thoughts, insight, ideas that are being used out there for an effective way to, "validate the accuracy of test reports" for operational risky raw material.

Thoughts???

Thanks

 

[Ninja]

Re: Validate the accuracy of test reports

Is that not simply incoming QA that you already do for critical parameters?

If a vendor tells me on the test report that this powder has SSA=7.8, and I measure and get 7.6, I have "validated the accuracy of the test report", no?

Perspective: BET SSA measurement error is in the neighborhood of +/- 0.5 at that range.

Sound like the standard is simply requiring you to do normal due diligence and call it "validating a test report"

 

[alspread]

Re: Validate the accuracy of test reports

Thanks for the feedback. I guess I should have been clearer about the kind of material.
The first part of the clause requires evaluation of the data in the test reports (check the certificate information/details against the specification requirement). the second part of the clause requires validation of the accuracy of said test report.

In my case, raw material is bar stock, plate stock, aluminum, titanium, nickel alloys, etc.

 

[Ninja]

Re: Validate the accuracy of test reports

So you got a bar of nickel alloy and a test report.

What properties are "critical" or "significant operational risk"...and what did you do in incoming QA last month when the bar showed up to make sure it was what you needed?

My guess (hope) is that you are already complying with the requirement and have been for some time.

 

[Mike S.]

Here's my take:

When the supplier sends you their raw material test report, compare it to the spec’s for the material and see if the actual results fall within the spec windows. You have then evaluated the data.

If you periodically get a 3rd party to test the raw material and have them send you their test report, you could compare their data to both the spec and the 2nd party supplier’s data, and thus (hopefully) "confirm that the product meets requirements" a second time and "validate the accuracy of test reports" from the supplier.

 

[Sidney Vianna]

I am looking for any thoughts, insight, ideas that are being used out there for an effective way to, "validate the accuracy of test reports" for operational risky raw material.

Thoughts???

We don't need to go too far to see the reason for this requirement. The sensible way to validate the accuracy of test reports is to periodically have a sample tested by another reliable lab and compare the results.

How often to perform such validation? Taking into account risk, it would depend on the level of assurance you have from the supplier and how safety-critical the material is.

Let's remember that Boeing Commercial Airplanes had reinstated the requirement in their X31764 document, after the requirement was dropped in AS9100 Rev. C.

It reads:

Validation of Raw Material Test Reports:

When Seller utilizes test reports to accept Seller purchased raw material, the following requirements apply:

Test reports shall be checked 100% against Seller's requirements and applicable specifications.

Validation test requirement: Seller shall periodically validate test reports for raw material accepted on the basis of test reports. That validation shall be accomplished by Seller or other independent party through periodic, scheduled tests of raw material samples. Schedules for frequency of tests will be established by Seller based on historical performance of the raw material supplier.

Seller shall retain test reports provided by the raw material supplier, as well as Seller's validation test results as quality records traceable to the conformance of Goods, as specified elsewhere in this Contract.

 

[BoardGuy]

Here is the simple way to address this requirement without the expense of testing every piece of raw material received.

1) Purchase order must specific the specification to which the material must conform (with the revision listed) and that test reports must contain actual test results.
2) Receiving Inspector compares material certification actual results to specification requirements for the material. Inspector places check mark next to each conforming reading.
3) At completion of review receiving Inspector stamps and dates material test reports.

Now the check marks indicate that a specific item conforms to the specification requirement and the inspector stamp and date indicates who performed the inspection and accepted the material.

The second part of the process is then to select randomly (we use first receipt of material each year) a sample of the raw material to be independently tested to assure accuracy of received test reports from that supplier. We explain to the customer and auditors that we are verifying the truthfulness of the supplier certification.

 

[Sidney Vianna]

(we use first receipt of material each year)

That might not pass the risk based thinking litmus test...

From a risk based thinking perspective, one should use historical performance of the supplier, as well as the criticality of the product and the volume of material received.

 

[BoardGuy]

That might not pass the risk based thinking litmus test...

Although 6.1 (RBT) specifies that the organization shall plan action to address risk there is:

No requirement for formal methods for risk management
No requirement to document risk management process
No requirement to retain documented information as evidence of determination of risks.
[FONT=&quot]
So per Clause 6.1 and [/FONT][FONT=&quot]Annex A.4 what would be the litmus test? The requirement is that “[/FONT][FONT=&quot]The organization shall implement a process to validate the accuracy of test reports”.[/FONT]

 

[Sidney Vianna]

5.1.1 d) requires that top management promote the use of risk based thinking. Risk based thinking has to permeate throughout the quality business processes, in order to make sense. Remember that AS9100D also has requirements for product safety. This would be a classic example of gambling with product safety if the test report validations are far and few between.

But, if you forget the standard for a second, the BCA requirement, which I transposed talks about using supplier performance to establish the frequency of independent checks. That's a risk-based input. In my mind, I think, we should go beyond that and also use material criticality (if known) and volume of orders as well, in order to mitigate risks of accepting nonconforming products by solely relying on supplier's test reports.

If I were in the process of qualifying a new raw material supplier, I would also ensure that the independent validation of the supplier reports was done at a higher frequency than another supplier I have history of working together and, more importantly, confidence in.