Lessons Learned - from ANAB 2013/06/06

dsanabria

Quite Involved in Discussions
Issue: 262

Date: 2013/06/06

To: ANAB-AQMS Accredited and Applicant CBs and Accreditation Assessors
From: Bob Cruse, Aerospace Program Manager

Re: AQMS Lessons Learned
Through feedback from AAQG industry oversight, RMC meetings, and results of ANAB witnessed AQMS audits, two areas of concern emerged:

Issue 1: Use of copy-and-paste and audit reporting integrity While technology provides tools to increase productivity and efficiency, the use of these tools must not compromise the integrity of the audit reporting.

For an audit report to be deemed viable and in conformance with the requirements referred to below, the objective evidence that is ?collected, verified, and recorded? must be unique to each audit conducted and
unique to subsequent audits conducted at the same organization.

The expectation is that audit reporting will ?provide an accurate, concise and clear record of the audit to enable an informed certification decision
to be made? and should allow recreation of the audit trail, if necessary.

There has been a noticeable increase in the use of copy-and-paste in reports reviewed by ANAB accreditation assessors and other party (OP) assessors.

Auditors sometimes copy information verbatim from one audit to the next and/or from one client?s report to the next. Although the practice can increase efficiency and may be appropriate in some cases, the potential for abuse compromises the integrity of audit reporting.

When it reaches the point that requirements are no longer being met, the validity of the audit is in question and the report could potentially be considered falsification of information.

If this is found during CB AQMS audit report reviews, a major nonconformance will be issued and depending on the severity, suspension may be recommended.

CB personnel (e.g., AS technical managers and reviewers, administrative reviewers) need to be aware of this concern and should analyze the potential risks to the client?s certification and take action to ensure the integrity of the audit activity is not compromised.

References:
ISO/IEC 17021:2011, 9.1.9.5.1: During the audit, information relevant to the audit objectives, scope and criteria (including information relating to interfaces between functions, activities and processes) shall be collected by appropriate sampling and verified to become audit evidence.

ISO/IEC 17021:2011, 9.1.10.2: The audit team leader shall ensure that the audit report is prepared and shall be responsible for its content. The audit report shall provide an accurate, concise and clear record of the audit to enable an informed certification decision to be made and shall include or refer to the following:

i) audit findings, evidence and conclusions, consistent with the requirements of the type of audit; AS9104-0011, 8.1.5: The audit team leader shall be responsible for ensuring the completeness of the audit and the accuracy of the audit report, findings, and conclusions.

AS9101D, 4.2.2.5: The audit team shall record detailed objective evidence (e.g., reviewed procedures, shop orders, training records, products, verification records). The objective evidence shall be recorded on a standardized form [i.e., OER (see Appendix A)] or on the CB?s own
documentation. In this case, the CB document shall meet the intent of the OER. The completed form(s) shall be included in the audit records maintained by the CB.




Issue 2: Expected outcomes of special and/or short notice audits
To ensure effective, consistent, and objective deployment of a special or short notice audit process, the intent of the following requirements should be fully understood: AS9104-001, 6.7.j; ISO/IEC 17021:2011, 9.5.2 Short-notice audits; and AS9101D, 4.3.6 Special Audits.

The objectives of a special or short notice audit in relation to a transfer or scope extension are relatively straightforward.

However, when such audits are necessary ?in response to a customer or other interested party request or when a serious issue (supported by objective evidence) has been identified? (AS9101D, 4.3.6 a), the audit must be conducted as soon as possible, and objectives should include the corrective actions already taken by the organization and also focus on the processes that led to the issue(s) being raised. The following questions should be considered during the planning stage of the special or short notice audit:

1. Which processes, as defined by the organization per ISO 9001:2008, 4.1, were likely not to have achieved the organization?s ?planned results? and, therefore, should have prompted corrective action from the organization to correct poor process performance issues?

2. Does the organization?s communication process ensure that top management is made aware of poor process performance?

Is the process robust enough to ensure timely visibility and notification so action can be taken to mitigate any risk to the customer (e.g., additional resources, process improvements, equipment needs)? (AS9100C 5.6, 7.1.2, 8.2.2, 8.2.4, 8.5.3, 8.5.4) When a ?serious or major conformance issue? has been raised by a stakeholder (e.g., OEM, regulatory agency) and a special short notice audit is warranted, the organization is responsible for corrective action containment to bring the issue under control to the satisfaction of the stakeholder.

In most cases, the stakeholder has already documented the issue and the client has containment well under way prior to the CB?s special or short notice audit.

The expected outcome of the audit is an AS9101D audit report documenting the investigation and audit of associated processes to ensure conformance to requirements (AS9100:2009, 4.1, in particular).

During the closing meeting, the CB lead auditor is required to present the audit conclusions, including the recommendation regarding certification (ISO/IEC 17021, 9.1.9.8.1).

The CB must take appropriate action based on the results of the special or short notice audit.

CBs may want to consider the use of the AS9101D, Appendix C (Process Effectiveness Assessment Report), to better understand the contributing processes (e.g., management review, customer complaints) that led to the special or short notice audit, regardless of whether the processes are specific to clause 7 (Product Realization).

This would also assist in understanding how the organization determines whether processes are functioning effectively and meeting ?planned results.? CBs are reminded that ANAB Heads Ups are public documents and they are allowed to share the information with their auditors.
 

dsanabria

Quite Involved in Discussions
Issue: 262

Date: 2013/06/06

To: ANAB-AQMS Accredited and Applicant CBs and Accreditation Assessors
From: Bob Cruse, Aerospace Program Manager

Re: AQMS Lessons Learned
Through feedback from AAQG industry oversight, RMC meetings, and results of ANAB witnessed AQMS audits, two areas of concern emerged:

Issue 1: Use of copy-and-paste and audit reporting integrity While technology provides tools to increase productivity and efficiency, the use of these tools must not compromise the integrity of the audit reporting.

For an audit report to be deemed viable and in conformance with the requirements referred to below, the objective evidence that is ?collected, verified, and recorded? must be unique to each audit conducted and
unique to subsequent audits conducted at the same organization.

The expectation is that audit reporting will ?provide an accurate, concise and clear record of the audit to enable an informed certification decision
to be made? and should allow recreation of the audit trail, if necessary.

There has been a noticeable increase in the use of copy-and-paste in reports reviewed by ANAB accreditation assessors and other party (OP) assessors.

Auditors sometimes copy information verbatim from one audit to the next and/or from one client?s report to the next. Although the practice can increase efficiency and may be appropriate in some cases, the potential for abuse compromises the integrity of audit reporting.

When it reaches the point that requirements are no longer being met, the validity of the audit is in question and the report could potentially be considered falsification of information.

If this is found during CB AQMS audit report reviews, a major nonconformance will be issued and depending on the severity, suspension may be recommended.

CB personnel (e.g., AS technical managers and reviewers, administrative reviewers) need to be aware of this concern and should analyze the potential risks to the client?s certification and take action to ensure the integrity of the audit activity is not compromised.

References:
ISO/IEC 17021:2011, 9.1.9.5.1: During the audit, information relevant to the audit objectives, scope and criteria (including information relating to interfaces between functions, activities and processes) shall be collected by appropriate sampling and verified to become audit evidence.

ISO/IEC 17021:2011, 9.1.10.2: The audit team leader shall ensure that the audit report is prepared and shall be responsible for its content. The audit report shall provide an accurate, concise and clear record of the audit to enable an informed certification decision to be made and shall include or refer to the following:

i) audit findings, evidence and conclusions, consistent with the requirements of the type of audit; AS9104-0011, 8.1.5: The audit team leader shall be responsible for ensuring the completeness of the audit and the accuracy of the audit report, findings, and conclusions.

AS9101D, 4.2.2.5: The audit team shall record detailed objective evidence (e.g., reviewed procedures, shop orders, training records, products, verification records). The objective evidence shall be recorded on a standardized form [i.e., OER (see Appendix A)] or on the CB?s own
documentation. In this case, the CB document shall meet the intent of the OER. The completed form(s) shall be included in the audit records maintained by the CB.




Issue 2: Expected outcomes of special and/or short notice audits
To ensure effective, consistent, and objective deployment of a special or short notice audit process, the intent of the following requirements should be fully understood: AS9104-001, 6.7.j; ISO/IEC 17021:2011, 9.5.2 Short-notice audits; and AS9101D, 4.3.6 Special Audits.

The objectives of a special or short notice audit in relation to a transfer or scope extension are relatively straightforward.

However, when such audits are necessary ?in response to a customer or other interested party request or when a serious issue (supported by objective evidence) has been identified? (AS9101D, 4.3.6 a), the audit must be conducted as soon as possible, and objectives should include the corrective actions already taken by the organization and also focus on the processes that led to the issue(s) being raised. The following questions should be considered during the planning stage of the special or short notice audit:

1. Which processes, as defined by the organization per ISO 9001:2008, 4.1, were likely not to have achieved the organization?s ?planned results? and, therefore, should have prompted corrective action from the organization to correct poor process performance issues?

2. Does the organization?s communication process ensure that top management is made aware of poor process performance?

Is the process robust enough to ensure timely visibility and notification so action can be taken to mitigate any risk to the customer (e.g., additional resources, process improvements, equipment needs)? (AS9100C 5.6, 7.1.2, 8.2.2, 8.2.4, 8.5.3, 8.5.4) When a ?serious or major conformance issue? has been raised by a stakeholder (e.g., OEM, regulatory agency) and a special short notice audit is warranted, the organization is responsible for corrective action containment to bring the issue under control to the satisfaction of the stakeholder.

In most cases, the stakeholder has already documented the issue and the client has containment well under way prior to the CB?s special or short notice audit.

The expected outcome of the audit is an AS9101D audit report documenting the investigation and audit of associated processes to ensure conformance to requirements (AS9100:2009, 4.1, in particular).

During the closing meeting, the CB lead auditor is required to present the audit conclusions, including the recommendation regarding certification (ISO/IEC 17021, 9.1.9.8.1).

The CB must take appropriate action based on the results of the special or short notice audit.

CBs may want to consider the use of the AS9101D, Appendix C (Process Effectiveness Assessment Report), to better understand the contributing processes (e.g., management review, customer complaints) that led to the special or short notice audit, regardless of whether the processes are specific to clause 7 (Product Realization).

This would also assist in understanding how the organization determines whether processes are functioning effectively and meeting ?planned results.? CBs are reminded that ANAB Heads Ups are public documents and they are allowed to share the information with their auditors.


Just had my audits and some auditors are going "Old School" - hand written report - they don't want to be accused of "Copy / Paste" syndrome.:notme:
 
Top Bottom