The Civil Aviation Safety Authority did not have a defined process for a robust, systematic approach to the assessment and approval of alternative non-destructive inspection procedures to ensure that the proposed method provided an equivalent, or better, level of safety than the original procedure.
The ATSB acknowledges that CASA has made a number of improvements to their procedures and educational material; however, the ATSB does not feel that these changes have resulted in a completely robust systematic approach to the review and approval of NDT procedures. At this point in time, it is clear that CASA do not intend to take any further changes to their procedures, so the ATSB is closing the safety issue as partially addressed, but encourages CASA to continue to review their systems and incorporate a systematic engineering approach.
On 7 October 2015, the Civil Aviation Safety Authority (CASA) advised the ATSB that:
CASA has considered the issue raised and is in the process of developing a policy and/or supporting procedure to address this. The procedure will give recognition to the relationship between the consensus standard for non-destructive inspection (NDI) AS3669 and reg. 2A of the Civil Aviation Regulations 1988, ensuring a minimum equivalent level of safety for alternate NDI procedures and approved maintenance data.
The ATSB has assessed the proposed safety action by CASA and considers that it should adequately address the identified safety issue. However, the ATSB will reconsider this assessment when the proposed safety action has taken place to determine if it remains adequate.
Revision of CAR 2A approval procedures
In a letter to the ATSB on 21 March 2017, CASA advised that they have '... among other things, significantly revised and formalised its data approval processes in the following manner:
a. Published instructions in the Safety Assurance Branch Inspector Handbook for Airworthiness and Engineering Standards Branch (AEB) to be consulted in the event a Civil Aviation Regulations (CAR) 2A Delegate requires further technical advice in respect to the approval of maintenance data, including Non Destructive Testing (NDT) data or procedures;
b. Published a CAR 2A maintenance data approval process, which applies to NDT data and procedures, in the AEB Procedures Manual. The procedure advises the engagement of the National Aviation NDT Board (NANDTB) as subject matter experts, who have been formally contracted to provide advice to CASA in the area of NDT data and procedures, and
c. CASA has requested NANDTB formalise its procedure for receiving CASA requests for the assessment of NDT data. This procedure involves the NANDTB assessing document requests provided:
i. against known standards,
ii. using best practice,
iii. including relevant information,
iv. applying the NDT Level 3 assessors best judgement, and
v. offering written comments which directly address the nature of the request.
Revision of Certificate of Approval Handbook
In an email on 27 April 2017, CASA informed the ATSB that:
'In response to your further query, please be advised that CASA’s Certificate of Approval Handbook Chapter 9 (see attached) has been updated to reflect the CAR 2A maintenance data approval process for CASA Regional Offices as follows:
9.2.11 Procedures for the approval of Alternate Maintenance Data
If there is a requirement for a certificate of approval holder to develop and use maintenance data that requires approval under CAR2A, the manual will include instruction on how the data is developed, submitted to CASA for approval and subsequently controlled as approved maintenance
data.
Note: Refer to CASA’s Airworthiness and Engineering Branch Procedures Manual Section 3.6 for the CASA process for the approval of data under CAR2A.
This above amendment is in addition to the changes made to the AEB Procedures Manual for CAR 2A approvals and the Draft NANDT Board internal procedure for assessing and approving advice to CASA previously submitted to the ATSB.
ATSB assessment of CASA actions from 21 March 2017 and 27 April 2017
On 24 May 2017, the ATSB wrote a letter to CASA advising of the following assessment of the actions CASA informed the ATSB of having taken in correspondence on 21 March 2017 and 27 April 2017:
The ATSB acknowledges improvement in the processes that CASA has developed to address safety issue AO-2013-187-Sl-08 including the formalisation of a CAR 2A approval process and the inclusion of references to non-destructive testing (NDT) approvals. The addition of a peer review within the process will also provide an additional layer of assurance to the approval process.
Further, CASA's process also now includes consideration of seeking the views of the manufacturer for alternative methods of NDT approvals, which the ATSB believes will also provide an improved level of safety if such contact is made.
However, the ATSB considers that the safety actions taken by CASA to date have not addressed safety issue AO-2013-187-Sl-08 to the extent that, we feel would prevent a similar occurrence to the M18 eddy current inspection procedure.
While CASA's new procedures include references to comparison with standards for NDT they only provide information on how a particular inspection method should be carried out, not whether such a method is suitable for a particular application.
The assessment and approvals process also appears to rely on the engineering judgment of the assessor and lacks a systematic engineering approach to ensure that the applicant has shown that the proposed procedure provides an equivalent, or better, level of defect detection capability and probability.
The ATSB suggests that CASA's procedure be further developed to require the applicant to provide minimum justification data as part of their application. This would put the onus upon the applicant to satisfy CASA that the required level of safety has been assured without the need for CASA/NANDTB to provide "engineering judgments" during the approval process.
In addition, the process changes have only been applied to CAR 2A approvals and the ATSB is concerned that the same situation could occur with other types of alternate NDT approvals. For example, the CAR 2A approval procedure would not apply to the assessment of an application for an Alternate Means of Compliance (AMOC) to an Airworthiness Directive which needs to be approved under CASR 39. There is just as much, if not a greater, probability of an application for an AMOC than for a CAR 2A approval.
The ATSB considers that a formal process or guideline for CASA to follow as part of their assessment and approval of alternative non-destructive inspection procedures would assist CASA to fully address the safety issue. Some guidance to industry on the minimum requirements for non-destructive inspection processes may also assist in ensuring that adequate justification is provided with an application.
Airworthiness Directive procedures
On 15 June 2017, in response to the ATSB letter of 24 May 2017, CASA wrote a letter to the ATSB advising that:
'CASA previously responded to this safety issue on 21 March 17 and 27 April 17. While the ATSB has acknowledged these responses it still contends that CASA's responses to date are insufficient to prevent a similar occurrence.
CASA concurs that additional guidance to industry on the issue of the approval of non-destructive testing (NDT) data is appropriate. I am advised that an Airworthiness Bulletin specifically addressing this matter is currently under development. Furthermore, CASA has re-instigated an ongoing NDT refresher training program for its Airworthiness Inspectorate and Safety Regulator technical staff. Finally, CASA's existing procedures outlined in the Airworthiness Directives (AD) Manual for the assessment of an alternative means of compliance to an AD already embodies a satisfactory peer review and approval process which is limited to those delegates with Civil Aviation Safety Regulations (CASR) 1998 Part 39.004 delegations.
While CASA acknowledges the validity of the original safety issue, it now considers that the safety actions that have been undertaken adequately address those safety concerns.
ATSB assessment of CASA actions in 15 June 2017
At the time of the letter, the mentioned Airworthiness Bulletin (AWB) had not been released and was not provided directly to the ATSB. A search of the AWBs on the CASA website identified one AWB relating to NDT approvals. That AWB, number 02-060 titled ‘NDT Data – The Use of During the Course of Maintenance’ was issued on 18 September 2017.
The ‘Background’ section in the AWB states:
'CASA has identified cases of NDT inspection data being used for purposes other than that for which it was approved. Additionally, some NDT data has been identified which is lacking appropriate documentation necessary to allow the end user to determine whether the data is approved.'
The AWB then goes on to detail the order of precedence of document approvals (for example, and AD takes precedence over CAR 2A). While this is of value of reminding industry of the appropriate level of approved documents to be using in certain situations (such as safety issue AO-2013-187-SI-03, also identified during the course of the investigation), this does not cover the CASA procedure for the approval of NDT procedures.
CASA identified that they have re-instigated an ongoing refresher training program for its AWIs and Safety Regulator staff. While the ATSB accepts that there is benefit in such training, no detail was provided to show how it provides for a systematic engineering approach to the assessment and approval of proposed NDT procedures.
Finally, CASA refer to the procedures in the Airworthiness Directives (AD) Manual. A copy of that manual was not supplied with the letter. During the course of the accident investigation, the ATSB obtained a copy of the manual at Version 2.0, dated August 2004. The ATSB also obtained a copy of an extracted section from Version 3.4, dated June 2014, pertaining to Alternative Means of Compliance (AMOC) and Exclusions. The CASA website includes a list of the CASA Manuals (Manuals, handbooks and templates | Civil Aviation Safety Authority) but this list did not include the Airworthiness Directives Manual, so the current edition could not be confirmed, or reviewed. During the course of the investigation, the ATSB assessed the contents of these documents and were not satisfied that CASA’s assessment process provided a systematic engineering methodology to the assessment of proposed alternative NDT inspections. That revision included a requirement for a peer review, but it was a generic requirement only and contained no specifics as to what that review entailed.
Although CASA has taken a number of actions that will have most probably improved their approval systems, there is no evidence that it has provided the ‘robust, systematic approach to the assessment and approval of alternative non-destructive inspection procedures to ensure that the proposed method provided an equivalent, or better, level of safety than the original procedure’ that was found to be lacking during the investigation.
