Periodic Reporting for period 2 - TAPAS (Towards an Automated and exPlainable ATM System)
Période du rapport: 2021-06-01 au 2022-11-30
TAPAS (Towards an Automated and exPlainable ATM System) addresses the effectiveness of introducing AI/ML solutions in order to increase the levels of automation in ATM, considering the need of the operator to trust the system (taken as the ability to understand and explain its behaviour and outcomes).
The main objective for the project was the exploration of highly automated XAI scenarios through validation activities and Visual Analytics (VA), in order to identify needs and strategies to address transparency and explainability in the operational cases considered, paving the way for the application of these AI/ML technologies in ATM environments, in particular in automation levels 2 and 3 as expressed in the successive editions of the European ATM Master Plan.
This is reflected in two key technical objectives:
Objective 1: Identification of principles and criteria for AI/ML transparency/explainability in ATM domain scenarios
This objective applies to the two operational cases considered (ATFCM – Air Traffic Flow and Capacity Management; and CD&R – Conflict Detection and Resolution) and with the target to identify transparency requirements for AI/ML methods in general, limiting domain-specific results. The strategy to achieve this goal was based in addressing different temporal, functional and safety-critical perspectives, as those provided by the complementary operational cases considered in TAPAS. The project put specific focus to maximise the applicability of results to different operational environments, while setting the limitations when this is not feasible.
The project explored the use of XAI and VA to apply them in the operational cases considered, through practical experiments and validation activities in simulation platforms. In particular, for each level of automation considered in each operational case, the project implemented a distribution of functionalities between the human and the machine including AI/ML ones. These were verified using real-time simulations (RTS) including operational staff (Air Traffic Controllers) both providing a-priori and a-posteriori expert judgement, and objective criteria verification.
Objetive 2: Selection and development of suitable and explainable AI/ML methods in the operational cases identified
This selection had the goal to fit the needs of transparency as expressed in the explainability criteria developed for each automation level and according to actors’ needs.
The project developed prototypes of XAI methods which addressed the balance between explainability and effectiveness according to specific needs, but also in search of developing a more general taxonomy of AI/ML techniques considering the two aforementioned magnitudes. Given the early Technology Readiness Level (TRL) of this project (initially pre-TRL 1, although eventually the project reached a "TRL 2 ongoing" classification), these prototypes were focused on testing purposes.
TAPAS project achieved both objectives, also reaching fully TRL 1 maturity and partly TRL 2.
The main objective for the project was the exploration of highly automated XAI scenarios through validation activities and Visual Analytics (VA), in order to identify needs and strategies to address transparency and explainability in the operational cases considered, paving the way for the application of these AI/ML technologies in ATM environments, in particular in automation levels 2 and 3 as expressed in the successive editions of the European ATM Master Plan.
This is reflected in two key technical objectives:
Objective 1: Identification of principles and criteria for AI/ML transparency/explainability in ATM domain scenarios
This objective applies to the two operational cases considered (ATFCM – Air Traffic Flow and Capacity Management; and CD&R – Conflict Detection and Resolution) and with the target to identify transparency requirements for AI/ML methods in general, limiting domain-specific results. The strategy to achieve this goal was based in addressing different temporal, functional and safety-critical perspectives, as those provided by the complementary operational cases considered in TAPAS. The project put specific focus to maximise the applicability of results to different operational environments, while setting the limitations when this is not feasible.
The project explored the use of XAI and VA to apply them in the operational cases considered, through practical experiments and validation activities in simulation platforms. In particular, for each level of automation considered in each operational case, the project implemented a distribution of functionalities between the human and the machine including AI/ML ones. These were verified using real-time simulations (RTS) including operational staff (Air Traffic Controllers) both providing a-priori and a-posteriori expert judgement, and objective criteria verification.
Objetive 2: Selection and development of suitable and explainable AI/ML methods in the operational cases identified
This selection had the goal to fit the needs of transparency as expressed in the explainability criteria developed for each automation level and according to actors’ needs.
The project developed prototypes of XAI methods which addressed the balance between explainability and effectiveness according to specific needs, but also in search of developing a more general taxonomy of AI/ML techniques considering the two aforementioned magnitudes. Given the early Technology Readiness Level (TRL) of this project (initially pre-TRL 1, although eventually the project reached a "TRL 2 ongoing" classification), these prototypes were focused on testing purposes.
TAPAS project achieved both objectives, also reaching fully TRL 1 maturity and partly TRL 2.
The project has addressed its full lifecycle completing its intended technical scope. This includes the development of XAI and VA prototypes for the two use cases considered (ATFCM and CD&R) in Automation Levels 2 & 3 as defined in ATM Master Plan. This is particularly significant in the case of Automation level 3, which implies direct implementation by the AI/ML system of its own decisions, without direct intervention from the human operator which plays a monitoring role in this case. This process is significantly different from current operations.
The application of XAI techniques to ATM domain has also been completed and integrated successfully with existing system in both use cases, proving the technical feasibility of the proposed AI/ML for its adoption in ATM.
The project has extracted the meaningful conclusions from both experimental validations and extrapolated them to generate a framework of Principles for Transparency in automation levels 2 and 3, to serve other potential future application irrespective of its particular use case. This is intended to serve as a reference for future work in this direction.
TAPAS has put focus in the communication and exploitation of its results, with the following overview of actions:
o A specific website (https://tapas-atm.eu/)
o A Twitter account
o A LinkedIn account
o Participation in SESAR Newsletter
o Participation in SESAR Innovation Days 2020 and 2022
o Participation in ER4 Automation Workshops (including EASN 2022)
o Participation in SESAR Digital Academy
o Organization of specific workshops
o Participation in WAC 2022
o Participation in SESAR Annual Conference 2022
o Production of a whitepaper in coordination with Automation ER4 projects
o Publication of seven peer-reviewed scientific papers, including journals and conferences
The application of XAI techniques to ATM domain has also been completed and integrated successfully with existing system in both use cases, proving the technical feasibility of the proposed AI/ML for its adoption in ATM.
The project has extracted the meaningful conclusions from both experimental validations and extrapolated them to generate a framework of Principles for Transparency in automation levels 2 and 3, to serve other potential future application irrespective of its particular use case. This is intended to serve as a reference for future work in this direction.
TAPAS has put focus in the communication and exploitation of its results, with the following overview of actions:
o A specific website (https://tapas-atm.eu/)
o A Twitter account
o A LinkedIn account
o Participation in SESAR Newsletter
o Participation in SESAR Innovation Days 2020 and 2022
o Participation in ER4 Automation Workshops (including EASN 2022)
o Participation in SESAR Digital Academy
o Organization of specific workshops
o Participation in WAC 2022
o Participation in SESAR Annual Conference 2022
o Production of a whitepaper in coordination with Automation ER4 projects
o Publication of seven peer-reviewed scientific papers, including journals and conferences
TAPAS Project has made a step forward in the deployment of AI/ML technologies in ATM by:
-Developing XAI (eXplainable AI) prototypes for Automation levels 2 and 3 (as defined in ATM MP) in two scenarios: ATFCM and Conflict Detection&Resolution. This is a pioneer activity for level 3,including automated action implementation.
-Validation with experienced ATCOs in RTS, exploring different strategies for human-machine cooperation (what, when and how AI decissions need to be explained). These validation activities were conducted in a realistic ATC paltform simulator and helped to achieve a TRL-2 ongoing maturity level
-Producing a Framework of Explainability/Transparency Principles for future AI/ML applications, from requirements to testing.
The impact achieved by the project is, in line with the expectations, “to increase the body of knowledge of Safety and Resilience of the ATM system by identifying areas of improvement, providing design guidelines and identifying future research needs and in doing so supporting decision”.
In detail, the expected impacts are:
• Enabling AI/ML Automation for ATM: This has a direct impact in KPA Capacity according to SESAR Performance Framework, and as a consequence also addresses the corresponding SESAR Performance Ambition.
• Reduction of Air Navigation System (ANS) Costs: This will bring benefits in terms of Technology cost, a contributing factor for KPA Cost Efficiency (equates to ANS Productivity) as described in SESAR Performance Framework. This impact is fully aligned with the SESAR Performance Ambition.
• Increase of ATCO Productivity: . This benefit will increase the ATCO Productivity that again, has a direct positive impact on KPA Cost-Efficiency and also to the SESAR Performance Ambition.
• Ensure Safety Level through Situational Awareness: Ensuring situational awareness is an essential focus area for Safety and Human Performance KPAs. Contributing to it in AI/ML automated environments where other KPAs are improved, is a key condition for the validity of TAPAS’ outcomes. Again, this impact addresses the SESAR Performance Ambition.
-Developing XAI (eXplainable AI) prototypes for Automation levels 2 and 3 (as defined in ATM MP) in two scenarios: ATFCM and Conflict Detection&Resolution. This is a pioneer activity for level 3,including automated action implementation.
-Validation with experienced ATCOs in RTS, exploring different strategies for human-machine cooperation (what, when and how AI decissions need to be explained). These validation activities were conducted in a realistic ATC paltform simulator and helped to achieve a TRL-2 ongoing maturity level
-Producing a Framework of Explainability/Transparency Principles for future AI/ML applications, from requirements to testing.
The impact achieved by the project is, in line with the expectations, “to increase the body of knowledge of Safety and Resilience of the ATM system by identifying areas of improvement, providing design guidelines and identifying future research needs and in doing so supporting decision”.
In detail, the expected impacts are:
• Enabling AI/ML Automation for ATM: This has a direct impact in KPA Capacity according to SESAR Performance Framework, and as a consequence also addresses the corresponding SESAR Performance Ambition.
• Reduction of Air Navigation System (ANS) Costs: This will bring benefits in terms of Technology cost, a contributing factor for KPA Cost Efficiency (equates to ANS Productivity) as described in SESAR Performance Framework. This impact is fully aligned with the SESAR Performance Ambition.
• Increase of ATCO Productivity: . This benefit will increase the ATCO Productivity that again, has a direct positive impact on KPA Cost-Efficiency and also to the SESAR Performance Ambition.
• Ensure Safety Level through Situational Awareness: Ensuring situational awareness is an essential focus area for Safety and Human Performance KPAs. Contributing to it in AI/ML automated environments where other KPAs are improved, is a key condition for the validity of TAPAS’ outcomes. Again, this impact addresses the SESAR Performance Ambition.