Learning Assurance Analysis for Further Certification Process of Machine Learning Techniques: Case-Study Air Traffic Conflict Detection Predictor
Designing and developing artificial intelligence (AI)-based systems that can be trusted justifiably is one of the main issues aviation must face in the coming years. European Union Aviation Safety Agency (EASA) has developed a user guide that could be potentially transformed as means of compliance f...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-10-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/22/19/7680 |
| _version_ | 1827652981745516544 |
|---|---|
| author | Javier A. Pérez-Castán Luis Pérez Sanz Marta Fernández-Castellano Tomislav Radišić Kristina Samardžić Ivan Tukarić |
| author_facet | Javier A. Pérez-Castán Luis Pérez Sanz Marta Fernández-Castellano Tomislav Radišić Kristina Samardžić Ivan Tukarić |
| author_sort | Javier A. Pérez-Castán |
| collection | DOAJ |
| description | Designing and developing artificial intelligence (AI)-based systems that can be trusted justifiably is one of the main issues aviation must face in the coming years. European Union Aviation Safety Agency (EASA) has developed a user guide that could be potentially transformed as means of compliance for future AI-based regulation. Designers and developers must understand how the learning assurance process of any machine learning (ML) model impacts trust. ML is a narrow branch of AI that uses statistical models to perform predictions. This work deals with the learning assurance process for ML-based systems in the field of air traffic control. A conflict detection tool has been developed to identify separation infringements among aircraft pairs, and the ML algorithm used for classification and regression was extreme gradient boosting. This paper analyses the validity and adaptability of EASA W-shaped methodology for ML-based systems. The results have identified the lack of the EASA W-shaped methodology in time-dependent analysis, by showing how time can impact ML algorithms designed in the case where no time requirements are considered. Another meaningful conclusion is, for systems that depend highly on when the prediction is made, classification and regression metrics cannot be one-size-fits-all because they vary over time. |
| first_indexed | 2024-03-09T21:09:42Z |
| format | Article |
| id | doaj.art-fe88aa3394384d7db102da76b2dc8ed3 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T21:09:42Z |
| publishDate | 2022-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-fe88aa3394384d7db102da76b2dc8ed32023-11-23T21:53:05ZengMDPI AGSensors1424-82202022-10-012219768010.3390/s22197680Learning Assurance Analysis for Further Certification Process of Machine Learning Techniques: Case-Study Air Traffic Conflict Detection PredictorJavier A. Pérez-Castán0Luis Pérez Sanz1Marta Fernández-Castellano2Tomislav Radišić3Kristina Samardžić4Ivan Tukarić5ETSI Aeronáutica y del Espacio, Plaza Cardenal Cisneros, Universidad Politécnica de Madrid, 28008 Madrid, SpainETSI Aeronáutica y del Espacio, Plaza Cardenal Cisneros, Universidad Politécnica de Madrid, 28008 Madrid, SpainETSI Aeronáutica y del Espacio, Plaza Cardenal Cisneros, Universidad Politécnica de Madrid, 28008 Madrid, SpainFaculty of Transport and Traffic Sciences, University of Zagreb, Borongajska Cesta, 10000 Zagreb, CroatiaFaculty of Transport and Traffic Sciences, University of Zagreb, Borongajska Cesta, 10000 Zagreb, CroatiaFaculty of Transport and Traffic Sciences, University of Zagreb, Borongajska Cesta, 10000 Zagreb, CroatiaDesigning and developing artificial intelligence (AI)-based systems that can be trusted justifiably is one of the main issues aviation must face in the coming years. European Union Aviation Safety Agency (EASA) has developed a user guide that could be potentially transformed as means of compliance for future AI-based regulation. Designers and developers must understand how the learning assurance process of any machine learning (ML) model impacts trust. ML is a narrow branch of AI that uses statistical models to perform predictions. This work deals with the learning assurance process for ML-based systems in the field of air traffic control. A conflict detection tool has been developed to identify separation infringements among aircraft pairs, and the ML algorithm used for classification and regression was extreme gradient boosting. This paper analyses the validity and adaptability of EASA W-shaped methodology for ML-based systems. The results have identified the lack of the EASA W-shaped methodology in time-dependent analysis, by showing how time can impact ML algorithms designed in the case where no time requirements are considered. Another meaningful conclusion is, for systems that depend highly on when the prediction is made, classification and regression metrics cannot be one-size-fits-all because they vary over time.https://www.mdpi.com/1424-8220/22/19/7680air transportconflict detectionmachine learninglearning assurancetrustworthiness |
| spellingShingle | Javier A. Pérez-Castán Luis Pérez Sanz Marta Fernández-Castellano Tomislav Radišić Kristina Samardžić Ivan Tukarić Learning Assurance Analysis for Further Certification Process of Machine Learning Techniques: Case-Study Air Traffic Conflict Detection Predictor Sensors air transport conflict detection machine learning learning assurance trustworthiness |
| title | Learning Assurance Analysis for Further Certification Process of Machine Learning Techniques: Case-Study Air Traffic Conflict Detection Predictor |
| title_full | Learning Assurance Analysis for Further Certification Process of Machine Learning Techniques: Case-Study Air Traffic Conflict Detection Predictor |
| title_fullStr | Learning Assurance Analysis for Further Certification Process of Machine Learning Techniques: Case-Study Air Traffic Conflict Detection Predictor |
| title_full_unstemmed | Learning Assurance Analysis for Further Certification Process of Machine Learning Techniques: Case-Study Air Traffic Conflict Detection Predictor |
| title_short | Learning Assurance Analysis for Further Certification Process of Machine Learning Techniques: Case-Study Air Traffic Conflict Detection Predictor |
| title_sort | learning assurance analysis for further certification process of machine learning techniques case study air traffic conflict detection predictor |
| topic | air transport conflict detection machine learning learning assurance trustworthiness |
| url | https://www.mdpi.com/1424-8220/22/19/7680 |
| work_keys_str_mv | AT javieraperezcastan learningassuranceanalysisforfurthercertificationprocessofmachinelearningtechniquescasestudyairtrafficconflictdetectionpredictor AT luisperezsanz learningassuranceanalysisforfurthercertificationprocessofmachinelearningtechniquescasestudyairtrafficconflictdetectionpredictor AT martafernandezcastellano learningassuranceanalysisforfurthercertificationprocessofmachinelearningtechniquescasestudyairtrafficconflictdetectionpredictor AT tomislavradisic learningassuranceanalysisforfurthercertificationprocessofmachinelearningtechniquescasestudyairtrafficconflictdetectionpredictor AT kristinasamardzic learningassuranceanalysisforfurthercertificationprocessofmachinelearningtechniquescasestudyairtrafficconflictdetectionpredictor AT ivantukaric learningassuranceanalysisforfurthercertificationprocessofmachinelearningtechniquescasestudyairtrafficconflictdetectionpredictor |