Cognitive Load Assessment of Air Traffic Controller Based on SCNN-TransE Network Using Speech Data
Due to increased air traffic flow, air traffic controllers (ATCs) operate in a state of high load or even overload for long periods of time, which can seriously affect the reliability and efficiency of controllers’ commands. Thus, the early identification of ATCs who are overworked is crucial to the...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-06-01
|
| Series: | Aerospace |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2226-4310/10/7/584 |
| _version_ | 1797590721129086976 |
|---|---|
| author | Jing Yang Hongyu Yang Zhengyuan Wu Xiping Wu |
| author_facet | Jing Yang Hongyu Yang Zhengyuan Wu Xiping Wu |
| author_sort | Jing Yang |
| collection | DOAJ |
| description | Due to increased air traffic flow, air traffic controllers (ATCs) operate in a state of high load or even overload for long periods of time, which can seriously affect the reliability and efficiency of controllers’ commands. Thus, the early identification of ATCs who are overworked is crucial to the maintenance of flight safety while increasing overall flight efficiency. This study uses a comprehensive comparison of existing cognitive load assessment methods combined with the characteristics of the ATC as a basis from which a method for the utilization of speech parameters to assess cognitive load is proposed. This method is ultimately selected due to the minimal interference of the collection equipment and the abundance of speech signals. The speech signal is pre-processed to generate a Mel spectrogram, which contains temporal information in addition to energy, tone, and other spatial information. Therefore, a speech cognitive load evaluation model based on a stacked convolutional neural network (CNN) and the Transformer encoder (SCNN-TransE) is proposed. The use of a CNN and the Transformer encoder allows us to extract spatial features and temporal features, respectively, from contextual information from speech data and facilitates the fusion of spatial features and temporal features into spatio-temporal features, which improves our method’s ability to capture the depth features of speech. We conduct experiments on air traffic control communication data, which show that the detection accuracy and F1 score of SCNN-TransE are better than the results from the support-vector machine (SVM), k-nearest neighbors (KNN), random forest (RF), adaptive boosting (AdaBoost), and stacked CNN parallel long short-term memory with attention (SCNN-LSTM-Attention) models, reaching values of 97.48% and 97.07%, respectively. Thus, our proposed model can realize the effective evaluation of cognitive load levels. |
| first_indexed | 2024-03-11T01:24:32Z |
| format | Article |
| id | doaj.art-7a4bdfd8d44c433ea44f6e725729776b |
| institution | Directory Open Access Journal |
| issn | 2226-4310 |
| language | English |
| last_indexed | 2024-03-11T01:24:32Z |
| publishDate | 2023-06-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Aerospace |
| spelling | doaj.art-7a4bdfd8d44c433ea44f6e725729776b2023-11-18T17:50:20ZengMDPI AGAerospace2226-43102023-06-0110758410.3390/aerospace10070584Cognitive Load Assessment of Air Traffic Controller Based on SCNN-TransE Network Using Speech DataJing Yang0Hongyu Yang1Zhengyuan Wu2Xiping Wu3College of Computer Science, Sichuan University, Chengdu 610065, ChinaCollege of Computer Science, Sichuan University, Chengdu 610065, ChinaCollege of Computer Science, Sichuan University, Chengdu 610065, ChinaCollege of Computer Science, Sichuan University, Chengdu 610065, ChinaDue to increased air traffic flow, air traffic controllers (ATCs) operate in a state of high load or even overload for long periods of time, which can seriously affect the reliability and efficiency of controllers’ commands. Thus, the early identification of ATCs who are overworked is crucial to the maintenance of flight safety while increasing overall flight efficiency. This study uses a comprehensive comparison of existing cognitive load assessment methods combined with the characteristics of the ATC as a basis from which a method for the utilization of speech parameters to assess cognitive load is proposed. This method is ultimately selected due to the minimal interference of the collection equipment and the abundance of speech signals. The speech signal is pre-processed to generate a Mel spectrogram, which contains temporal information in addition to energy, tone, and other spatial information. Therefore, a speech cognitive load evaluation model based on a stacked convolutional neural network (CNN) and the Transformer encoder (SCNN-TransE) is proposed. The use of a CNN and the Transformer encoder allows us to extract spatial features and temporal features, respectively, from contextual information from speech data and facilitates the fusion of spatial features and temporal features into spatio-temporal features, which improves our method’s ability to capture the depth features of speech. We conduct experiments on air traffic control communication data, which show that the detection accuracy and F1 score of SCNN-TransE are better than the results from the support-vector machine (SVM), k-nearest neighbors (KNN), random forest (RF), adaptive boosting (AdaBoost), and stacked CNN parallel long short-term memory with attention (SCNN-LSTM-Attention) models, reaching values of 97.48% and 97.07%, respectively. Thus, our proposed model can realize the effective evaluation of cognitive load levels.https://www.mdpi.com/2226-4310/10/7/584air traffic controllercognitive load assessmentMel spectrogramtransformer |
| spellingShingle | Jing Yang Hongyu Yang Zhengyuan Wu Xiping Wu Cognitive Load Assessment of Air Traffic Controller Based on SCNN-TransE Network Using Speech Data Aerospace air traffic controller cognitive load assessment Mel spectrogram transformer |
| title | Cognitive Load Assessment of Air Traffic Controller Based on SCNN-TransE Network Using Speech Data |
| title_full | Cognitive Load Assessment of Air Traffic Controller Based on SCNN-TransE Network Using Speech Data |
| title_fullStr | Cognitive Load Assessment of Air Traffic Controller Based on SCNN-TransE Network Using Speech Data |
| title_full_unstemmed | Cognitive Load Assessment of Air Traffic Controller Based on SCNN-TransE Network Using Speech Data |
| title_short | Cognitive Load Assessment of Air Traffic Controller Based on SCNN-TransE Network Using Speech Data |
| title_sort | cognitive load assessment of air traffic controller based on scnn transe network using speech data |
| topic | air traffic controller cognitive load assessment Mel spectrogram transformer |
| url | https://www.mdpi.com/2226-4310/10/7/584 |
| work_keys_str_mv | AT jingyang cognitiveloadassessmentofairtrafficcontrollerbasedonscnntransenetworkusingspeechdata AT hongyuyang cognitiveloadassessmentofairtrafficcontrollerbasedonscnntransenetworkusingspeechdata AT zhengyuanwu cognitiveloadassessmentofairtrafficcontrollerbasedonscnntransenetworkusingspeechdata AT xipingwu cognitiveloadassessmentofairtrafficcontrollerbasedonscnntransenetworkusingspeechdata |