Robust Attitude Control of an Agile Aircraft Using Improved Q-Learning
Attitude control of a novel regional truss-braced wing (TBW) aircraft with low stability characteristics is addressed in this paper using Reinforcement Learning (RL). In recent years, RL has been increasingly employed in challenging applications, particularly, autonomous flight control. However, a s...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-12-01
|
| Series: | Actuators |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-0825/11/12/374 |
| _version_ | 1797461954272428032 |
|---|---|
| author | Mohsen Zahmatkesh Seyyed Ali Emami Afshin Banazadeh Paolo Castaldi |
| author_facet | Mohsen Zahmatkesh Seyyed Ali Emami Afshin Banazadeh Paolo Castaldi |
| author_sort | Mohsen Zahmatkesh |
| collection | DOAJ |
| description | Attitude control of a novel regional truss-braced wing (TBW) aircraft with low stability characteristics is addressed in this paper using Reinforcement Learning (RL). In recent years, RL has been increasingly employed in challenging applications, particularly, autonomous flight control. However, a significant predicament confronting discrete RL algorithms is the dimension limitation of the state-action table and difficulties in defining the elements of the RL environment. To address these issues, in this paper, a detailed mathematical model of the mentioned aircraft is first developed to shape an RL environment. Subsequently, Q-learning, the most prevalent discrete RL algorithm, will be implemented in both the Markov Decision Process (MDP) and Partially Observable Markov Decision Process (POMDP) frameworks to control the longitudinal mode of the proposed aircraft. In order to eliminate residual fluctuations that are a consequence of discrete action selection, and simultaneously track variable pitch angles, a Fuzzy Action Assignment (FAA) method is proposed to generate continuous control commands using the trained optimal Q-table. Accordingly, it will be proved that by defining a comprehensive reward function based on dynamic behavior considerations, along with observing all crucial states (equivalent to satisfying the Markov Property), the air vehicle would be capable of tracking the desired attitude in the presence of different uncertain dynamics including measurement noises, atmospheric disturbances, actuator faults, and model uncertainties where the performance of the introduced control system surpasses a well-tuned Proportional–Integral–Derivative (PID) controller. |
| first_indexed | 2024-03-09T17:27:37Z |
| format | Article |
| id | doaj.art-1cf3a0e17e7047f6b993b38beaebcb42 |
| institution | Directory Open Access Journal |
| issn | 2076-0825 |
| language | English |
| last_indexed | 2024-03-09T17:27:37Z |
| publishDate | 2022-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Actuators |
| spelling | doaj.art-1cf3a0e17e7047f6b993b38beaebcb422023-11-24T12:35:27ZengMDPI AGActuators2076-08252022-12-01111237410.3390/act11120374Robust Attitude Control of an Agile Aircraft Using Improved Q-LearningMohsen Zahmatkesh0Seyyed Ali Emami1Afshin Banazadeh2Paolo Castaldi3Department of Aerospace Engineering, Sharif University of Technology, Tehran 1458889694, IranDepartment of Aerospace Engineering, Sharif University of Technology, Tehran 1458889694, IranDepartment of Aerospace Engineering, Sharif University of Technology, Tehran 1458889694, IranDepartment of Electrical, Electronic and Information Engineering “Guglielmo Marconi”, University of Bologna, Via Dell’Universit‘a 50, 40126 Cesena, ItalyAttitude control of a novel regional truss-braced wing (TBW) aircraft with low stability characteristics is addressed in this paper using Reinforcement Learning (RL). In recent years, RL has been increasingly employed in challenging applications, particularly, autonomous flight control. However, a significant predicament confronting discrete RL algorithms is the dimension limitation of the state-action table and difficulties in defining the elements of the RL environment. To address these issues, in this paper, a detailed mathematical model of the mentioned aircraft is first developed to shape an RL environment. Subsequently, Q-learning, the most prevalent discrete RL algorithm, will be implemented in both the Markov Decision Process (MDP) and Partially Observable Markov Decision Process (POMDP) frameworks to control the longitudinal mode of the proposed aircraft. In order to eliminate residual fluctuations that are a consequence of discrete action selection, and simultaneously track variable pitch angles, a Fuzzy Action Assignment (FAA) method is proposed to generate continuous control commands using the trained optimal Q-table. Accordingly, it will be proved that by defining a comprehensive reward function based on dynamic behavior considerations, along with observing all crucial states (equivalent to satisfying the Markov Property), the air vehicle would be capable of tracking the desired attitude in the presence of different uncertain dynamics including measurement noises, atmospheric disturbances, actuator faults, and model uncertainties where the performance of the introduced control system surpasses a well-tuned Proportional–Integral–Derivative (PID) controller.https://www.mdpi.com/2076-0825/11/12/374reinforcement learningq-learningfuzzy q-learningattitude controltruss-braced wingflight control |
| spellingShingle | Mohsen Zahmatkesh Seyyed Ali Emami Afshin Banazadeh Paolo Castaldi Robust Attitude Control of an Agile Aircraft Using Improved Q-Learning Actuators reinforcement learning q-learning fuzzy q-learning attitude control truss-braced wing flight control |
| title | Robust Attitude Control of an Agile Aircraft Using Improved Q-Learning |
| title_full | Robust Attitude Control of an Agile Aircraft Using Improved Q-Learning |
| title_fullStr | Robust Attitude Control of an Agile Aircraft Using Improved Q-Learning |
| title_full_unstemmed | Robust Attitude Control of an Agile Aircraft Using Improved Q-Learning |
| title_short | Robust Attitude Control of an Agile Aircraft Using Improved Q-Learning |
| title_sort | robust attitude control of an agile aircraft using improved q learning |
| topic | reinforcement learning q-learning fuzzy q-learning attitude control truss-braced wing flight control |
| url | https://www.mdpi.com/2076-0825/11/12/374 |
| work_keys_str_mv | AT mohsenzahmatkesh robustattitudecontrolofanagileaircraftusingimprovedqlearning AT seyyedaliemami robustattitudecontrolofanagileaircraftusingimprovedqlearning AT afshinbanazadeh robustattitudecontrolofanagileaircraftusingimprovedqlearning AT paolocastaldi robustattitudecontrolofanagileaircraftusingimprovedqlearning |