Hierarchical multi‐agent reinforcement learning for multi‐aircraft close‐range air combat
Abstract The close‐range autonomous air combat has gained significant attention from researchers involved in applications related to artificial intelligence (AI). A majority of the previous studies on autonomous air combat were focused on one‐on‐one air combat scenarios, however, the modern air comb...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2023-09-01
|
| Series: | IET Control Theory & Applications |
| Subjects: | |
| Online Access: | https://doi.org/10.1049/cth2.12413 |
| _version_ | 1797729143763238912 |
|---|---|
| author | Wei‐ren Kong De‐yun Zhou Yong‐jie Du Ying Zhou Yi‐yang Zhao |
| author_facet | Wei‐ren Kong De‐yun Zhou Yong‐jie Du Ying Zhou Yi‐yang Zhao |
| author_sort | Wei‐ren Kong |
| collection | DOAJ |
| description | Abstract The close‐range autonomous air combat has gained significant attention from researchers involved in applications related to artificial intelligence (AI). A majority of the previous studies on autonomous air combat were focused on one‐on‐one air combat scenarios, however, the modern air combat is mostly conducted in formations. With regard to the aforementioned factors, a novel hierarchical maneuvering control architecture is introduced that is applied to the multi‐aircraft close‐range air combat scenario, which can handle air combat scenarios with variable‐size formation. Subsequently, three air combat sub‐tasks are designed, and recurrent soft actor‐critic (RSAC) algorithm combined with competitive self‐play (SP) is incorporated to learn the sub‐strategies. A novel hierarchical multi‐agent reinforcement learning (HMARL) algorithm is proposed to obtain the high‐level strategy for target and sub‐strategy selection. The training performance of the training algorithm of sub‐strategies and high‐level strategy in different air combat scenarios is evaluated. The obtained strategies are analyzed and it is found that the formations exhibit effective cooperative behavior in symmetric and asymmetric scenarios. Finally, the ideas of engineering implementation of the maneuvering control architecture are given. The study provides a solution for future multi‐aircraft autonomous air combat. |
| first_indexed | 2024-03-12T11:24:47Z |
| format | Article |
| id | doaj.art-0c6fbc40553140868223752bc4aba667 |
| institution | Directory Open Access Journal |
| issn | 1751-8644 1751-8652 |
| language | English |
| last_indexed | 2024-03-12T11:24:47Z |
| publishDate | 2023-09-01 |
| publisher | Wiley |
| record_format | Article |
| series | IET Control Theory & Applications |
| spelling | doaj.art-0c6fbc40553140868223752bc4aba6672023-09-01T09:21:24ZengWileyIET Control Theory & Applications1751-86441751-86522023-09-0117131840186210.1049/cth2.12413Hierarchical multi‐agent reinforcement learning for multi‐aircraft close‐range air combatWei‐ren Kong0De‐yun Zhou1Yong‐jie Du2Ying Zhou3Yi‐yang Zhao4Northwestern Polytechnical University Xi'an ChinaNorthwestern Polytechnical University Xi'an ChinaNorthwestern Polytechnical University Xi'an ChinaNorthwestern Polytechnical University Xi'an ChinaNorthwestern Polytechnical University Xi'an ChinaAbstract The close‐range autonomous air combat has gained significant attention from researchers involved in applications related to artificial intelligence (AI). A majority of the previous studies on autonomous air combat were focused on one‐on‐one air combat scenarios, however, the modern air combat is mostly conducted in formations. With regard to the aforementioned factors, a novel hierarchical maneuvering control architecture is introduced that is applied to the multi‐aircraft close‐range air combat scenario, which can handle air combat scenarios with variable‐size formation. Subsequently, three air combat sub‐tasks are designed, and recurrent soft actor‐critic (RSAC) algorithm combined with competitive self‐play (SP) is incorporated to learn the sub‐strategies. A novel hierarchical multi‐agent reinforcement learning (HMARL) algorithm is proposed to obtain the high‐level strategy for target and sub‐strategy selection. The training performance of the training algorithm of sub‐strategies and high‐level strategy in different air combat scenarios is evaluated. The obtained strategies are analyzed and it is found that the formations exhibit effective cooperative behavior in symmetric and asymmetric scenarios. Finally, the ideas of engineering implementation of the maneuvering control architecture are given. The study provides a solution for future multi‐aircraft autonomous air combat.https://doi.org/10.1049/cth2.12413autonomous air combatartificial intelligencecompetitive self‐playmaneuver decision‐makingmulti‐agent reinforcement learning |
| spellingShingle | Wei‐ren Kong De‐yun Zhou Yong‐jie Du Ying Zhou Yi‐yang Zhao Hierarchical multi‐agent reinforcement learning for multi‐aircraft close‐range air combat IET Control Theory & Applications autonomous air combat artificial intelligence competitive self‐play maneuver decision‐making multi‐agent reinforcement learning |
| title | Hierarchical multi‐agent reinforcement learning for multi‐aircraft close‐range air combat |
| title_full | Hierarchical multi‐agent reinforcement learning for multi‐aircraft close‐range air combat |
| title_fullStr | Hierarchical multi‐agent reinforcement learning for multi‐aircraft close‐range air combat |
| title_full_unstemmed | Hierarchical multi‐agent reinforcement learning for multi‐aircraft close‐range air combat |
| title_short | Hierarchical multi‐agent reinforcement learning for multi‐aircraft close‐range air combat |
| title_sort | hierarchical multi agent reinforcement learning for multi aircraft close range air combat |
| topic | autonomous air combat artificial intelligence competitive self‐play maneuver decision‐making multi‐agent reinforcement learning |
| url | https://doi.org/10.1049/cth2.12413 |
| work_keys_str_mv | AT weirenkong hierarchicalmultiagentreinforcementlearningformultiaircraftcloserangeaircombat AT deyunzhou hierarchicalmultiagentreinforcementlearningformultiaircraftcloserangeaircombat AT yongjiedu hierarchicalmultiagentreinforcementlearningformultiaircraftcloserangeaircombat AT yingzhou hierarchicalmultiagentreinforcementlearningformultiaircraftcloserangeaircombat AT yiyangzhao hierarchicalmultiagentreinforcementlearningformultiaircraftcloserangeaircombat |