Deep Multi-Agent Reinforcement Learning With Minimal Cross-Agent Communication for SFC Partitioning
Network Function Virtualization (NFV) decouples network functions from the underlying specialized devices, enabling network processing with higher flexibility and resource efficiency. This promotes the use of virtual network functions (VNFs), which can be grouped to form a service function chain (SF...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10107385/ |
| _version_ | 1797838071330242560 |
|---|---|
| author | Angelos Pentelas Danny De Vleeschauwer Chia-Yu Chang Koen De Schepper Panagiotis Papadimitriou |
| author_facet | Angelos Pentelas Danny De Vleeschauwer Chia-Yu Chang Koen De Schepper Panagiotis Papadimitriou |
| author_sort | Angelos Pentelas |
| collection | DOAJ |
| description | Network Function Virtualization (NFV) decouples network functions from the underlying specialized devices, enabling network processing with higher flexibility and resource efficiency. This promotes the use of virtual network functions (VNFs), which can be grouped to form a service function chain (SFC). A critical challenge in NFV is SFC partitioning (SFCP), which is mathematically expressed as a graph-to-graph mapping problem. Given its NP-hardness, SFCP is commonly solved by approximation methods. Yet, the relevant literature exhibits a gradual shift towards data-driven SFCP frameworks, such as (deep) reinforcement learning (RL). In this article, we initially identify crucial limitations of existing RL-based SFCP approaches. In particular, we argue that most of them stem from the centralized implementation of RL schemes. Therefore, we devise a cooperative deep multi-agent reinforcement learning (DMARL) scheme for decentralized SFCP, which fosters the efficient communication of neighboring agents. Our simulation results (i) demonstrate that DMARL outperforms a state-of-the-art centralized double deep <inline-formula> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -learning algorithm, (ii) unfold the fundamental behaviors learned by the team of agents, (iii) highlight the importance of information exchange between agents, and (iv) showcase the implications stemming from various network topologies on the DMARL efficiency. |
| first_indexed | 2024-04-09T15:34:56Z |
| format | Article |
| id | doaj.art-df15bfa3754243bf8681528473d29de3 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-09T15:34:56Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-df15bfa3754243bf8681528473d29de32023-04-27T23:00:53ZengIEEEIEEE Access2169-35362023-01-0111403844039810.1109/ACCESS.2023.326957610107385Deep Multi-Agent Reinforcement Learning With Minimal Cross-Agent Communication for SFC PartitioningAngelos Pentelas0https://orcid.org/0000-0002-8502-7872Danny De Vleeschauwer1https://orcid.org/0000-0002-0718-8048Chia-Yu Chang2https://orcid.org/0000-0002-7020-8451Koen De Schepper3https://orcid.org/0000-0003-2839-7006Panagiotis Papadimitriou4https://orcid.org/0000-0001-5005-8866Nokia Bell Laboratories, Antwerp, BelgiumNokia Bell Laboratories, Antwerp, BelgiumNokia Bell Laboratories, Antwerp, BelgiumNokia Bell Laboratories, Antwerp, BelgiumDepartment of Applied Informatics, University of Macedonia, Thessaloniki, GreeceNetwork Function Virtualization (NFV) decouples network functions from the underlying specialized devices, enabling network processing with higher flexibility and resource efficiency. This promotes the use of virtual network functions (VNFs), which can be grouped to form a service function chain (SFC). A critical challenge in NFV is SFC partitioning (SFCP), which is mathematically expressed as a graph-to-graph mapping problem. Given its NP-hardness, SFCP is commonly solved by approximation methods. Yet, the relevant literature exhibits a gradual shift towards data-driven SFCP frameworks, such as (deep) reinforcement learning (RL). In this article, we initially identify crucial limitations of existing RL-based SFCP approaches. In particular, we argue that most of them stem from the centralized implementation of RL schemes. Therefore, we devise a cooperative deep multi-agent reinforcement learning (DMARL) scheme for decentralized SFCP, which fosters the efficient communication of neighboring agents. Our simulation results (i) demonstrate that DMARL outperforms a state-of-the-art centralized double deep <inline-formula> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -learning algorithm, (ii) unfold the fundamental behaviors learned by the team of agents, (iii) highlight the importance of information exchange between agents, and (iv) showcase the implications stemming from various network topologies on the DMARL efficiency.https://ieeexplore.ieee.org/document/10107385/Multi-agent reinforcement learningnetwork function virtualizationself-learning orchestration |
| spellingShingle | Angelos Pentelas Danny De Vleeschauwer Chia-Yu Chang Koen De Schepper Panagiotis Papadimitriou Deep Multi-Agent Reinforcement Learning With Minimal Cross-Agent Communication for SFC Partitioning IEEE Access Multi-agent reinforcement learning network function virtualization self-learning orchestration |
| title | Deep Multi-Agent Reinforcement Learning With Minimal Cross-Agent Communication for SFC Partitioning |
| title_full | Deep Multi-Agent Reinforcement Learning With Minimal Cross-Agent Communication for SFC Partitioning |
| title_fullStr | Deep Multi-Agent Reinforcement Learning With Minimal Cross-Agent Communication for SFC Partitioning |
| title_full_unstemmed | Deep Multi-Agent Reinforcement Learning With Minimal Cross-Agent Communication for SFC Partitioning |
| title_short | Deep Multi-Agent Reinforcement Learning With Minimal Cross-Agent Communication for SFC Partitioning |
| title_sort | deep multi agent reinforcement learning with minimal cross agent communication for sfc partitioning |
| topic | Multi-agent reinforcement learning network function virtualization self-learning orchestration |
| url | https://ieeexplore.ieee.org/document/10107385/ |
| work_keys_str_mv | AT angelospentelas deepmultiagentreinforcementlearningwithminimalcrossagentcommunicationforsfcpartitioning AT dannydevleeschauwer deepmultiagentreinforcementlearningwithminimalcrossagentcommunicationforsfcpartitioning AT chiayuchang deepmultiagentreinforcementlearningwithminimalcrossagentcommunicationforsfcpartitioning AT koendeschepper deepmultiagentreinforcementlearningwithminimalcrossagentcommunicationforsfcpartitioning AT panagiotispapadimitriou deepmultiagentreinforcementlearningwithminimalcrossagentcommunicationforsfcpartitioning |