Model Predictive Control-Based Reinforcement Learning Using Expected Sarsa
Recent studies have shown the potential of Reinforcement Learning (RL) algorithms in tuning the parameters of Model Predictive Controllers (MPC), including the weights of the cost function and unknown parameters of the MPC model. However, a framework for easy and straightforward implementation that...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2022-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9846979/ |
| _version_ | 1811342517888614400 |
|---|---|
| author | Hoomaan Moradimaryamnegari Marco Frego Angelika Peer |
| author_facet | Hoomaan Moradimaryamnegari Marco Frego Angelika Peer |
| author_sort | Hoomaan Moradimaryamnegari |
| collection | DOAJ |
| description | Recent studies have shown the potential of Reinforcement Learning (RL) algorithms in tuning the parameters of Model Predictive Controllers (MPC), including the weights of the cost function and unknown parameters of the MPC model. However, a framework for easy and straightforward implementation that allows training in just a few episodes and overcoming the need for imposing extra constraints as required by state-of-the-art methods, is still missing. In this study, we present two implementations to achieve these goals. In the first approach, a nonlinear MPC plays the role of a function approximator for an Expected Sarsa RL algorithm. In the second approach, only the MPC cost function is considered as the function approximator, while the unknown parameters of the MPC model are updated based on more classical system identification. In order to evaluate the performance of the proposed algorithms, first numerical simulations are performed on a coupled tanks system. Then, both algorithms are applied to the real system and their closed-loop performance and convergence speed are compared with each other. The results indicate that the proposed algorithms allow tuning of MPCs over very few episodes. Finally, also the disturbance rejection ability of the proposed methods is demonstrated. |
| first_indexed | 2024-04-13T19:12:25Z |
| format | Article |
| id | doaj.art-80328c61febd45e0a03c6025a35763c6 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-13T19:12:25Z |
| publishDate | 2022-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-80328c61febd45e0a03c6025a35763c62022-12-22T02:33:46ZengIEEEIEEE Access2169-35362022-01-0110811778119110.1109/ACCESS.2022.31955309846979Model Predictive Control-Based Reinforcement Learning Using Expected SarsaHoomaan Moradimaryamnegari0https://orcid.org/0000-0002-0858-0420Marco Frego1https://orcid.org/0000-0003-2855-9052Angelika Peer2https://orcid.org/0000-0002-2896-9011Human-Centered Technologies and Machine Intelligence Laboratory, Faculty of Science and Technology, Free University of Bozen-Bolzano, Bolzano, ItalyHuman-Centered Technologies and Machine Intelligence Laboratory, Faculty of Science and Technology, Free University of Bozen-Bolzano, Bolzano, ItalyHuman-Centered Technologies and Machine Intelligence Laboratory, Faculty of Science and Technology, Free University of Bozen-Bolzano, Bolzano, ItalyRecent studies have shown the potential of Reinforcement Learning (RL) algorithms in tuning the parameters of Model Predictive Controllers (MPC), including the weights of the cost function and unknown parameters of the MPC model. However, a framework for easy and straightforward implementation that allows training in just a few episodes and overcoming the need for imposing extra constraints as required by state-of-the-art methods, is still missing. In this study, we present two implementations to achieve these goals. In the first approach, a nonlinear MPC plays the role of a function approximator for an Expected Sarsa RL algorithm. In the second approach, only the MPC cost function is considered as the function approximator, while the unknown parameters of the MPC model are updated based on more classical system identification. In order to evaluate the performance of the proposed algorithms, first numerical simulations are performed on a coupled tanks system. Then, both algorithms are applied to the real system and their closed-loop performance and convergence speed are compared with each other. The results indicate that the proposed algorithms allow tuning of MPCs over very few episodes. Finally, also the disturbance rejection ability of the proposed methods is demonstrated.https://ieeexplore.ieee.org/document/9846979/Model predictive controlreinforcement learningexpected Sarsa algorithmmodel-based learning method |
| spellingShingle | Hoomaan Moradimaryamnegari Marco Frego Angelika Peer Model Predictive Control-Based Reinforcement Learning Using Expected Sarsa IEEE Access Model predictive control reinforcement learning expected Sarsa algorithm model-based learning method |
| title | Model Predictive Control-Based Reinforcement Learning Using Expected Sarsa |
| title_full | Model Predictive Control-Based Reinforcement Learning Using Expected Sarsa |
| title_fullStr | Model Predictive Control-Based Reinforcement Learning Using Expected Sarsa |
| title_full_unstemmed | Model Predictive Control-Based Reinforcement Learning Using Expected Sarsa |
| title_short | Model Predictive Control-Based Reinforcement Learning Using Expected Sarsa |
| title_sort | model predictive control based reinforcement learning using expected sarsa |
| topic | Model predictive control reinforcement learning expected Sarsa algorithm model-based learning method |
| url | https://ieeexplore.ieee.org/document/9846979/ |
| work_keys_str_mv | AT hoomaanmoradimaryamnegari modelpredictivecontrolbasedreinforcementlearningusingexpectedsarsa AT marcofrego modelpredictivecontrolbasedreinforcementlearningusingexpectedsarsa AT angelikapeer modelpredictivecontrolbasedreinforcementlearningusingexpectedsarsa |