Feedback controller parameterizations for reinforcement learning
Reinforcement Learning offers a very general framework for learning controllers, but its effectiveness is closely tied to the controller parameterization used. Especially when learning feedback controllers for weakly stable systems, ineffective parameterizations can result in unstable controllers an...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | en_US |
| Published: |
Institute of Electrical and Electronics Engineers
2011
|
| Online Access: | http://hdl.handle.net/1721.1/67496 https://orcid.org/0000-0002-8712-7092 |
| _version_ | 1826210165304590336 |
|---|---|
| author | Roberts, John William Manchester, Ian R. Tedrake, Russell Louis |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Roberts, John William Manchester, Ian R. Tedrake, Russell Louis |
| author_sort | Roberts, John William |
| collection | MIT |
| description | Reinforcement Learning offers a very general framework for learning controllers, but its effectiveness is closely tied to the controller parameterization used. Especially when learning feedback controllers for weakly stable systems, ineffective parameterizations can result in unstable controllers and poor performance both in terms of learning convergence and in the cost of the resulting policy. In this paper we explore four linear controller parameterizations in the context of REINFORCE, applying them to the control of a reaching task with a linearized flexible manipulator. We find that some natural but naive parameterizations perform very poorly, while the Youla Parameterization (a popular parameterization from the controls literature) offers a number of robustness and performance advantages. |
| first_indexed | 2024-09-23T14:44:56Z |
| format | Article |
| id | mit-1721.1/67496 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T14:44:56Z |
| publishDate | 2011 |
| publisher | Institute of Electrical and Electronics Engineers |
| record_format | dspace |
| spelling | mit-1721.1/674962022-10-01T22:14:56Z Feedback controller parameterizations for reinforcement learning Roberts, John William Manchester, Ian R. Tedrake, Russell Louis Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology. Department of Mechanical Engineering Tedrake, Russell Louis Roberts, John William Manchester, Ian R. Tedrake, Russell Louis Reinforcement Learning offers a very general framework for learning controllers, but its effectiveness is closely tied to the controller parameterization used. Especially when learning feedback controllers for weakly stable systems, ineffective parameterizations can result in unstable controllers and poor performance both in terms of learning convergence and in the cost of the resulting policy. In this paper we explore four linear controller parameterizations in the context of REINFORCE, applying them to the control of a reaching task with a linearized flexible manipulator. We find that some natural but naive parameterizations perform very poorly, while the Youla Parameterization (a popular parameterization from the controls literature) offers a number of robustness and performance advantages. National Science Foundation (U.S.) (Award IIS-0746194) 2011-12-09T19:30:35Z 2011-12-09T19:30:35Z 2011-04 2011-04 Article http://purl.org/eprint/type/ConferencePaper 978-1-4244-9887-1 http://hdl.handle.net/1721.1/67496 Roberts, John William et al. "Feedback controller parameterizations for reinforcement learning." Forthcoming in Proceedings of the 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL) https://orcid.org/0000-0002-8712-7092 en_US http://dx.doi.org/10.1109/ADPRL.2011.5967370 Proceedings of the 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL) Creative Commons Attribution-Noncommercial-Share Alike 3.0 http://creativecommons.org/licenses/by-nc-sa/3.0/ application/pdf Institute of Electrical and Electronics Engineers MIT web domain |
| spellingShingle | Roberts, John William Manchester, Ian R. Tedrake, Russell Louis Feedback controller parameterizations for reinforcement learning |
| title | Feedback controller parameterizations for reinforcement learning |
| title_full | Feedback controller parameterizations for reinforcement learning |
| title_fullStr | Feedback controller parameterizations for reinforcement learning |
| title_full_unstemmed | Feedback controller parameterizations for reinforcement learning |
| title_short | Feedback controller parameterizations for reinforcement learning |
| title_sort | feedback controller parameterizations for reinforcement learning |
| url | http://hdl.handle.net/1721.1/67496 https://orcid.org/0000-0002-8712-7092 |
| work_keys_str_mv | AT robertsjohnwilliam feedbackcontrollerparameterizationsforreinforcementlearning AT manchesterianr feedbackcontrollerparameterizationsforreinforcementlearning AT tedrakerusselllouis feedbackcontrollerparameterizationsforreinforcementlearning |