Safe reinforcement learning for dynamical systems using barrier certificates
Safety control is a fundamental problem in policy design. Basic reinforcement learning is effective at learning policy with goal-reaching property. However, it does not guarantee safety property of the learned policy. This paper integrates barrier certificates into actor-critic-based reinforcement l...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Taylor & Francis Group
2022-12-01
|
| Series: | Connection Science |
| Subjects: | |
| Online Access: | http://dx.doi.org/10.1080/09540091.2022.2151567 |
| _version_ | 1797683979174805504 |
|---|---|
| author | Qingye Zhao Yi Zhang Xuandong Li |
| author_facet | Qingye Zhao Yi Zhang Xuandong Li |
| author_sort | Qingye Zhao |
| collection | DOAJ |
| description | Safety control is a fundamental problem in policy design. Basic reinforcement learning is effective at learning policy with goal-reaching property. However, it does not guarantee safety property of the learned policy. This paper integrates barrier certificates into actor-critic-based reinforcement learning methods in a feedback-driven framework to learn safe policies for dynamical systems. The safe reinforcement learning framework is composed of two interactive parts: Learner and Verifier. Learner trains the policy to satisfy goal-reaching and safety properties. Since the policy is trained on training datasets, the two properties may not be retained on the whole system. Verifier validates the learned policy on the whole system. If the validation fails, Verifier returns the counterexamples to Learner for retraining the policy in the next iteration. We implement a safe policy learning tool SRLBC and evaluate its performance on three control tasks. Experimental results show that SRLBC achieves safety with no more than 0.5× time overhead compared to the baseline reinforcement learning method, showing the feasibility and effectiveness of our framework. |
| first_indexed | 2024-03-12T00:23:42Z |
| format | Article |
| id | doaj.art-04bba19aafa14115b823278f19a094fd |
| institution | Directory Open Access Journal |
| issn | 0954-0091 1360-0494 |
| language | English |
| last_indexed | 2024-03-12T00:23:42Z |
| publishDate | 2022-12-01 |
| publisher | Taylor & Francis Group |
| record_format | Article |
| series | Connection Science |
| spelling | doaj.art-04bba19aafa14115b823278f19a094fd2023-09-15T10:48:01ZengTaylor & Francis GroupConnection Science0954-00911360-04942022-12-013412822284410.1080/09540091.2022.21515672151567Safe reinforcement learning for dynamical systems using barrier certificatesQingye Zhao0Yi Zhang1Xuandong Li2State Key Laboratory for Novel Software Technology, Nanjing UniversityState Key Laboratory for Novel Software Technology, Nanjing UniversityState Key Laboratory for Novel Software Technology, Nanjing UniversitySafety control is a fundamental problem in policy design. Basic reinforcement learning is effective at learning policy with goal-reaching property. However, it does not guarantee safety property of the learned policy. This paper integrates barrier certificates into actor-critic-based reinforcement learning methods in a feedback-driven framework to learn safe policies for dynamical systems. The safe reinforcement learning framework is composed of two interactive parts: Learner and Verifier. Learner trains the policy to satisfy goal-reaching and safety properties. Since the policy is trained on training datasets, the two properties may not be retained on the whole system. Verifier validates the learned policy on the whole system. If the validation fails, Verifier returns the counterexamples to Learner for retraining the policy in the next iteration. We implement a safe policy learning tool SRLBC and evaluate its performance on three control tasks. Experimental results show that SRLBC achieves safety with no more than 0.5× time overhead compared to the baseline reinforcement learning method, showing the feasibility and effectiveness of our framework.http://dx.doi.org/10.1080/09540091.2022.2151567safe reinforcement learningsafety controldynamical systemsbarrier certificatesneural networks |
| spellingShingle | Qingye Zhao Yi Zhang Xuandong Li Safe reinforcement learning for dynamical systems using barrier certificates Connection Science safe reinforcement learning safety control dynamical systems barrier certificates neural networks |
| title | Safe reinforcement learning for dynamical systems using barrier certificates |
| title_full | Safe reinforcement learning for dynamical systems using barrier certificates |
| title_fullStr | Safe reinforcement learning for dynamical systems using barrier certificates |
| title_full_unstemmed | Safe reinforcement learning for dynamical systems using barrier certificates |
| title_short | Safe reinforcement learning for dynamical systems using barrier certificates |
| title_sort | safe reinforcement learning for dynamical systems using barrier certificates |
| topic | safe reinforcement learning safety control dynamical systems barrier certificates neural networks |
| url | http://dx.doi.org/10.1080/09540091.2022.2151567 |
| work_keys_str_mv | AT qingyezhao safereinforcementlearningfordynamicalsystemsusingbarriercertificates AT yizhang safereinforcementlearningfordynamicalsystemsusingbarriercertificates AT xuandongli safereinforcementlearningfordynamicalsystemsusingbarriercertificates |