A prior knowledge-embedded reinforcement learning method for real-time active power corrective control in complex power systems
With the increasing uncertainty and complexity of modern power grids, the real-time active power corrective control problem becomes intractable, bringing significant challenges to the stable operation of future power systems. To promote effective and efficient active power corrective control, a prio...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2022-09-01
|
| Series: | Frontiers in Energy Research |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fenrg.2022.1009545/full |
| _version_ | 1811206609191305216 |
|---|---|
| author | Peidong Xu Jun Zhang Jixiang Lu Haoran Zhang Tianlu Gao Siyuan Chen |
| author_facet | Peidong Xu Jun Zhang Jixiang Lu Haoran Zhang Tianlu Gao Siyuan Chen |
| author_sort | Peidong Xu |
| collection | DOAJ |
| description | With the increasing uncertainty and complexity of modern power grids, the real-time active power corrective control problem becomes intractable, bringing significant challenges to the stable operation of future power systems. To promote effective and efficient active power corrective control, a prior knowledge-embedded reinforcement learning method is proposed in this paper, to improve the performance of the deep reinforcement learning agent while maintaining the real-time control manner. The system-level feature is first established based on prior knowledge and cooperating with the equipment-level features, to provide a thorough description of the power network states. A global-local network structure is then constructed to integrate the two-level information accordingly by introducing the graph pooling method. Based on the multi-level representation of power system states, the Deep Q-learning from Demonstrations method is adopted to guide the deep reinforcement learning agent to learn from the expert policy along with the interactive improving process. Considering the infrequent corrective control actions in practice, the double-prioritized training mechanism combined with the λ-return is further developed to help the agent lay emphasis on learning from critical control experience. Simulation results demonstrate that the proposed method prevails over the conventional deep reinforcement learning methods in training efficiency and control effects, and has the potential to solve the complex active power corrective control problem in the future. |
| first_indexed | 2024-04-12T03:50:06Z |
| format | Article |
| id | doaj.art-be6ae9b5ea794af9a12f65de1a709ed9 |
| institution | Directory Open Access Journal |
| issn | 2296-598X |
| language | English |
| last_indexed | 2024-04-12T03:50:06Z |
| publishDate | 2022-09-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Energy Research |
| spelling | doaj.art-be6ae9b5ea794af9a12f65de1a709ed92022-12-22T03:49:01ZengFrontiers Media S.A.Frontiers in Energy Research2296-598X2022-09-011010.3389/fenrg.2022.10095451009545A prior knowledge-embedded reinforcement learning method for real-time active power corrective control in complex power systemsPeidong Xu0Jun Zhang1Jixiang Lu2Haoran Zhang3Tianlu Gao4Siyuan Chen5School of Electrical Engineering and Automation, Wuhan University, Wuhan, ChinaSchool of Electrical Engineering and Automation, Wuhan University, Wuhan, ChinaTechnology Research Center, State Key Laboratory of Intelligent Power Grid Protection and Operation Control, NARI Group Corporation, Nanjing, ChinaSchool of Electrical Engineering and Automation, Wuhan University, Wuhan, ChinaSchool of Electrical Engineering and Automation, Wuhan University, Wuhan, ChinaSchool of Electrical Engineering and Automation, Wuhan University, Wuhan, ChinaWith the increasing uncertainty and complexity of modern power grids, the real-time active power corrective control problem becomes intractable, bringing significant challenges to the stable operation of future power systems. To promote effective and efficient active power corrective control, a prior knowledge-embedded reinforcement learning method is proposed in this paper, to improve the performance of the deep reinforcement learning agent while maintaining the real-time control manner. The system-level feature is first established based on prior knowledge and cooperating with the equipment-level features, to provide a thorough description of the power network states. A global-local network structure is then constructed to integrate the two-level information accordingly by introducing the graph pooling method. Based on the multi-level representation of power system states, the Deep Q-learning from Demonstrations method is adopted to guide the deep reinforcement learning agent to learn from the expert policy along with the interactive improving process. Considering the infrequent corrective control actions in practice, the double-prioritized training mechanism combined with the λ-return is further developed to help the agent lay emphasis on learning from critical control experience. Simulation results demonstrate that the proposed method prevails over the conventional deep reinforcement learning methods in training efficiency and control effects, and has the potential to solve the complex active power corrective control problem in the future.https://www.frontiersin.org/articles/10.3389/fenrg.2022.1009545/fullactive power corrective controldeep Q-learning from demonstrationsgraph poolingpower systemprior knowledge |
| spellingShingle | Peidong Xu Jun Zhang Jixiang Lu Haoran Zhang Tianlu Gao Siyuan Chen A prior knowledge-embedded reinforcement learning method for real-time active power corrective control in complex power systems Frontiers in Energy Research active power corrective control deep Q-learning from demonstrations graph pooling power system prior knowledge |
| title | A prior knowledge-embedded reinforcement learning method for real-time active power corrective control in complex power systems |
| title_full | A prior knowledge-embedded reinforcement learning method for real-time active power corrective control in complex power systems |
| title_fullStr | A prior knowledge-embedded reinforcement learning method for real-time active power corrective control in complex power systems |
| title_full_unstemmed | A prior knowledge-embedded reinforcement learning method for real-time active power corrective control in complex power systems |
| title_short | A prior knowledge-embedded reinforcement learning method for real-time active power corrective control in complex power systems |
| title_sort | prior knowledge embedded reinforcement learning method for real time active power corrective control in complex power systems |
| topic | active power corrective control deep Q-learning from demonstrations graph pooling power system prior knowledge |
| url | https://www.frontiersin.org/articles/10.3389/fenrg.2022.1009545/full |
| work_keys_str_mv | AT peidongxu apriorknowledgeembeddedreinforcementlearningmethodforrealtimeactivepowercorrectivecontrolincomplexpowersystems AT junzhang apriorknowledgeembeddedreinforcementlearningmethodforrealtimeactivepowercorrectivecontrolincomplexpowersystems AT jixianglu apriorknowledgeembeddedreinforcementlearningmethodforrealtimeactivepowercorrectivecontrolincomplexpowersystems AT haoranzhang apriorknowledgeembeddedreinforcementlearningmethodforrealtimeactivepowercorrectivecontrolincomplexpowersystems AT tianlugao apriorknowledgeembeddedreinforcementlearningmethodforrealtimeactivepowercorrectivecontrolincomplexpowersystems AT siyuanchen apriorknowledgeembeddedreinforcementlearningmethodforrealtimeactivepowercorrectivecontrolincomplexpowersystems AT peidongxu priorknowledgeembeddedreinforcementlearningmethodforrealtimeactivepowercorrectivecontrolincomplexpowersystems AT junzhang priorknowledgeembeddedreinforcementlearningmethodforrealtimeactivepowercorrectivecontrolincomplexpowersystems AT jixianglu priorknowledgeembeddedreinforcementlearningmethodforrealtimeactivepowercorrectivecontrolincomplexpowersystems AT haoranzhang priorknowledgeembeddedreinforcementlearningmethodforrealtimeactivepowercorrectivecontrolincomplexpowersystems AT tianlugao priorknowledgeembeddedreinforcementlearningmethodforrealtimeactivepowercorrectivecontrolincomplexpowersystems AT siyuanchen priorknowledgeembeddedreinforcementlearningmethodforrealtimeactivepowercorrectivecontrolincomplexpowersystems |