Mixed Deep Reinforcement Learning Considering Discrete-continuous Hybrid Action Space for Smart Home Energy Management
This paper develops deep reinforcement learning (DRL) algorithms for optimizing the operation of home energy system which consists of photovoltaic (PV) panels, battery energy storage system, and household appliances. Model-free DRL algorithms can efficiently handle the difficulty of energy system mo...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2022-01-01
|
| Series: | Journal of Modern Power Systems and Clean Energy |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9682649/ |
| _version_ | 1818212969496969216 |
|---|---|
| author | Chao Huang Hongcai Zhang Long Wang Xiong Luo Yonghua Song |
| author_facet | Chao Huang Hongcai Zhang Long Wang Xiong Luo Yonghua Song |
| author_sort | Chao Huang |
| collection | DOAJ |
| description | This paper develops deep reinforcement learning (DRL) algorithms for optimizing the operation of home energy system which consists of photovoltaic (PV) panels, battery energy storage system, and household appliances. Model-free DRL algorithms can efficiently handle the difficulty of energy system modeling and uncertainty of PV generation. However, discrete-continuous hybrid action space of the considered home energy system challenges existing DRL algorithms for either discrete actions or continuous actions. Thus, a mixed deep reinforcement learning (MDRL) algorithm is proposed, which integrates deep Q-learning (DQL) algorithm and deep deterministic policy gradient (DDPG) algorithm. The DQL algorithm deals with discrete actions, while the DDPG algorithm handles continuous actions. The MDRL algorithm learns optimal strategy by trial-and-error interactions with the environment. However, unsafe actions, which violate system constraints, can give rise to great cost. To handle such problem, a safe-MDRL algorithm is further proposed. Simulation studies demonstrate that the proposed MDRL algorithm can efficiently handle the challenge from discrete-continuous hybrid action space for home energy management. The proposed MDRL algorithm reduces the operation cost while maintaining the human thermal comfort by comparing with benchmark algorithms on the test dataset. Moreover, the safe-MDRL algorithm greatly reduces the loss of thermal comfort in the learning stage by the proposed MDRL algorithm. |
| first_indexed | 2024-12-12T05:56:50Z |
| format | Article |
| id | doaj.art-f9f62b462c56461dab226411d7283359 |
| institution | Directory Open Access Journal |
| issn | 2196-5420 |
| language | English |
| last_indexed | 2024-12-12T05:56:50Z |
| publishDate | 2022-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | Journal of Modern Power Systems and Clean Energy |
| spelling | doaj.art-f9f62b462c56461dab226411d72833592022-12-22T00:35:32ZengIEEEJournal of Modern Power Systems and Clean Energy2196-54202022-01-0110374375410.35833/MPCE.2021.0003949682649Mixed Deep Reinforcement Learning Considering Discrete-continuous Hybrid Action Space for Smart Home Energy ManagementChao Huang0Hongcai Zhang1Long Wang2Xiong Luo3Yonghua Song4State Key Laboratory of Internet of Things for Smart City, University of Macau,Macau,S. A. R., ChinaState Key Laboratory of Internet of Things for Smart City, University of Macau,Department of Electrical and Computer Engineering,Macau,S. A. R., ChinaSchool of Computer and Communication Engineering, University of Science and Technology Beijing,Beijing,China,100083School of Computer and Communication Engineering, University of Science and Technology Beijing,Beijing,China,100083State Key Laboratory of Internet of Things for Smart City, University of Macau,Department of Electrical and Computer Engineering,Macau,S. A. R., ChinaThis paper develops deep reinforcement learning (DRL) algorithms for optimizing the operation of home energy system which consists of photovoltaic (PV) panels, battery energy storage system, and household appliances. Model-free DRL algorithms can efficiently handle the difficulty of energy system modeling and uncertainty of PV generation. However, discrete-continuous hybrid action space of the considered home energy system challenges existing DRL algorithms for either discrete actions or continuous actions. Thus, a mixed deep reinforcement learning (MDRL) algorithm is proposed, which integrates deep Q-learning (DQL) algorithm and deep deterministic policy gradient (DDPG) algorithm. The DQL algorithm deals with discrete actions, while the DDPG algorithm handles continuous actions. The MDRL algorithm learns optimal strategy by trial-and-error interactions with the environment. However, unsafe actions, which violate system constraints, can give rise to great cost. To handle such problem, a safe-MDRL algorithm is further proposed. Simulation studies demonstrate that the proposed MDRL algorithm can efficiently handle the challenge from discrete-continuous hybrid action space for home energy management. The proposed MDRL algorithm reduces the operation cost while maintaining the human thermal comfort by comparing with benchmark algorithms on the test dataset. Moreover, the safe-MDRL algorithm greatly reduces the loss of thermal comfort in the learning stage by the proposed MDRL algorithm.https://ieeexplore.ieee.org/document/9682649/Demand responsedeep reinforcement learningdiscrete-continuous action spacehome energy managementsafe reinforcement learning |
| spellingShingle | Chao Huang Hongcai Zhang Long Wang Xiong Luo Yonghua Song Mixed Deep Reinforcement Learning Considering Discrete-continuous Hybrid Action Space for Smart Home Energy Management Journal of Modern Power Systems and Clean Energy Demand response deep reinforcement learning discrete-continuous action space home energy management safe reinforcement learning |
| title | Mixed Deep Reinforcement Learning Considering Discrete-continuous Hybrid Action Space for Smart Home Energy Management |
| title_full | Mixed Deep Reinforcement Learning Considering Discrete-continuous Hybrid Action Space for Smart Home Energy Management |
| title_fullStr | Mixed Deep Reinforcement Learning Considering Discrete-continuous Hybrid Action Space for Smart Home Energy Management |
| title_full_unstemmed | Mixed Deep Reinforcement Learning Considering Discrete-continuous Hybrid Action Space for Smart Home Energy Management |
| title_short | Mixed Deep Reinforcement Learning Considering Discrete-continuous Hybrid Action Space for Smart Home Energy Management |
| title_sort | mixed deep reinforcement learning considering discrete continuous hybrid action space for smart home energy management |
| topic | Demand response deep reinforcement learning discrete-continuous action space home energy management safe reinforcement learning |
| url | https://ieeexplore.ieee.org/document/9682649/ |
| work_keys_str_mv | AT chaohuang mixeddeepreinforcementlearningconsideringdiscretecontinuoushybridactionspaceforsmarthomeenergymanagement AT hongcaizhang mixeddeepreinforcementlearningconsideringdiscretecontinuoushybridactionspaceforsmarthomeenergymanagement AT longwang mixeddeepreinforcementlearningconsideringdiscretecontinuoushybridactionspaceforsmarthomeenergymanagement AT xiongluo mixeddeepreinforcementlearningconsideringdiscretecontinuoushybridactionspaceforsmarthomeenergymanagement AT yonghuasong mixeddeepreinforcementlearningconsideringdiscretecontinuoushybridactionspaceforsmarthomeenergymanagement |