Multi-agent reinforcement learning for the coordination of residential energy flexibility
<p>This thesis investigates whether residential energy flexibility can be coordinated without sharing personal data at scale to achieve a positive impact on energy users and the grid.</p> <p>To tackle climate change, energy uses are being electrified at pace, just as electricity i...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Language: | English |
| Published: |
2023
|
| Subjects: |
| _version_ | 1811139807963774976 |
|---|---|
| author | Charbonnier, F |
| author2 | Morstyn, T |
| author_facet | Morstyn, T Charbonnier, F |
| author_sort | Charbonnier, F |
| collection | OXFORD |
| description | <p>This thesis investigates whether residential energy flexibility can be coordinated without sharing personal data at scale to achieve a positive impact on energy users and the grid.</p>
<p>To tackle climate change, energy uses are being electrified at pace, just as electricity is increasingly provided by non-dispatchable renewable energy sources. These shifts increase the requirements for demand-side flexibility. Despite the potential of residential energy to provide such flexibility, it has remained largely untapped due to cost, social acceptance, and technical barriers. This thesis investigates the use of multi-agent reinforcement learning to overcome these challenges.</p>
<p>This thesis presents a novel testing environment, which models electric vehicles, space heating, and flexible household loads in a distribution network. Additionally, a generative adversarial network-based data generator is developed to obtain realistic training and testing data. Experiments conducted in this environment showed that standard independent learners fail to coordinate in the partially observable stochastic environment. To address this, additional coordination mechanisms are proposed for agents to practise coordination in a centralised simulated rehearsal, ahead of fully decentralised implementation.</p>
<p>Two such coordination mechanisms are proposed: optimisation-informed independent learning, and a centralised but factored critic network. In the former, agents lean from omniscient convex optimisation results ahead of fully decentralised coordination. This enables cooperation at scale where standard independent learners under partial observability could not be coordinated. In the latter, agents employ a deep neural factorisation network to learn to assess their impact on global rewards. This approach delivers comparable performance for four agents and more, with a 34-fold speed improvement for 30 agents and only first-order computational time growth.</p>
<p>Finally, the impacts of implementing implicit coordination using these multi- agent reinforcement learning methodologies are modelled. It is observed that even without explicit grid constraint management, cooperating energy users reduce the likelihood of voltage deviations. The cooperative management of voltage constraints could be further promoted by the MARL policies, whereby their likelihood could be reduced by 43.08% relative to an uncoordinated baseline, albeit with trade-offs in other costs. However, while this thesis demonstrates the technical feasibility of MARL-based cooperation, further market mechanisms are required to reward all participants for their cooperation.</p> |
| first_indexed | 2024-09-25T04:11:58Z |
| format | Thesis |
| id | oxford-uuid:4acf19ee-76b3-4e71-a424-259cd9e2d6f5 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-09-25T04:11:58Z |
| publishDate | 2023 |
| record_format | dspace |
| spelling | oxford-uuid:4acf19ee-76b3-4e71-a424-259cd9e2d6f52024-07-01T11:07:53ZMulti-agent reinforcement learning for the coordination of residential energy flexibilityThesishttp://purl.org/coar/resource_type/c_db06uuid:4acf19ee-76b3-4e71-a424-259cd9e2d6f5ElectrificationReinforcement learningMultiagent systemsDemand-side management (Electric utilities)EnglishHyrax Deposit2023Charbonnier, FMorstyn, TMcCulloch, M<p>This thesis investigates whether residential energy flexibility can be coordinated without sharing personal data at scale to achieve a positive impact on energy users and the grid.</p> <p>To tackle climate change, energy uses are being electrified at pace, just as electricity is increasingly provided by non-dispatchable renewable energy sources. These shifts increase the requirements for demand-side flexibility. Despite the potential of residential energy to provide such flexibility, it has remained largely untapped due to cost, social acceptance, and technical barriers. This thesis investigates the use of multi-agent reinforcement learning to overcome these challenges.</p> <p>This thesis presents a novel testing environment, which models electric vehicles, space heating, and flexible household loads in a distribution network. Additionally, a generative adversarial network-based data generator is developed to obtain realistic training and testing data. Experiments conducted in this environment showed that standard independent learners fail to coordinate in the partially observable stochastic environment. To address this, additional coordination mechanisms are proposed for agents to practise coordination in a centralised simulated rehearsal, ahead of fully decentralised implementation.</p> <p>Two such coordination mechanisms are proposed: optimisation-informed independent learning, and a centralised but factored critic network. In the former, agents lean from omniscient convex optimisation results ahead of fully decentralised coordination. This enables cooperation at scale where standard independent learners under partial observability could not be coordinated. In the latter, agents employ a deep neural factorisation network to learn to assess their impact on global rewards. This approach delivers comparable performance for four agents and more, with a 34-fold speed improvement for 30 agents and only first-order computational time growth.</p> <p>Finally, the impacts of implementing implicit coordination using these multi- agent reinforcement learning methodologies are modelled. It is observed that even without explicit grid constraint management, cooperating energy users reduce the likelihood of voltage deviations. The cooperative management of voltage constraints could be further promoted by the MARL policies, whereby their likelihood could be reduced by 43.08% relative to an uncoordinated baseline, albeit with trade-offs in other costs. However, while this thesis demonstrates the technical feasibility of MARL-based cooperation, further market mechanisms are required to reward all participants for their cooperation.</p> |
| spellingShingle | Electrification Reinforcement learning Multiagent systems Demand-side management (Electric utilities) Charbonnier, F Multi-agent reinforcement learning for the coordination of residential energy flexibility |
| title | Multi-agent reinforcement learning for the coordination of residential energy flexibility |
| title_full | Multi-agent reinforcement learning for the coordination of residential energy flexibility |
| title_fullStr | Multi-agent reinforcement learning for the coordination of residential energy flexibility |
| title_full_unstemmed | Multi-agent reinforcement learning for the coordination of residential energy flexibility |
| title_short | Multi-agent reinforcement learning for the coordination of residential energy flexibility |
| title_sort | multi agent reinforcement learning for the coordination of residential energy flexibility |
| topic | Electrification Reinforcement learning Multiagent systems Demand-side management (Electric utilities) |
| work_keys_str_mv | AT charbonnierf multiagentreinforcementlearningforthecoordinationofresidentialenergyflexibility |