An Optimization Method-Assisted Ensemble Deep Reinforcement Learning Algorithm to Solve Unit Commitment Problems
Unit commitment (UC) is a fundamental problem in the day-ahead electricity market, and it is critical to solve UC problems efficiently. Mathematical optimization techniques like dynamic programming, Lagrangian relaxation, and mixed-integer quadratic programming (MIQP) are commonly adopted for UC pro...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10247003/ |
| _version_ | 1797680919605149696 |
|---|---|
| author | Jingtao Qin Yuanqi Gao Mikhail Bragin Nanpeng Yu |
| author_facet | Jingtao Qin Yuanqi Gao Mikhail Bragin Nanpeng Yu |
| author_sort | Jingtao Qin |
| collection | DOAJ |
| description | Unit commitment (UC) is a fundamental problem in the day-ahead electricity market, and it is critical to solve UC problems efficiently. Mathematical optimization techniques like dynamic programming, Lagrangian relaxation, and mixed-integer quadratic programming (MIQP) are commonly adopted for UC problems. However, the calculation time of these methods increases at an exponential rate with the number of generators and energy resources, which is still the main bottleneck in the industry. Recent advances in artificial intelligence have demonstrated the capability of reinforcement learning (RL) to solve UC problems. Unfortunately, the existing research on solving UC problems with RL suffers from the curse of dimensionality when the size of UC problems grows. To deal with these problems, we propose an optimization method-assisted ensemble deep reinforcement learning algorithm, where UC problems are formulated as a Markov Decision Process (MDP) and solved by multi-step deep Q-learning in an ensemble framework. The proposed algorithm establishes a candidate action set by solving tailored optimization problems to ensure relatively high performance and the satisfaction of operational constraints. Numerical studies on three test systems show that our algorithm outperforms the baseline RL algorithm in terms of computation efficiency and operation cost. By employing the output of our proposed algorithm as a warm start, the MIQP technique can achieve further reductions in operational costs. Furthermore, the proposed algorithm shows strong generalization capacity under unforeseen operational conditions. |
| first_indexed | 2024-03-11T23:37:15Z |
| format | Article |
| id | doaj.art-bdfc628df8994f6e8a86a6cff9011120 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-03-11T23:37:15Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-bdfc628df8994f6e8a86a6cff90111202023-09-19T23:01:00ZengIEEEIEEE Access2169-35362023-01-011110012510013610.1109/ACCESS.2023.331399810247003An Optimization Method-Assisted Ensemble Deep Reinforcement Learning Algorithm to Solve Unit Commitment ProblemsJingtao Qin0https://orcid.org/0000-0002-7645-7547Yuanqi Gao1https://orcid.org/0000-0003-4078-6143Mikhail Bragin2https://orcid.org/0000-0002-7783-9053Nanpeng Yu3https://orcid.org/0000-0001-5086-5465Department of Electrical and Computer Engineering, University of California, Riverside, Riverside, CA, USADepartment of Electrical and Computer Engineering, University of California, Riverside, Riverside, CA, USADepartment of Electrical and Computer Engineering, University of California, Riverside, Riverside, CA, USADepartment of Electrical and Computer Engineering, University of California, Riverside, Riverside, CA, USAUnit commitment (UC) is a fundamental problem in the day-ahead electricity market, and it is critical to solve UC problems efficiently. Mathematical optimization techniques like dynamic programming, Lagrangian relaxation, and mixed-integer quadratic programming (MIQP) are commonly adopted for UC problems. However, the calculation time of these methods increases at an exponential rate with the number of generators and energy resources, which is still the main bottleneck in the industry. Recent advances in artificial intelligence have demonstrated the capability of reinforcement learning (RL) to solve UC problems. Unfortunately, the existing research on solving UC problems with RL suffers from the curse of dimensionality when the size of UC problems grows. To deal with these problems, we propose an optimization method-assisted ensemble deep reinforcement learning algorithm, where UC problems are formulated as a Markov Decision Process (MDP) and solved by multi-step deep Q-learning in an ensemble framework. The proposed algorithm establishes a candidate action set by solving tailored optimization problems to ensure relatively high performance and the satisfaction of operational constraints. Numerical studies on three test systems show that our algorithm outperforms the baseline RL algorithm in terms of computation efficiency and operation cost. By employing the output of our proposed algorithm as a warm start, the MIQP technique can achieve further reductions in operational costs. Furthermore, the proposed algorithm shows strong generalization capacity under unforeseen operational conditions.https://ieeexplore.ieee.org/document/10247003/Deep reinforcement learningmulti-step returnoptimization methodsunit commitment |
| spellingShingle | Jingtao Qin Yuanqi Gao Mikhail Bragin Nanpeng Yu An Optimization Method-Assisted Ensemble Deep Reinforcement Learning Algorithm to Solve Unit Commitment Problems IEEE Access Deep reinforcement learning multi-step return optimization methods unit commitment |
| title | An Optimization Method-Assisted Ensemble Deep Reinforcement Learning Algorithm to Solve Unit Commitment Problems |
| title_full | An Optimization Method-Assisted Ensemble Deep Reinforcement Learning Algorithm to Solve Unit Commitment Problems |
| title_fullStr | An Optimization Method-Assisted Ensemble Deep Reinforcement Learning Algorithm to Solve Unit Commitment Problems |
| title_full_unstemmed | An Optimization Method-Assisted Ensemble Deep Reinforcement Learning Algorithm to Solve Unit Commitment Problems |
| title_short | An Optimization Method-Assisted Ensemble Deep Reinforcement Learning Algorithm to Solve Unit Commitment Problems |
| title_sort | optimization method assisted ensemble deep reinforcement learning algorithm to solve unit commitment problems |
| topic | Deep reinforcement learning multi-step return optimization methods unit commitment |
| url | https://ieeexplore.ieee.org/document/10247003/ |
| work_keys_str_mv | AT jingtaoqin anoptimizationmethodassistedensembledeepreinforcementlearningalgorithmtosolveunitcommitmentproblems AT yuanqigao anoptimizationmethodassistedensembledeepreinforcementlearningalgorithmtosolveunitcommitmentproblems AT mikhailbragin anoptimizationmethodassistedensembledeepreinforcementlearningalgorithmtosolveunitcommitmentproblems AT nanpengyu anoptimizationmethodassistedensembledeepreinforcementlearningalgorithmtosolveunitcommitmentproblems AT jingtaoqin optimizationmethodassistedensembledeepreinforcementlearningalgorithmtosolveunitcommitmentproblems AT yuanqigao optimizationmethodassistedensembledeepreinforcementlearningalgorithmtosolveunitcommitmentproblems AT mikhailbragin optimizationmethodassistedensembledeepreinforcementlearningalgorithmtosolveunitcommitmentproblems AT nanpengyu optimizationmethodassistedensembledeepreinforcementlearningalgorithmtosolveunitcommitmentproblems |