Comparison of Deep Reinforcement Learning and PID Controllers for Automatic Cold Shutdown Operation
Many industries apply traditional controllers to automate manual control. In recent years, artificial intelligence controllers applied with deep-learning techniques have been suggested as advanced controllers that can achieve goals from many industrial domains, such as humans. Deep reinforcement lea...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-04-01
|
| Series: | Energies |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1996-1073/15/8/2834 |
| _version_ | 1797434798476623872 |
|---|---|
| author | Daeil Lee Seoryong Koo Inseok Jang Jonghyun Kim |
| author_facet | Daeil Lee Seoryong Koo Inseok Jang Jonghyun Kim |
| author_sort | Daeil Lee |
| collection | DOAJ |
| description | Many industries apply traditional controllers to automate manual control. In recent years, artificial intelligence controllers applied with deep-learning techniques have been suggested as advanced controllers that can achieve goals from many industrial domains, such as humans. Deep reinforcement learning (DRL) is a powerful method for these controllers to learn how to achieve their specific operational goals. As DRL controllers learn through sampling from a target system, they can overcome the limitations of traditional controllers, such as proportional-integral-derivative (PID) controllers. In nuclear power plants (NPPs), automatic systems can manage components during full-power operation. In contrast, startup and shutdown operations are less automated and are typically performed by operators. This study suggests DRL-based and PID-based controllers for cold shutdown operations, which are a part of startup operations. By comparing the suggested controllers, this study aims to verify that learning-based controllers can overcome the limitations of traditional controllers and achieve operational goals with minimal manipulation. First, to identify the required components, operational goals, and inputs/outputs of operations, this study analyzed the general operating procedures for cold shutdown operations. Then, PID- and DRL-based controllers are designed. The PID-based controller consists of PID controllers that are well-tuned using the Ziegler–Nichols rule. The DRL-based controller with long short-term memory (LSTM) is trained with a soft actor-critic algorithm that can reduce the training time by using distributed prioritized experience replay and distributed learning. The LSTM can process a plant time-series data to generate control signals. Subsequently, the suggested controllers were validated using an NPP simulator during the cold shutdown operation. Finally, this study discusses the operational performance by comparing PID- and DRL-based controllers. |
| first_indexed | 2024-03-09T10:38:54Z |
| format | Article |
| id | doaj.art-b31ee7ea7a954fe7ab66d2a9d9f89a4b |
| institution | Directory Open Access Journal |
| issn | 1996-1073 |
| language | English |
| last_indexed | 2024-03-09T10:38:54Z |
| publishDate | 2022-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Energies |
| spelling | doaj.art-b31ee7ea7a954fe7ab66d2a9d9f89a4b2023-12-01T20:48:58ZengMDPI AGEnergies1996-10732022-04-01158283410.3390/en15082834Comparison of Deep Reinforcement Learning and PID Controllers for Automatic Cold Shutdown OperationDaeil Lee0Seoryong Koo1Inseok Jang2Jonghyun Kim3Department of Nuclear Engineering, Chosun University, Dong-gu, Gwangju 61452, KoreaKorea Atomic Energy Research Institute, Yuseong-gu, Daejeon 34057, KoreaKorea Atomic Energy Research Institute, Yuseong-gu, Daejeon 34057, KoreaDepartment of Nuclear Engineering, Chosun University, Dong-gu, Gwangju 61452, KoreaMany industries apply traditional controllers to automate manual control. In recent years, artificial intelligence controllers applied with deep-learning techniques have been suggested as advanced controllers that can achieve goals from many industrial domains, such as humans. Deep reinforcement learning (DRL) is a powerful method for these controllers to learn how to achieve their specific operational goals. As DRL controllers learn through sampling from a target system, they can overcome the limitations of traditional controllers, such as proportional-integral-derivative (PID) controllers. In nuclear power plants (NPPs), automatic systems can manage components during full-power operation. In contrast, startup and shutdown operations are less automated and are typically performed by operators. This study suggests DRL-based and PID-based controllers for cold shutdown operations, which are a part of startup operations. By comparing the suggested controllers, this study aims to verify that learning-based controllers can overcome the limitations of traditional controllers and achieve operational goals with minimal manipulation. First, to identify the required components, operational goals, and inputs/outputs of operations, this study analyzed the general operating procedures for cold shutdown operations. Then, PID- and DRL-based controllers are designed. The PID-based controller consists of PID controllers that are well-tuned using the Ziegler–Nichols rule. The DRL-based controller with long short-term memory (LSTM) is trained with a soft actor-critic algorithm that can reduce the training time by using distributed prioritized experience replay and distributed learning. The LSTM can process a plant time-series data to generate control signals. Subsequently, the suggested controllers were validated using an NPP simulator during the cold shutdown operation. Finally, this study discusses the operational performance by comparing PID- and DRL-based controllers.https://www.mdpi.com/1996-1073/15/8/2834nuclear power plantautonomous operationartificial intelligencedeep reinforcement learningsoft actor-critic algorithm |
| spellingShingle | Daeil Lee Seoryong Koo Inseok Jang Jonghyun Kim Comparison of Deep Reinforcement Learning and PID Controllers for Automatic Cold Shutdown Operation Energies nuclear power plant autonomous operation artificial intelligence deep reinforcement learning soft actor-critic algorithm |
| title | Comparison of Deep Reinforcement Learning and PID Controllers for Automatic Cold Shutdown Operation |
| title_full | Comparison of Deep Reinforcement Learning and PID Controllers for Automatic Cold Shutdown Operation |
| title_fullStr | Comparison of Deep Reinforcement Learning and PID Controllers for Automatic Cold Shutdown Operation |
| title_full_unstemmed | Comparison of Deep Reinforcement Learning and PID Controllers for Automatic Cold Shutdown Operation |
| title_short | Comparison of Deep Reinforcement Learning and PID Controllers for Automatic Cold Shutdown Operation |
| title_sort | comparison of deep reinforcement learning and pid controllers for automatic cold shutdown operation |
| topic | nuclear power plant autonomous operation artificial intelligence deep reinforcement learning soft actor-critic algorithm |
| url | https://www.mdpi.com/1996-1073/15/8/2834 |
| work_keys_str_mv | AT daeillee comparisonofdeepreinforcementlearningandpidcontrollersforautomaticcoldshutdownoperation AT seoryongkoo comparisonofdeepreinforcementlearningandpidcontrollersforautomaticcoldshutdownoperation AT inseokjang comparisonofdeepreinforcementlearningandpidcontrollersforautomaticcoldshutdownoperation AT jonghyunkim comparisonofdeepreinforcementlearningandpidcontrollersforautomaticcoldshutdownoperation |