Evaluation of Offline Reinforcement Learning for Blood Glucose Level Control in Type 1 Diabetes
Patients with Type 1 diabetes must closely monitor their blood glucose levels and inject insulin to control them. Automated glucose control methods that remove the need for human intervention have been proposed, and reinforcement learning has been used recently as an effective control method in simu...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10261006/ |
| _version_ | 1827801324452839424 |
|---|---|
| author | Phuwadol Viroonluecha Esteban Egea-Lopez Jose Santa |
| author_facet | Phuwadol Viroonluecha Esteban Egea-Lopez Jose Santa |
| author_sort | Phuwadol Viroonluecha |
| collection | DOAJ |
| description | Patients with Type 1 diabetes must closely monitor their blood glucose levels and inject insulin to control them. Automated glucose control methods that remove the need for human intervention have been proposed, and reinforcement learning has been used recently as an effective control method in simulation environments. However, its real-world application would require trial and error interaction with patients. As an alternative, offline reinforcement learning does not require interaction with humans and initial studies suggest promising results can be obtained with offline datasets, similar to classical machine learning algorithms. However, its application to glucose control has not yet been evaluated. In this study, we evaluated two offline reinforcement learning algorithms for blood glucose control and discussed their potential and shortcomings. We also evaluated the influence on training and performance of the method that generates the training datasets, as well as the influence of the type of trajectories used (single-method or mixed trajectories), the quality of the trajectories, and the size of the datasets. Our results show that one of the offline reinforcement learning algorithms evaluated, Trajectory Transformer, is able to perform at the same level as commonly used baselines such as PID and Proximal Policy Optimization. |
| first_indexed | 2024-03-11T20:23:42Z |
| format | Article |
| id | doaj.art-d55d6057c6da47b08d1aebe25a4680d9 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-03-11T20:23:42Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-d55d6057c6da47b08d1aebe25a4680d92023-10-02T23:00:39ZengIEEEIEEE Access2169-35362023-01-011110464310465510.1109/ACCESS.2023.331832410261006Evaluation of Offline Reinforcement Learning for Blood Glucose Level Control in Type 1 DiabetesPhuwadol Viroonluecha0https://orcid.org/0000-0001-8926-291XEsteban Egea-Lopez1https://orcid.org/0000-0002-6926-4923Jose Santa2https://orcid.org/0000-0001-9224-7112Department of Information and Communications Technologies, Universidad Politécnica de Cartagena (UPCT), Cartagena, SpainDepartment of Information and Communications Technologies, Universidad Politécnica de Cartagena (UPCT), Cartagena, SpainDepartment of Electronics, Computer Technology and Projects, Universidad Politécnica de Cartagena (UPCT), Cartagena, SpainPatients with Type 1 diabetes must closely monitor their blood glucose levels and inject insulin to control them. Automated glucose control methods that remove the need for human intervention have been proposed, and reinforcement learning has been used recently as an effective control method in simulation environments. However, its real-world application would require trial and error interaction with patients. As an alternative, offline reinforcement learning does not require interaction with humans and initial studies suggest promising results can be obtained with offline datasets, similar to classical machine learning algorithms. However, its application to glucose control has not yet been evaluated. In this study, we evaluated two offline reinforcement learning algorithms for blood glucose control and discussed their potential and shortcomings. We also evaluated the influence on training and performance of the method that generates the training datasets, as well as the influence of the type of trajectories used (single-method or mixed trajectories), the quality of the trajectories, and the size of the datasets. Our results show that one of the offline reinforcement learning algorithms evaluated, Trajectory Transformer, is able to perform at the same level as commonly used baselines such as PID and Proximal Policy Optimization.https://ieeexplore.ieee.org/document/10261006/T1D blood glucose controloffline reinforcement learningtransformerartificial pancreasmachine learning |
| spellingShingle | Phuwadol Viroonluecha Esteban Egea-Lopez Jose Santa Evaluation of Offline Reinforcement Learning for Blood Glucose Level Control in Type 1 Diabetes IEEE Access T1D blood glucose control offline reinforcement learning transformer artificial pancreas machine learning |
| title | Evaluation of Offline Reinforcement Learning for Blood Glucose Level Control in Type 1 Diabetes |
| title_full | Evaluation of Offline Reinforcement Learning for Blood Glucose Level Control in Type 1 Diabetes |
| title_fullStr | Evaluation of Offline Reinforcement Learning for Blood Glucose Level Control in Type 1 Diabetes |
| title_full_unstemmed | Evaluation of Offline Reinforcement Learning for Blood Glucose Level Control in Type 1 Diabetes |
| title_short | Evaluation of Offline Reinforcement Learning for Blood Glucose Level Control in Type 1 Diabetes |
| title_sort | evaluation of offline reinforcement learning for blood glucose level control in type 1 diabetes |
| topic | T1D blood glucose control offline reinforcement learning transformer artificial pancreas machine learning |
| url | https://ieeexplore.ieee.org/document/10261006/ |
| work_keys_str_mv | AT phuwadolviroonluecha evaluationofofflinereinforcementlearningforbloodglucoselevelcontrolintype1diabetes AT estebanegealopez evaluationofofflinereinforcementlearningforbloodglucoselevelcontrolintype1diabetes AT josesanta evaluationofofflinereinforcementlearningforbloodglucoselevelcontrolintype1diabetes |