A geometry and dose-volume based performance monitoring of artificial intelligence models in radiotherapy treatment planning for prostate cancer
Background and Purpose: Clinical Artificial Intelligence (AI) implementations lack ground-truth when applied on real-world data. This study investigated how combined geometrical and dose-volume metrics can be used as performance monitoring tools to detect clinically relevant candidates for model ret...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2023-10-01
|
| Series: | Physics and Imaging in Radiation Oncology |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2405631623000854 |
| _version_ | 1797397583204712448 |
|---|---|
| author | Geert De Kerf Michaël Claessens Fadoua Raouassi Carole Mercier Daan Stas Piet Ost Piet Dirix Dirk Verellen |
| author_facet | Geert De Kerf Michaël Claessens Fadoua Raouassi Carole Mercier Daan Stas Piet Ost Piet Dirix Dirk Verellen |
| author_sort | Geert De Kerf |
| collection | DOAJ |
| description | Background and Purpose: Clinical Artificial Intelligence (AI) implementations lack ground-truth when applied on real-world data. This study investigated how combined geometrical and dose-volume metrics can be used as performance monitoring tools to detect clinically relevant candidates for model retraining. Materials and Methods: Fifty patients were analyzed for both AI-segmentation and planning. For AI-segmentation, geometrical (Standard Surface Dice 3 mm and Local Surface Dice 3 mm) and dose-volume based parameters were calculated for two organs (bladder and anorectum) to compare AI output against the clinically corrected structure. A Local Surface Dice was introduced to detect geometrical changes in the vicinity of the target volumes, while an Absolute Dose Difference (ADD) evaluation increased focus on dose-volume related changes. AI-planning performance was evaluated using clinical goal analysis in combination with volume and target overlap metrics. Results: The Local Surface Dice reported equal or lower values compared to the Standard Surface Dice (anorectum: (0.93 ± 0.11) vs (0.98 ± 0.04); bladder: (0.97 ± 0.06) vs (0.98 ± 0.04)). The ADD metric showed a difference of (0.9 ± 0.8)Gy for the anorectum D1cm3. The bladder D5cm3 reported a difference of (0.7 ± 1.5)Gy. Mandatory clinical goals were fulfilled in 90 % of the DLP plans. Conclusions: Combining dose-volume and geometrical metrics allowed detection of clinically relevant changes, applied to both auto-segmentation and auto-planning output and the Local Surface Dice was more sensitive to local changes compared to the Standard Surface Dice. This monitoring is able to evaluate AI behavior in clinical practice and allows candidate selection for active learning. |
| first_indexed | 2024-03-09T01:12:11Z |
| format | Article |
| id | doaj.art-f047cc880a7148d98829b010a407d729 |
| institution | Directory Open Access Journal |
| issn | 2405-6316 |
| language | English |
| last_indexed | 2024-03-09T01:12:11Z |
| publishDate | 2023-10-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Physics and Imaging in Radiation Oncology |
| spelling | doaj.art-f047cc880a7148d98829b010a407d7292023-12-11T04:16:28ZengElsevierPhysics and Imaging in Radiation Oncology2405-63162023-10-0128100494A geometry and dose-volume based performance monitoring of artificial intelligence models in radiotherapy treatment planning for prostate cancerGeert De Kerf0Michaël Claessens1Fadoua Raouassi2Carole Mercier3Daan Stas4Piet Ost5Piet Dirix6Dirk Verellen7Department of Radiation Oncology, Iridium Netwerk, Wilrijk (Antwerp), Belgium; Corresponding author.Department of Radiation Oncology, Iridium Netwerk, Wilrijk (Antwerp), Belgium; Centre for Oncological Research (CORE), Integrated Personalized and Precision Oncology Network (IPPON), University of Antwerp, Antwerp, BelgiumDepartment of Radiation Oncology, Iridium Netwerk, Wilrijk (Antwerp), BelgiumDepartment of Radiation Oncology, Iridium Netwerk, Wilrijk (Antwerp), Belgium; Centre for Oncological Research (CORE), Integrated Personalized and Precision Oncology Network (IPPON), University of Antwerp, Antwerp, BelgiumDepartment of Radiation Oncology, Iridium Netwerk, Wilrijk (Antwerp), Belgium; Faculty of Medicine and Health Sciences, University of Antwerp, Antwerp, BelgiumDepartment of Radiation Oncology, Iridium Netwerk, Wilrijk (Antwerp), Belgium; Centre for Oncological Research (CORE), Integrated Personalized and Precision Oncology Network (IPPON), University of Antwerp, Antwerp, BelgiumDepartment of Radiation Oncology, Iridium Netwerk, Wilrijk (Antwerp), Belgium; Centre for Oncological Research (CORE), Integrated Personalized and Precision Oncology Network (IPPON), University of Antwerp, Antwerp, BelgiumDepartment of Radiation Oncology, Iridium Netwerk, Wilrijk (Antwerp), Belgium; Centre for Oncological Research (CORE), Integrated Personalized and Precision Oncology Network (IPPON), University of Antwerp, Antwerp, BelgiumBackground and Purpose: Clinical Artificial Intelligence (AI) implementations lack ground-truth when applied on real-world data. This study investigated how combined geometrical and dose-volume metrics can be used as performance monitoring tools to detect clinically relevant candidates for model retraining. Materials and Methods: Fifty patients were analyzed for both AI-segmentation and planning. For AI-segmentation, geometrical (Standard Surface Dice 3 mm and Local Surface Dice 3 mm) and dose-volume based parameters were calculated for two organs (bladder and anorectum) to compare AI output against the clinically corrected structure. A Local Surface Dice was introduced to detect geometrical changes in the vicinity of the target volumes, while an Absolute Dose Difference (ADD) evaluation increased focus on dose-volume related changes. AI-planning performance was evaluated using clinical goal analysis in combination with volume and target overlap metrics. Results: The Local Surface Dice reported equal or lower values compared to the Standard Surface Dice (anorectum: (0.93 ± 0.11) vs (0.98 ± 0.04); bladder: (0.97 ± 0.06) vs (0.98 ± 0.04)). The ADD metric showed a difference of (0.9 ± 0.8)Gy for the anorectum D1cm3. The bladder D5cm3 reported a difference of (0.7 ± 1.5)Gy. Mandatory clinical goals were fulfilled in 90 % of the DLP plans. Conclusions: Combining dose-volume and geometrical metrics allowed detection of clinically relevant changes, applied to both auto-segmentation and auto-planning output and the Local Surface Dice was more sensitive to local changes compared to the Standard Surface Dice. This monitoring is able to evaluate AI behavior in clinical practice and allows candidate selection for active learning.http://www.sciencedirect.com/science/article/pii/S2405631623000854Performance monitoringArtificial intelligenceSBRT prostateDeep Learning SegmentationDeep Learning PlanningClinical metrics |
| spellingShingle | Geert De Kerf Michaël Claessens Fadoua Raouassi Carole Mercier Daan Stas Piet Ost Piet Dirix Dirk Verellen A geometry and dose-volume based performance monitoring of artificial intelligence models in radiotherapy treatment planning for prostate cancer Physics and Imaging in Radiation Oncology Performance monitoring Artificial intelligence SBRT prostate Deep Learning Segmentation Deep Learning Planning Clinical metrics |
| title | A geometry and dose-volume based performance monitoring of artificial intelligence models in radiotherapy treatment planning for prostate cancer |
| title_full | A geometry and dose-volume based performance monitoring of artificial intelligence models in radiotherapy treatment planning for prostate cancer |
| title_fullStr | A geometry and dose-volume based performance monitoring of artificial intelligence models in radiotherapy treatment planning for prostate cancer |
| title_full_unstemmed | A geometry and dose-volume based performance monitoring of artificial intelligence models in radiotherapy treatment planning for prostate cancer |
| title_short | A geometry and dose-volume based performance monitoring of artificial intelligence models in radiotherapy treatment planning for prostate cancer |
| title_sort | geometry and dose volume based performance monitoring of artificial intelligence models in radiotherapy treatment planning for prostate cancer |
| topic | Performance monitoring Artificial intelligence SBRT prostate Deep Learning Segmentation Deep Learning Planning Clinical metrics |
| url | http://www.sciencedirect.com/science/article/pii/S2405631623000854 |
| work_keys_str_mv | AT geertdekerf ageometryanddosevolumebasedperformancemonitoringofartificialintelligencemodelsinradiotherapytreatmentplanningforprostatecancer AT michaelclaessens ageometryanddosevolumebasedperformancemonitoringofartificialintelligencemodelsinradiotherapytreatmentplanningforprostatecancer AT fadouaraouassi ageometryanddosevolumebasedperformancemonitoringofartificialintelligencemodelsinradiotherapytreatmentplanningforprostatecancer AT carolemercier ageometryanddosevolumebasedperformancemonitoringofartificialintelligencemodelsinradiotherapytreatmentplanningforprostatecancer AT daanstas ageometryanddosevolumebasedperformancemonitoringofartificialintelligencemodelsinradiotherapytreatmentplanningforprostatecancer AT pietost ageometryanddosevolumebasedperformancemonitoringofartificialintelligencemodelsinradiotherapytreatmentplanningforprostatecancer AT pietdirix ageometryanddosevolumebasedperformancemonitoringofartificialintelligencemodelsinradiotherapytreatmentplanningforprostatecancer AT dirkverellen ageometryanddosevolumebasedperformancemonitoringofartificialintelligencemodelsinradiotherapytreatmentplanningforprostatecancer AT geertdekerf geometryanddosevolumebasedperformancemonitoringofartificialintelligencemodelsinradiotherapytreatmentplanningforprostatecancer AT michaelclaessens geometryanddosevolumebasedperformancemonitoringofartificialintelligencemodelsinradiotherapytreatmentplanningforprostatecancer AT fadouaraouassi geometryanddosevolumebasedperformancemonitoringofartificialintelligencemodelsinradiotherapytreatmentplanningforprostatecancer AT carolemercier geometryanddosevolumebasedperformancemonitoringofartificialintelligencemodelsinradiotherapytreatmentplanningforprostatecancer AT daanstas geometryanddosevolumebasedperformancemonitoringofartificialintelligencemodelsinradiotherapytreatmentplanningforprostatecancer AT pietost geometryanddosevolumebasedperformancemonitoringofartificialintelligencemodelsinradiotherapytreatmentplanningforprostatecancer AT pietdirix geometryanddosevolumebasedperformancemonitoringofartificialintelligencemodelsinradiotherapytreatmentplanningforprostatecancer AT dirkverellen geometryanddosevolumebasedperformancemonitoringofartificialintelligencemodelsinradiotherapytreatmentplanningforprostatecancer |