A clinical evaluation of the performance of five commercial artificial intelligence contouring systems for radiotherapy
Purpose/objective(s)Auto-segmentation with artificial intelligence (AI) offers an opportunity to reduce inter- and intra-observer variability in contouring, to improve the quality of contours, as well as to reduce the time taken to conduct this manual task. In this work we benchmark the AI auto-segm...
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-08-01
|
| Series: | Frontiers in Oncology |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fonc.2023.1213068/full |
| _version_ | 1797754591380504576 |
|---|---|
| author | Paul J. Doolan Stefanie Charalambous Yiannis Roussakis Agnes Leczynski Mary Peratikou Melka Benjamin Konstantinos Ferentinos Konstantinos Ferentinos Iosif Strouthos Iosif Strouthos Constantinos Zamboglou Constantinos Zamboglou Constantinos Zamboglou Efstratios Karagiannis Efstratios Karagiannis |
| author_facet | Paul J. Doolan Stefanie Charalambous Yiannis Roussakis Agnes Leczynski Mary Peratikou Melka Benjamin Konstantinos Ferentinos Konstantinos Ferentinos Iosif Strouthos Iosif Strouthos Constantinos Zamboglou Constantinos Zamboglou Constantinos Zamboglou Efstratios Karagiannis Efstratios Karagiannis |
| author_sort | Paul J. Doolan |
| collection | DOAJ |
| description | Purpose/objective(s)Auto-segmentation with artificial intelligence (AI) offers an opportunity to reduce inter- and intra-observer variability in contouring, to improve the quality of contours, as well as to reduce the time taken to conduct this manual task. In this work we benchmark the AI auto-segmentation contours produced by five commercial vendors against a common dataset.Methods and materialsThe organ at risk (OAR) contours generated by five commercial AI auto-segmentation solutions (Mirada (Mir), MVision (MV), Radformation (Rad), RayStation (Ray) and TheraPanacea (Ther)) were compared to manually-drawn expert contours from 20 breast, 20 head and neck, 20 lung and 20 prostate patients. Comparisons were made using geometric similarity metrics including volumetric and surface Dice similarity coefficient (vDSC and sDSC), Hausdorff distance (HD) and Added Path Length (APL). To assess the time saved, the time taken to manually draw the expert contours, as well as the time to correct the AI contours, were recorded.ResultsThere are differences in the number of CT contours offered by each AI auto-segmentation solution at the time of the study (Mir 99; MV 143; Rad 83; Ray 67; Ther 86), with all offering contours of some lymph node levels as well as OARs. Averaged across all structures, the median vDSCs were good for all systems and compared favorably with existing literature: Mir 0.82; MV 0.88; Rad 0.86; Ray 0.87; Ther 0.88. All systems offer substantial time savings, ranging between: breast 14-20 mins; head and neck 74-93 mins; lung 20-26 mins; prostate 35-42 mins. The time saved, averaged across all structures, was similar for all systems: Mir 39.8 mins; MV 43.6 mins; Rad 36.6 min; Ray 43.2 mins; Ther 45.2 mins.ConclusionsAll five commercial AI auto-segmentation solutions evaluated in this work offer high quality contours in significantly reduced time compared to manual contouring, and could be used to render the radiotherapy workflow more efficient and standardized. |
| first_indexed | 2024-03-12T17:35:35Z |
| format | Article |
| id | doaj.art-73b718373bcd4d169191e3121313e96b |
| institution | Directory Open Access Journal |
| issn | 2234-943X |
| language | English |
| last_indexed | 2024-03-12T17:35:35Z |
| publishDate | 2023-08-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Oncology |
| spelling | doaj.art-73b718373bcd4d169191e3121313e96b2023-08-04T11:33:14ZengFrontiers Media S.A.Frontiers in Oncology2234-943X2023-08-011310.3389/fonc.2023.12130681213068A clinical evaluation of the performance of five commercial artificial intelligence contouring systems for radiotherapyPaul J. Doolan0Stefanie Charalambous1Yiannis Roussakis2Agnes Leczynski3Mary Peratikou4Melka Benjamin5Konstantinos Ferentinos6Konstantinos Ferentinos7Iosif Strouthos8Iosif Strouthos9Constantinos Zamboglou10Constantinos Zamboglou11Constantinos Zamboglou12Efstratios Karagiannis13Efstratios Karagiannis14Department of Medical Physics, German Oncology Center, Limassol, CyprusDepartment of Radiation Oncology, German Oncology Center, Limassol, CyprusDepartment of Medical Physics, German Oncology Center, Limassol, CyprusDepartment of Radiation Oncology, German Oncology Center, Limassol, CyprusDepartment of Radiation Oncology, German Oncology Center, Limassol, CyprusDepartment of Radiation Oncology, German Oncology Center, Limassol, CyprusDepartment of Radiation Oncology, German Oncology Center, Limassol, CyprusSchool of Medicine, European University Cyprus, Nicosia, CyprusDepartment of Radiation Oncology, German Oncology Center, Limassol, CyprusSchool of Medicine, European University Cyprus, Nicosia, CyprusDepartment of Radiation Oncology, German Oncology Center, Limassol, CyprusSchool of Medicine, European University Cyprus, Nicosia, CyprusDepartment of Radiation Oncology, Medical Center – University of Freiberg, Freiberg, GermanyDepartment of Radiation Oncology, German Oncology Center, Limassol, CyprusSchool of Medicine, European University Cyprus, Nicosia, CyprusPurpose/objective(s)Auto-segmentation with artificial intelligence (AI) offers an opportunity to reduce inter- and intra-observer variability in contouring, to improve the quality of contours, as well as to reduce the time taken to conduct this manual task. In this work we benchmark the AI auto-segmentation contours produced by five commercial vendors against a common dataset.Methods and materialsThe organ at risk (OAR) contours generated by five commercial AI auto-segmentation solutions (Mirada (Mir), MVision (MV), Radformation (Rad), RayStation (Ray) and TheraPanacea (Ther)) were compared to manually-drawn expert contours from 20 breast, 20 head and neck, 20 lung and 20 prostate patients. Comparisons were made using geometric similarity metrics including volumetric and surface Dice similarity coefficient (vDSC and sDSC), Hausdorff distance (HD) and Added Path Length (APL). To assess the time saved, the time taken to manually draw the expert contours, as well as the time to correct the AI contours, were recorded.ResultsThere are differences in the number of CT contours offered by each AI auto-segmentation solution at the time of the study (Mir 99; MV 143; Rad 83; Ray 67; Ther 86), with all offering contours of some lymph node levels as well as OARs. Averaged across all structures, the median vDSCs were good for all systems and compared favorably with existing literature: Mir 0.82; MV 0.88; Rad 0.86; Ray 0.87; Ther 0.88. All systems offer substantial time savings, ranging between: breast 14-20 mins; head and neck 74-93 mins; lung 20-26 mins; prostate 35-42 mins. The time saved, averaged across all structures, was similar for all systems: Mir 39.8 mins; MV 43.6 mins; Rad 36.6 min; Ray 43.2 mins; Ther 45.2 mins.ConclusionsAll five commercial AI auto-segmentation solutions evaluated in this work offer high quality contours in significantly reduced time compared to manual contouring, and could be used to render the radiotherapy workflow more efficient and standardized.https://www.frontiersin.org/articles/10.3389/fonc.2023.1213068/fullAIcontouringradiotherapybreasthead and necklung |
| spellingShingle | Paul J. Doolan Stefanie Charalambous Yiannis Roussakis Agnes Leczynski Mary Peratikou Melka Benjamin Konstantinos Ferentinos Konstantinos Ferentinos Iosif Strouthos Iosif Strouthos Constantinos Zamboglou Constantinos Zamboglou Constantinos Zamboglou Efstratios Karagiannis Efstratios Karagiannis A clinical evaluation of the performance of five commercial artificial intelligence contouring systems for radiotherapy Frontiers in Oncology AI contouring radiotherapy breast head and neck lung |
| title | A clinical evaluation of the performance of five commercial artificial intelligence contouring systems for radiotherapy |
| title_full | A clinical evaluation of the performance of five commercial artificial intelligence contouring systems for radiotherapy |
| title_fullStr | A clinical evaluation of the performance of five commercial artificial intelligence contouring systems for radiotherapy |
| title_full_unstemmed | A clinical evaluation of the performance of five commercial artificial intelligence contouring systems for radiotherapy |
| title_short | A clinical evaluation of the performance of five commercial artificial intelligence contouring systems for radiotherapy |
| title_sort | clinical evaluation of the performance of five commercial artificial intelligence contouring systems for radiotherapy |
| topic | AI contouring radiotherapy breast head and neck lung |
| url | https://www.frontiersin.org/articles/10.3389/fonc.2023.1213068/full |
| work_keys_str_mv | AT pauljdoolan aclinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT stefaniecharalambous aclinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT yiannisroussakis aclinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT agnesleczynski aclinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT maryperatikou aclinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT melkabenjamin aclinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT konstantinosferentinos aclinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT konstantinosferentinos aclinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT iosifstrouthos aclinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT iosifstrouthos aclinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT constantinoszamboglou aclinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT constantinoszamboglou aclinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT constantinoszamboglou aclinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT efstratioskaragiannis aclinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT efstratioskaragiannis aclinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT pauljdoolan clinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT stefaniecharalambous clinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT yiannisroussakis clinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT agnesleczynski clinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT maryperatikou clinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT melkabenjamin clinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT konstantinosferentinos clinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT konstantinosferentinos clinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT iosifstrouthos clinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT iosifstrouthos clinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT constantinoszamboglou clinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT constantinoszamboglou clinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT constantinoszamboglou clinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT efstratioskaragiannis clinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy AT efstratioskaragiannis clinicalevaluationoftheperformanceoffivecommercialartificialintelligencecontouringsystemsforradiotherapy |