An interpretable AI model for recurrence prediction after surgery in gastrointestinal stromal tumour: an observational cohort studyResearch in context
Summary: Background: There are several models that predict the risk of recurrence following resection of localised, primary gastrointestinal stromal tumour (GIST). However, assessment of calibration is not always feasible and when performed, calibration of current GIST models appears to be suboptim...
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2023-10-01
|
| Series: | EClinicalMedicine |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2589537023003772 |
| _version_ | 1797689270611214336 |
|---|---|
| author | Dimitris Bertsimas Georgios Antonios Margonis Seehanah Tang Angelos Koulouras Cristina R. Antonescu Murray F. Brennan Javier Martin-Broto Piotr Rutkowski Georgios Stasinos Jane Wang Emmanouil Pikoulis Elzbieta Bylina Pawel Sobczuk Antonio Gutierrez Bhumika Jadeja William D. Tap Ping Chi Samuel Singer |
| author_facet | Dimitris Bertsimas Georgios Antonios Margonis Seehanah Tang Angelos Koulouras Cristina R. Antonescu Murray F. Brennan Javier Martin-Broto Piotr Rutkowski Georgios Stasinos Jane Wang Emmanouil Pikoulis Elzbieta Bylina Pawel Sobczuk Antonio Gutierrez Bhumika Jadeja William D. Tap Ping Chi Samuel Singer |
| author_sort | Dimitris Bertsimas |
| collection | DOAJ |
| description | Summary: Background: There are several models that predict the risk of recurrence following resection of localised, primary gastrointestinal stromal tumour (GIST). However, assessment of calibration is not always feasible and when performed, calibration of current GIST models appears to be suboptimal. We aimed to develop a prognostic model to predict the recurrence of GIST after surgery with both good discrimination and calibration by uncovering and harnessing the non-linear relationships among variables that predict recurrence. Methods: In this observational cohort study, the data of 395 adult patients who underwent complete resection (R0 or R1) of a localised, primary GIST in the pre-imatinib era at Memorial Sloan Kettering Cancer Center (NY, USA) (recruited 1982–2001) and a European consortium (Spanish Group for Research in Sarcomas, 80 sites) (recruited 1987–2011) were used to train an interpretable Artificial Intelligence (AI)-based model called Optimal Classification Trees (OCT). The OCT predicted the probability of recurrence after surgery by capturing non-linear relationships among predictors of recurrence. The data of an additional 596 patients from another European consortium (Polish Clinical GIST Registry, 7 sites) (recruited 1981–2013) who were also treated in the pre-imatinib era were used to externally validate the OCT predictions with regard to discrimination (Harrell's C-index and Brier score) and calibration (calibration curve, Brier score, and Hosmer-Lemeshow test). The calibration of the Memorial Sloan Kettering (MSK) GIST nomogram was used as a comparative gold standard. We also evaluated the clinical utility of the OCT and the MSK nomogram by performing a Decision Curve Analysis (DCA). Findings: The internal cohort included 395 patients (median [IQR] age, 63 [54–71] years; 214 men [54.2%]) and the external cohort included 556 patients (median [IQR] age, 60 [52–68] years; 308 men [55.4%]). The Harrell's C-index of the OCT in the external validation cohort was greater than that of the MSK nomogram (0.805 (95% CI: 0.803–0.808) vs 0.788 (95% CI: 0.786–0.791), respectively). In the external validation cohort, the slope and intercept of the calibration curve of the main OCT were 1.041 and 0.038, respectively. In comparison, the slope and intercept of the calibration curve for the MSK nomogram was 0.681 and 0.032, respectively. The MSK nomogram overestimated the recurrence risk throughout the entire calibration curve. Of note, the Brier score was lower for the OCT compared to the MSK nomogram (0.147 vs 0.564, respectively), and the Hosmer-Lemeshow test was insignificant (P = 0.087) for the OCT model but significant (P < 0.001) for the MSK nomogram. Both results confirmed the superior discrimination and calibration of the OCT over the MSK nomogram. A decision curve analysis showed that the AI-based OCT model allowed for superior decision making compared to the MSK nomogram for both patients with 25–50% recurrence risk as well as those with >50% risk of recurrence. Interpretation: We present the first prognostic models of recurrence risk in GIST that demonstrate excellent discrimination, calibration, and clinical utility on external validation. Additional studies for further validation are warranted. With further validation, these tools could potentially improve patient counseling and selection for adjuvant therapy. Funding: The NCI SPORE in Soft Tissue Sarcoma and NCI Cancer Center Support Grants. |
| first_indexed | 2024-03-12T01:43:19Z |
| format | Article |
| id | doaj.art-53e3b7f9ab6149689ef9632e7de90721 |
| institution | Directory Open Access Journal |
| issn | 2589-5370 |
| language | English |
| last_indexed | 2024-03-12T01:43:19Z |
| publishDate | 2023-10-01 |
| publisher | Elsevier |
| record_format | Article |
| series | EClinicalMedicine |
| spelling | doaj.art-53e3b7f9ab6149689ef9632e7de907212023-09-10T04:24:39ZengElsevierEClinicalMedicine2589-53702023-10-0164102200An interpretable AI model for recurrence prediction after surgery in gastrointestinal stromal tumour: an observational cohort studyResearch in contextDimitris Bertsimas0Georgios Antonios Margonis1Seehanah Tang2Angelos Koulouras3Cristina R. Antonescu4Murray F. Brennan5Javier Martin-Broto6Piotr Rutkowski7Georgios Stasinos8Jane Wang9Emmanouil Pikoulis10Elzbieta Bylina11Pawel Sobczuk12Antonio Gutierrez13Bhumika Jadeja14William D. Tap15Ping Chi16Samuel Singer17Operations Research Center, Massachusetts Institute of Technology, Cambridge, MA, USADepartment of Surgery, Memorial Sloan Kettering Cancer Center, New York, NY, USAOperations Research Center, Massachusetts Institute of Technology, Cambridge, MA, USAOperations Research Center, Massachusetts Institute of Technology, Cambridge, MA, USADepartment of Pathology and Laboratory Medicine, Memorial Sloan Kettering Cancer Center, New York, NY, USADepartment of Surgery, Memorial Sloan Kettering Cancer Center, New York, NY, USAMedical Oncology Department, Fundación Jimenez Diaz University Hospital, Madrid, Spain; Hospital General de Villalba, Madrid, Spain; Instituto de Investigacion Sanitaria Fundacion Jimenez Diaz (IIS/FJD; UAM), Madrid, SpainMaria Sklodowska-Curie National Research Institute of Oncology, Warsaw, PolandTechnical Chamber of Greece, Athens, GreeceDepartment of Surgery, University of California San Francisco, San Francisco, CA, USAThird Department of Surgery, Attikon University Hospital, Athens, GreeceMaria Sklodowska-Curie National Research Institute of Oncology, Warsaw, PolandMaria Sklodowska-Curie National Research Institute of Oncology, Warsaw, PolandMedical Oncology Department, Fundación Jimenez Diaz University Hospital, Madrid, Spain; Hospital General de Villalba, Madrid, Spain; Instituto de Investigacion Sanitaria Fundacion Jimenez Diaz (IIS/FJD; UAM), Madrid, SpainDepartment of Surgery, Memorial Sloan Kettering Cancer Center, New York, NY, USADepartment of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY, USADepartment of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY, USA; Human Oncology and Pathogenesis Program (HOPP), Memorial Sloan Kettering Cancer Center, New York, NY, USA; Department of Medicine, Weill Cornell Medical College, New York, NY, USADepartment of Surgery, Memorial Sloan Kettering Cancer Center, New York, NY, USA; Corresponding author. Howard 1205, Memorial Sloan Kettering Cancer Center, 1275 York Ave, New York, NY, 10065, USA.Summary: Background: There are several models that predict the risk of recurrence following resection of localised, primary gastrointestinal stromal tumour (GIST). However, assessment of calibration is not always feasible and when performed, calibration of current GIST models appears to be suboptimal. We aimed to develop a prognostic model to predict the recurrence of GIST after surgery with both good discrimination and calibration by uncovering and harnessing the non-linear relationships among variables that predict recurrence. Methods: In this observational cohort study, the data of 395 adult patients who underwent complete resection (R0 or R1) of a localised, primary GIST in the pre-imatinib era at Memorial Sloan Kettering Cancer Center (NY, USA) (recruited 1982–2001) and a European consortium (Spanish Group for Research in Sarcomas, 80 sites) (recruited 1987–2011) were used to train an interpretable Artificial Intelligence (AI)-based model called Optimal Classification Trees (OCT). The OCT predicted the probability of recurrence after surgery by capturing non-linear relationships among predictors of recurrence. The data of an additional 596 patients from another European consortium (Polish Clinical GIST Registry, 7 sites) (recruited 1981–2013) who were also treated in the pre-imatinib era were used to externally validate the OCT predictions with regard to discrimination (Harrell's C-index and Brier score) and calibration (calibration curve, Brier score, and Hosmer-Lemeshow test). The calibration of the Memorial Sloan Kettering (MSK) GIST nomogram was used as a comparative gold standard. We also evaluated the clinical utility of the OCT and the MSK nomogram by performing a Decision Curve Analysis (DCA). Findings: The internal cohort included 395 patients (median [IQR] age, 63 [54–71] years; 214 men [54.2%]) and the external cohort included 556 patients (median [IQR] age, 60 [52–68] years; 308 men [55.4%]). The Harrell's C-index of the OCT in the external validation cohort was greater than that of the MSK nomogram (0.805 (95% CI: 0.803–0.808) vs 0.788 (95% CI: 0.786–0.791), respectively). In the external validation cohort, the slope and intercept of the calibration curve of the main OCT were 1.041 and 0.038, respectively. In comparison, the slope and intercept of the calibration curve for the MSK nomogram was 0.681 and 0.032, respectively. The MSK nomogram overestimated the recurrence risk throughout the entire calibration curve. Of note, the Brier score was lower for the OCT compared to the MSK nomogram (0.147 vs 0.564, respectively), and the Hosmer-Lemeshow test was insignificant (P = 0.087) for the OCT model but significant (P < 0.001) for the MSK nomogram. Both results confirmed the superior discrimination and calibration of the OCT over the MSK nomogram. A decision curve analysis showed that the AI-based OCT model allowed for superior decision making compared to the MSK nomogram for both patients with 25–50% recurrence risk as well as those with >50% risk of recurrence. Interpretation: We present the first prognostic models of recurrence risk in GIST that demonstrate excellent discrimination, calibration, and clinical utility on external validation. Additional studies for further validation are warranted. With further validation, these tools could potentially improve patient counseling and selection for adjuvant therapy. Funding: The NCI SPORE in Soft Tissue Sarcoma and NCI Cancer Center Support Grants.http://www.sciencedirect.com/science/article/pii/S2589537023003772GISTPrognosisRecurrenceArtificial intelligence |
| spellingShingle | Dimitris Bertsimas Georgios Antonios Margonis Seehanah Tang Angelos Koulouras Cristina R. Antonescu Murray F. Brennan Javier Martin-Broto Piotr Rutkowski Georgios Stasinos Jane Wang Emmanouil Pikoulis Elzbieta Bylina Pawel Sobczuk Antonio Gutierrez Bhumika Jadeja William D. Tap Ping Chi Samuel Singer An interpretable AI model for recurrence prediction after surgery in gastrointestinal stromal tumour: an observational cohort studyResearch in context EClinicalMedicine GIST Prognosis Recurrence Artificial intelligence |
| title | An interpretable AI model for recurrence prediction after surgery in gastrointestinal stromal tumour: an observational cohort studyResearch in context |
| title_full | An interpretable AI model for recurrence prediction after surgery in gastrointestinal stromal tumour: an observational cohort studyResearch in context |
| title_fullStr | An interpretable AI model for recurrence prediction after surgery in gastrointestinal stromal tumour: an observational cohort studyResearch in context |
| title_full_unstemmed | An interpretable AI model for recurrence prediction after surgery in gastrointestinal stromal tumour: an observational cohort studyResearch in context |
| title_short | An interpretable AI model for recurrence prediction after surgery in gastrointestinal stromal tumour: an observational cohort studyResearch in context |
| title_sort | interpretable ai model for recurrence prediction after surgery in gastrointestinal stromal tumour an observational cohort studyresearch in context |
| topic | GIST Prognosis Recurrence Artificial intelligence |
| url | http://www.sciencedirect.com/science/article/pii/S2589537023003772 |
| work_keys_str_mv | AT dimitrisbertsimas aninterpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT georgiosantoniosmargonis aninterpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT seehanahtang aninterpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT angeloskoulouras aninterpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT cristinarantonescu aninterpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT murrayfbrennan aninterpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT javiermartinbroto aninterpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT piotrrutkowski aninterpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT georgiosstasinos aninterpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT janewang aninterpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT emmanouilpikoulis aninterpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT elzbietabylina aninterpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT pawelsobczuk aninterpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT antoniogutierrez aninterpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT bhumikajadeja aninterpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT williamdtap aninterpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT pingchi aninterpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT samuelsinger aninterpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT dimitrisbertsimas interpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT georgiosantoniosmargonis interpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT seehanahtang interpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT angeloskoulouras interpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT cristinarantonescu interpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT murrayfbrennan interpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT javiermartinbroto interpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT piotrrutkowski interpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT georgiosstasinos interpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT janewang interpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT emmanouilpikoulis interpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT elzbietabylina interpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT pawelsobczuk interpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT antoniogutierrez interpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT bhumikajadeja interpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT williamdtap interpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT pingchi interpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext AT samuelsinger interpretableaimodelforrecurrencepredictionaftersurgeryingastrointestinalstromaltumouranobservationalcohortstudyresearchincontext |